[00:14:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [00:19:35] [Grafana] RESOLVED: DatasourceNoData [00:23:34] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [00:23:35] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:23:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:23] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 32.44, 18.70, 8.02 [00:59:23] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.94, 11.49, 9.20 [01:01:23] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.46, 7.84, 8.14 [01:02:30] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! [01:04:41] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:07:14] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 8.52, 3.88 [01:07:30] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [01:08:47] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.060 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:09:13] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.97, 5.89, 3.49 [01:12:30] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceNoData [01:15:26] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/cw_history.sql (END - exit=0) [01:15:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:22] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/patches/patch-cw_comments-modify-cw_comment_timestamp.sql (END - exit=0) [01:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:32] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/patches/patch-cw_requests-add-cw_extra.sql (END - exit=0) [01:18:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:55] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/patches/patch-cw_requests-add-cw_locked.sql (END - exit=0) [01:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:50] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03data-service [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/60c3723c76d2...53c8696032a7 [01:19:53] [02CreateWiki] 07Universal-Omega 0353c8696 - Fix [01:19:55] [02CreateWiki] 07Universal-Omega synchronize pull request 03#567: Major rewrite/cleanup of CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/567 [01:20:26] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/patches/patch-cw_requests-modify-cw_private.sql (END - exit=0) [01:20:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:31] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/patches/patch-cw_requests-modify-cw_timestamp.sql (END - exit=0) [01:22:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:10] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawiki /srv/mediawiki-staging/1.42/extensions/CreateWiki/sql/patches/patch-cw_wikis-update-smallint-to-tinyint.sql --wikidb=mhglobal (END - exit=0) [01:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:55] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [01:30:28] miraheze/CreateWiki - Universal-Omega the build passed. [01:36:55] [Grafana] RESOLVED: DatasourceNoData [01:45:16] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/25613ea00886...2e66ef847cfc [01:45:17] [02puppet] 07Universal-Omega 032e66ef8 - Use stdlib::ensure_packages [01:46:29] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2e66ef847cfc...115b7ffaf1af [01:46:32] [02puppet] 07Universal-Omega 03115b7ff - Use stdlib::ensure_packages [01:48:53] PROBLEM - fanojo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanojo.wiki' expires in 15 day(s) (Tue 22 Oct 2024 01:41:50 AM GMT +0000). [01:49:04] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8bb33614cc46...a9dcf4184f43 [01:49:05] [02ssl] 07WikiTideSSLBot 03a9dcf41 - Bot: Update SSL cert for fanojo.wiki [01:58:35] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/056c1417b7a4...98a35ead12f5 [01:58:37] [02RottenLinks] 07BlankEclair 0398a35ea - Appease CI, config edition [01:58:38] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [02:03:20] finally [02:03:38] miraheze/RottenLinks - BlankEclair the build passed. [02:12:19] wait, who reviews these PRs? [02:46:53] RECOVERY - fanojo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fanojo.wiki' will expire on Sat 04 Jan 2025 12:50:28 AM GMT +0000. [02:52:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [03:02:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] RESOLVED: DatasourceNoData [03:07:35] [Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceError [04:08:30] What PRs? [04:08:42] cosmicalpha: miraheze extensions [04:08:45] in this case, rottenlinks [04:08:53] I can if you want [04:08:59] sure ^_^ [04:29:16] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/115b7ffaf1af...bb48edb3983c [04:29:18] [02puppet] 07Universal-Omega 03bb48edb - Make mwtask181 a staging server again [04:32:06] !log [@mwtask181] starting deploy of {'config': True} to all [04:32:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:32:41] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 35s [04:32:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:23] Well I just found out nothing has been auto deployed in 2 weeks [04:33:28] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [04:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:56] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 28s [04:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:35:19] !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': ['DataMaps', 'SpeciallyCursed']} to all [04:35:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:07] !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': ['DataMaps', 'SpeciallyCursed']} to all - SUCCESS in 47s [04:36:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:51] !log [universalomega@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.42'} to all [04:36:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:37:14] !log [universalomega@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.42'} to all - SUCCESS in 24s [04:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:37:50] wee [04:37:50] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.42'} to all [04:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:44:17] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.42'} to all - SUCCESS in 386s [04:44:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:59:55] !log [universalomega@mwtask181] ALTER TABLE ed_url_cache DROP post_vars; on rainversewiki [05:00:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:30] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! [05:06:30] [Grafana] RESOLVED: DatasourceError [05:12:04] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 136.35 ms [05:18:13] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.34 ms [06:04:03] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:06:04] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.32 ms [06:15:21] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 136.54 ms [06:19:28] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.34 ms [07:01:30] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [07:06:30] [Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceError [07:33:30] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 136.35 ms [07:35:31] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.32 ms [07:38:26] PROBLEM - with.cpt-ra.bid - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:00:29] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788949734 [08:00:32] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788950357 [08:00:35] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788950015 [08:00:37] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788950461 [08:00:39] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788949453 [08:00:42] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788950775 [08:00:44] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788950073 [08:00:47] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788952211 [08:00:48] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788952917 [08:00:49] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788951597 [08:00:51] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788953061 [08:00:53] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788954151 [08:00:55] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788951056 [08:00:56] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788952323 [08:00:59] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788952744 [08:01:01] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788955310 [08:01:03] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788954781 [08:01:04] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788953467 [08:01:06] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788955528 [08:01:09] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788956099 [08:01:12] [02mw-config] 07Universal-Omega reviewed pull request 03#5705 commit - 13https://github.com/miraheze/mw-config/pull/5705#discussion_r1788920131 [08:02:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [08:07:07] RECOVERY - with.cpt-ra.bid - Cloudflare on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Thu 28 Nov 2024 01:43:10 PM GMT +0000. [08:07:35] [Grafana] RESOLVED: DatasourceNoData [08:07:53] [02RottenLinks] 07BlankEclair reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788959168 [08:08:53] [02mw-config] 07songnguxyz synchronize pull request 03#5705: More configurations for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5705 [08:08:59] [02RottenLinks] 07BlankEclair reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788959733 [08:09:48] miraheze/mw-config - songnguxyz the build passed. [08:10:32] [02RottenLinks] 07BlankEclair reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788960449 [08:12:36] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788961355 [08:13:22] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788961659 [08:16:13] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788962797 [08:17:35] [02mw-config] 07Universal-Omega closed pull request 03#5705: More configurations for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5705 [08:17:38] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/8a82ee733762...1fec7d3c874c [08:17:40] [02mw-config] 07songnguxyz 031fec7d3 - More configurations for lhmnwiki (#5705) [08:18:30] miraheze/mw-config - Universal-Omega the build passed. [08:23:18] !log [@test151] starting deploy of {'config': True} to test151 [08:23:19] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [08:23:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:23:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:27:03] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+1/-0/±8] 13https://github.com/miraheze/RottenLinks/compare/98a35ead12f5...a038cc59324d [08:27:04] [02RottenLinks] 07BlankEclair 03a038cc5 - Make changes as requested [08:27:06] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [08:29:15] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/a038cc59324d...0ac472b62449 [08:29:16] [02RottenLinks] 07BlankEclair 030ac472b - Change namespace as requested in review [08:29:18] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [08:31:56] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/0ac472b62449...2e5be4e69811 [08:31:57] [02RottenLinks] 07BlankEclair 032e5be4e - Remove trailing newline [08:31:57] !log [@mwtask181] starting deploy of {'config': True} to all [08:31:58] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [08:32:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:32:17] i'm sure CI loves me that i keep making commits as it's running [08:32:36] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 38s [08:32:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:36:23] miraheze/RottenLinks - BlankEclair the build passed. [08:41:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [08:42:48] BlankEclair: it should auto cancel [08:42:52] it does [08:42:56] but i still feel bad for it [08:43:18] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+0/-0/±2] 13https://github.com/miraheze/RottenLinks/compare/2e5be4e69811...b2960ec07a6f [08:43:21] [02RottenLinks] 07BlankEclair 03b2960ec - Type hint RottenLinksLuaLibrary::onGetStatus() per request on IRC [08:43:22] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [08:43:40] icinga-miraheze: Cloudflare would like to disagree with you [08:47:49] miraheze/RottenLinks - BlankEclair the build passed. [08:51:35] [Grafana] RESOLVED: DatasourceNoData [08:56:43] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.37 ms [08:57:04] DatasourceNoData very helpful [08:57:21] at least it's resolved [08:57:33] CosmicAlpha: the new Cloudflare alerts are broken [08:57:38] I need to reopen that task [08:58:49] Yeah I figured lol [08:59:11] The dashboards work though alerts are broken it seems? [08:59:32] CosmicAlpha: the dashboards don't all work for me [08:59:44] oh maybe I wasn't seeing enough [09:00:36] We did afterall miss the fact we had no auto deployments in production for 2 weeks. I mean at least we did have a few manual deployements coincidentally though. [09:01:27] https://github.com/miraheze/puppet/commit/bb48edb3983c78fc1ceab681b7b25f56f3bcf31e [09:01:45] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [09:02:32] CosmicAlpha: why [09:02:33] Why [09:02:47] Our observability really sucks [09:02:49] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.90 ms [09:03:08] > [06/10/2024 20:02] Our observability really sucks [09:03:10] CosmicAlpha: grafana is completely broken for me tbh [09:03:18] technically i observed that extensions weren't being installed [09:03:34] BlankEclair: human observability is too late [09:03:35] but no one (visibly) took much action on it until i subtly reminded ca [09:03:59] Yeah and investigating that is what made me realize a lot of things werent properly working [09:04:10] There is such a thing called PCC [09:04:50] RhinosF1: I am going to setup a log file that updates whenever an auto deploy is ran then an icinga alert to monitor its age and alert if auto deploys don't run in say an hour. [09:05:08] CosmicAlpha: an hour is too short [09:05:15] We don't do commits every hour [09:05:32] CosmicAlpha: I have far better idea [09:05:41] Have deploy tool write the commit its deploying to a file [09:05:50] It was an example I don't literally mean an hour [09:05:50] Check if that matches the latest from git [09:05:59] could work [09:06:09] lets do it [09:06:38] my idea was just based on the backups check [09:06:46] yours is indeed better [09:07:21] i probably should check if i can restore from my (personal) backups lol [09:07:39] BlankEclair: yes, disaster resilience plans should always be tested [09:11:45] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [09:16:45] [Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceNoData[Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceNoData [09:22:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [09:26:26] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788992412 [09:26:29] [02RottenLinks] 07Universal-Omega reviewed pull request 03#81 commit - 13https://github.com/miraheze/RottenLinks/pull/81#discussion_r1788991572 [09:28:53] [02RottenLinks] 07BlankEclair created branch 03REL1_40 13https://github.com/miraheze/RottenLinks/commit/573f18a77aef96e33f8e70301512aaba47912344 [09:28:54] [02RottenLinks] 07BlankEclair created branch 03REL1_40 - 13https://github.com/miraheze/RottenLinks [09:28:59] [02RottenLinks] 07BlankEclair created branch 03REL1_41 - 13https://github.com/miraheze/RottenLinks [09:29:01] [02RottenLinks] 07BlankEclair created branch 03REL1_41 13https://github.com/miraheze/RottenLinks/commit/573f18a77aef96e33f8e70301512aaba47912344 [09:31:31] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+0/-0/±2] 13https://github.com/miraheze/RottenLinks/compare/b2960ec07a6f...5f10c7c102ef [09:31:33] [02RottenLinks] 07BlankEclair 035f10c7c - Bump versions and add changelog [09:31:35] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [09:32:35] [Grafana] RESOLVED: DatasourceNoData [09:34:24] [02RottenLinks] 07BlankEclair pushed 031 commit to 03T12708 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/5f10c7c102ef...2c3a6abdea28 [09:34:25] [02RottenLinks] 07BlankEclair 032c3a6ab - Update CHANGELOG.md [09:34:28] [02RottenLinks] 07BlankEclair synchronize pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [09:37:22] [02RottenLinks] 07BlankEclair closed pull request 03#81: T12708: Add APIs for RottenLinks to allow fetching a link's status - 13https://github.com/miraheze/RottenLinks/pull/81 [09:37:23] [02RottenLinks] 07BlankEclair pushed 031 commit to 03master [+7/-0/±7] 13https://github.com/miraheze/RottenLinks/compare/573f18a77aef...a2019146ef7f [09:37:25] [02RottenLinks] 07BlankEclair 03a201914 - T12708: Add APIs for RottenLinks to allow fetching a link's status (#81) [09:37:38] !log [universalomega@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'RottenLinks'} to all [09:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:38:12] !log [universalomega@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'RottenLinks'} to all - SUCCESS in 34s [09:38:26] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42'} to all [09:38:54] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'lang': 'en', 'versions': '1.42'} to all - SUCCESS in 29s [09:38:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:39:04] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.42'} to all [09:39:04] miraheze/RottenLinks - BlankEclair the build passed. [09:39:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:42:08] miraheze/RottenLinks - BlankEclair the build passed. [09:42:21] are you really running CI for all four branches? [09:42:23] alright then [09:42:32] wait no, i think it's just two [09:45:33] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.42'} to all - SUCCESS in 389s [09:45:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:47:37] [02RottenLinks] 07BlankEclair deleted branch 03T12708 - 13https://github.com/miraheze/RottenLinks [09:47:39] [02RottenLinks] 07BlankEclair deleted branch 03T12708 [10:45:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [10:50:35] [Grafana] RESOLVED: DatasourceNoData [10:50:49] icinga-miraheze: just stop talking mate [10:51:07] /kick #miraheze-tech-ops icinga-miraheze [10:56:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [10:58:13] icinga-miraheze: no, your data source is sick [10:58:46] They are all healthy [10:58:50] BlankEclair: stupid bot [10:59:04] lol [10:59:32] I don't think I have the power to mute it [11:01:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [11:01:56] You're missing a \n too [11:02:35] Why https://github.com/miraheze/puppet/blob/master/modules/monitoring/files/grafana-webhook.py#L52 [11:03:00] BlankEclair: what am I missing [11:03:09] i'm busy debugging another issue [11:03:20] by using reception as shell.php [11:04:30] Heh [11:05:14] apparently littering debug exceptions in production is 'not an acceptable thing to do' (/j), so this is my next best alternative [11:05:42] BlankEclair: I so wish we could give you shell [11:05:49] We really need to finish isolating beta [11:05:59] this is a prod-only issue unfortunately [11:06:04] What [11:06:05] How [11:06:09] wiki specific [11:06:15] awww [11:06:35] [Grafana] RESOLVED: DatasourceError[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] RESOLVED: DatasourceError[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [11:11:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [11:16:35] [Grafana] RESOLVED: DatasourceNoData [11:36:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [11:46:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [11:56:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] RESOLVED: DatasourceNoData [12:05:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [12:14:45] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.40 ms [12:16:46] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.09 ms [12:35:35] [Grafana] RESOLVED: DatasourceNoData [12:40:35] [Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [12:45:26] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.36 ms [12:49:31] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.36 ms [12:50:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [13:00:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [13:01:51] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 136.37 ms [13:03:52] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.37 ms [13:05:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [13:10:35] [Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceError[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [13:12:10] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 136.38 ms [13:14:11] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.36 ms [13:34:45] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 136.29 ms [13:35:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [13:36:46] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.40 ms [13:45:04] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 136.84 ms [13:45:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [13:49:06] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.53 ms [13:57:26] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 136.26 ms [14:00:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [14:01:30] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.35 ms [14:09:49] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 136.37 ms [14:16:02] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.32 ms [14:30:26] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 136.36 ms [14:32:27] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.43 ms [14:35:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [14:40:45] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 136.38 ms [14:46:50] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.36 ms [14:50:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [14:57:11] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 136.32 ms [14:59:12] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.65 ms [15:05:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] RESOLVED: DatasourceNoData [15:07:31] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:08:41] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:08:42] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:35] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [15:11:00] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:11:16] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:11:25] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 14.13, 10.18, 5.10 [15:11:34] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.30 ms [15:12:13] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:12:47] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.092 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [15:12:49] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [15:12:55] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:13:15] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [15:13:24] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 6.63, 9.44, 5.50 [15:14:07] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [15:15:35] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [15:20:35] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: DatasourceError[Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] RESOLVED: DatasourceError[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [15:28:05] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 136.42 ms [15:34:16] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.37 ms [15:45:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [15:45:37] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.32 ms [15:47:38] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.35 ms [15:50:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [15:50:35] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bb48edb3983c...df76735c594d [15:50:36] [02puppet] 07paladox 03df76735 - changeprop: set disable_delayed_execution to true [15:50:59] RECOVERY - changeprop151 Puppet on changeprop151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:54:59] PROBLEM - changeprop151 Puppet on changeprop151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [15:58:55] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 136.41 ms [16:00:35] [Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [16:03:01] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.36 ms [16:05:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [16:12:20] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 137.34 ms [16:16:28] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.19 ms [16:17:09] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5f88935af8b7 [16:17:11] [02puppet] 07paladox 035f88935 - Cloudflare: temporarily stop service [16:17:13] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:17:15] [02puppet] 07paladox opened pull request 03#3995: Cloudflare: temporarily stop service - 13https://github.com/miraheze/puppet/pull/3995 [16:17:21] [02puppet] 07coderabbitai[bot] commented on pull request 03#3995: Cloudflare: temporarily stop service - 13https://github.com/miraheze/puppet/pull/3995#issuecomment-2395494861 [16:17:47] [02puppet] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/5f88935af8b7...6e8d7a0faf18 [16:17:48] [02puppet] 07github-actions 036e8d7a0 - CI: lint puppet code to standards [16:17:50] [02puppet] 07github-actions[bot] synchronize pull request 03#3995: Cloudflare: temporarily stop service - 13https://github.com/miraheze/puppet/pull/3995 [16:18:18] [02puppet] 07paladox closed pull request 03#3995: Cloudflare: temporarily stop service - 13https://github.com/miraheze/puppet/pull/3995 [16:18:19] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/df76735c594d...85fe29e85a28 [16:18:22] [02puppet] 07paladox 0385fe29e - Cloudflare: temporarily stop service (#3995) [16:18:25] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:18:28] [02puppet] 07paladox deleted branch 03paladox-patch-1 [16:20:05] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:20:08] [02puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/402c0d502221 [16:20:11] [02puppet] 07paladox 03402c0d5 - Cloudflare: re-eanble [16:20:13] [02puppet] 07paladox opened pull request 03#3996: Cloudflare: re-eanble - 13https://github.com/miraheze/puppet/pull/3996 [16:20:24] [02puppet] 07paladox closed pull request 03#3996: Cloudflare: re-eanble - 13https://github.com/miraheze/puppet/pull/3996 [16:20:27] [02puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/85fe29e85a28...ad22c6822506 [16:20:29] [02puppet] 07paladox 03ad22c68 - Cloudflare: re-eanble (#3996) [16:20:31] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [16:20:32] [02puppet] 07paladox deleted branch 03paladox-patch-1 [16:20:33] [02puppet] 07coderabbitai[bot] commented on pull request 03#3996: Cloudflare: re-eanble - 13https://github.com/miraheze/puppet/pull/3996#issuecomment-2395495870 [16:25:35] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] RESOLVED: DatasourceNoData[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare[Grafana] RESOLVED: DatasourceNoData [16:27:00] [02CreateWiki] 07Universal-Omega edited pull request 03#567: Major rewrite/cleanup of CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/567 [16:28:20] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03data-service [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/53c8696032a7...d2da50b8b65a [16:28:21] [02CreateWiki] 07Universal-Omega 03d2da50b - Update comment [16:28:22] [02CreateWiki] 07Universal-Omega synchronize pull request 03#567: Major rewrite/cleanup of CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/567 [16:30:35] [Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceError [16:41:57] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.46 ms [16:43:58] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.43 ms [16:44:28] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03data-service [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d2da50b8b65a...62b72cc239a2 [16:44:29] [02CreateWiki] 07Universal-Omega 0362b72cc - Add a getID method [16:44:31] [02CreateWiki] 07Universal-Omega synchronize pull request 03#567: Major rewrite/cleanup of CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/567 [16:54:46] miraheze/CreateWiki - Universal-Omega the build passed. [17:01:42] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare![Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare [17:05:54] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03data-service [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/62b72cc239a2...a8d26aeb7776 [17:05:55] [02CreateWiki] 07Universal-Omega 03a8d26ae - Fix and cleanup [17:05:59] [02CreateWiki] 07Universal-Omega synchronize pull request 03#567: Major rewrite/cleanup of CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/567 [17:06:42] [Grafana] RESOLVED: DatasourceError[Grafana] RESOLVED: DatasourceError [17:16:02] miraheze/CreateWiki - Universal-Omega the build passed. [17:38:42] PROBLEM - familiacorsi.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'familiacorsi.com' expires in 15 day(s) (Tue 22 Oct 2024 05:16:02 PM GMT +0000). [17:38:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a9dcf4184f43...018f2157dfa5 [17:38:56] [02ssl] 07WikiTideSSLBot 03018f215 - Bot: Update SSL cert for familiacorsi.com [17:45:38] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.38 ms [17:47:39] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 136.45 ms [18:07:52] RECOVERY - familiacorsi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'familiacorsi.com' will expire on Sat 04 Jan 2025 04:40:18 PM GMT +0000. [18:16:38] PROBLEM - echoes-wiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'echoes-wiki.com' expires in 15 day(s) (Tue 22 Oct 2024 05:50:06 PM GMT +0000). [18:16:50] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/018f2157dfa5...54cfd7edd52f [18:16:51] [02ssl] 07WikiTideSSLBot 0354cfd7e - Bot: Update SSL cert for echoes-wiki.com [18:18:04] PROBLEM - kagaga.jp - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kagaga.jp' expires in 15 day(s) (Tue 22 Oct 2024 06:13:40 PM GMT +0000). [18:18:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/54cfd7edd52f...0d854e3bd9d0 [18:18:24] [02ssl] 07WikiTideSSLBot 030d854e3 - Bot: Update SSL cert for kagaga.jp [19:14:49] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1fec7d3c874c...f2441c8cbdae [19:14:52] [02mw-config] 07OAuthority 03f2441c8 - mark TitleKey and CirrusSearch as incompatible extensions [19:15:54] RECOVERY - echoes-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'echoes-wiki.com' will expire on Sat 04 Jan 2025 05:18:15 PM GMT +0000. [19:15:55] miraheze/mw-config - OAuthority the build passed. [19:15:58] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f2441c8cbdae...2b65951768b1 [19:15:59] [02mw-config] 07silentgamer568 032b65951 - CodeMirror configuaration for abyssalwiki (#5700) [19:16:01] [02mw-config] 07OAuthority closed pull request 03#5700: CodeMirror configuaration for abyssalwiki - 13https://github.com/miraheze/mw-config/pull/5700 [19:16:49] miraheze/mw-config - OAuthority the build passed. [19:17:13] RECOVERY - kagaga.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'kagaga.jp' will expire on Sat 04 Jan 2025 05:19:43 PM GMT +0000. [19:17:33] !log [oa@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:17:37]