[00:08:24] <icinga-miraheze>	 PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.58, 3.28, 2.80
[00:10:24] <icinga-miraheze>	 RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.58, 3.16, 2.82
[00:15:25] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 51438MiB (11% inode=99%);
[00:26:34] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.16, 14.18, 6.99
[00:32:34] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 3.93, 13.08, 9.43
[01:09:26] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49784MiB (10% inode=99%);
[04:00:50] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.04, 14.12, 6.53
[04:02:50] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 20.08, 16.31, 8.27
[04:06:50] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.53, 19.34, 11.33
[04:12:50] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.87, 22.60, 15.27
[04:18:50] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 7.18, 18.54, 16.36
[04:39:10] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[04:39:56] <MacFanBot>	 [02mw-config] 07songnguxyz readied pull request #5829: Update lhmnwiki's configs (07miraheze:03master...07songnguxyz:03patch-4) 13https://github.com/miraheze/mw-config/pull/5829
[04:40:27] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[06:10:11] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/renameDatabase.php --wiki=loginwiki --old=the51stinfernocompanywiki --new=51stinfernocompanywiki (END - exit=0)
[06:10:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:10:58] <icinga-miraheze>	 PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 21914MiB (9% inode=99%);
[06:11:30] <icinga-miraheze>	 PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 21707MiB (9% inode=99%);
[06:28:58] <icinga-miraheze>	 RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 25333MiB (11% inode=99%);
[06:31:31] <icinga-miraheze>	 RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 25009MiB (11% inode=99%);
[06:52:19] <icinga-miraheze>	 PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 24332MiB (10% inode=99%);
[06:59:36] <icinga-miraheze>	 PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23949MiB (10% inode=99%);
[06:59:56] <icinga-miraheze>	 RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 24498MiB (11% inode=99%);
[07:03:45] <icinga-miraheze>	 PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23946MiB (10% inode=99%);
[09:26:58] <icinga-miraheze>	 PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 13084MiB (5% inode=99%);
[09:37:18] <MacFanBot>	 [02mw-config] 07Universal-Omega requested changes on pull request #5829 13https://github.com/miraheze/mw-config/pull/5829#pullrequestreview-2626189157
[09:37:20] <MacFanBot>	 [02mw-config] 07Universal-Omega left a file comment in pull request #5829 0346199a1: Remove 13https://github.com/miraheze/mw-config/pull/5829#discussion_r1961308622
[09:40:51] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[09:42:58] <icinga-miraheze>	 PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 17254MiB (7% inode=99%);
[09:43:35] <MacFanBot>	 [02mw-config] 07Universal-Omega left a file comment in pull request #5829 038fd6ddd: Lol sorry I wasn't clear, readd the line but remove the extra tabbing you had so this has no change being displayed at all. 13https://github.com/miraheze/mw-config/pull/5829#discussion_r1961319342
[12:14:42] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 21.99, 18.85, 11.21
[12:16:41] <icinga-miraheze>	 RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 13.73, 17.23, 11.54
[12:18:22] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.33, 8.69, 5.16
[12:22:07] <icinga-miraheze>	 PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 13.99, 9.44, 5.01
[12:22:23] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.81, 10.29, 6.58
[12:24:07] <icinga-miraheze>	 RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 3.70, 6.86, 4.59
[12:24:22] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.57, 8.87, 6.49
[12:34:45] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.21, 18.31, 13.90
[12:36:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:36:05] <icinga-miraheze>	 PROBLEM - mw184 Current Load on mw184 is WARNING: LOAD WARNING - total load average: 19.57, 20.57, 17.19
[12:36:45] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.12, 15.83, 13.51
[12:40:01] <icinga-miraheze>	 RECOVERY - mw184 Current Load on mw184 is OK: LOAD OK - total load average: 15.91, 19.13, 17.41
[12:40:22] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 17.88, 12.67, 9.26
[12:42:23] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.75, 11.84, 9.38
[12:44:22] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.53, 13.20, 10.20
[12:50:22] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.83, 9.18, 9.66
[12:51:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[14:20:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.42, 13.67, 6.65
[14:27:25] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27213MiB (5% inode=99%);
[14:34:34] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.41, 15.94, 7.85
[14:34:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.05, 22.91, 16.41
[14:36:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.79, 22.93, 17.20
[14:40:35] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.17, 20.20, 12.53
[14:42:43] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.94, 16.00, 16.51
[14:46:07] <icinga-miraheze>	 PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 11.95, 14.69, 8.29
[14:48:07] <icinga-miraheze>	 RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 2.98, 10.29, 7.45
[15:50:07] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=jbcstudioswiki cooleritemasylum_pages_full.xml --no-updates (START)
[15:50:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:51:31] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/refreshLinks.php --wiki=shintowiki (START)
[15:51:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:51:56] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/refreshLinks.php --wiki=metawiki (START)
[15:51:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:58:40] <icinga-miraheze>	 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server
[16:10:41] <icinga-miraheze>	 RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.222 second response time
[16:29:18] <MacFanBot>	 [02mw-config] 07Reception123 created 03Reception123-patch-1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/d3c4b272d611
[16:29:18] <MacFanBot>	 02mw-config/03Reception123-patch-1 07Reception123 03d3c4b27 resetwikicaches before CirrusSearch maintenance scripts
[16:29:31] <MacFanBot>	 [02mw-config] 07Reception123 opened pull request #5830: resetwikicaches before CirrusSearch maintenance scripts (03master...03Reception123-patch-1) 13https://github.com/miraheze/mw-config/pull/5830
[16:30:30] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[16:31:19] <MacFanBot>	 [02mw-config] 07Reception123 merged pull request #5830: resetwikicaches before CirrusSearch maintenance scripts (03master...03Reception123-patch-1) 13https://github.com/miraheze/mw-config/pull/5830
[16:31:19] <MacFanBot>	 [02mw-config] 07Reception123 pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/6dfedb56b46fce3555639912d8b6eb717cb90437
[16:31:19] <MacFanBot>	 02mw-config/03master 07Reception123 036dfedb5 resetwikicaches before CirrusSearch maintenance scripts (#5830)
[16:31:21] <MacFanBot>	 [02mw-config] 07Reception123 04deleted 03Reception123-patch-1 at 03d3c4b27 13https://github.com/miraheze/mw-config/commit/d3c4b27
[16:31:58] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[16:31:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:32:20] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s
[16:32:20] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[16:32:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:49:04] <icinga-miraheze>	 PROBLEM - ecole.science - LetsEncrypt on sslhost is CRITICAL: connect to address ecole.science and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:07] <icinga-miraheze>	 PROBLEM - rodzinka.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address rodzinka.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:09] <icinga-miraheze>	 PROBLEM - landofliberos.com - LetsEncrypt on sslhost is CRITICAL: connect to address landofliberos.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:13] <icinga-miraheze>	 PROBLEM - exmormon.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address exmormon.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:16] <icinga-miraheze>	 PROBLEM - dkpedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address dkpedia.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:16] <icinga-miraheze>	 PROBLEM - rainverse.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address rainverse.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:18] <icinga-miraheze>	 PROBLEM - corru.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address corru.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:49:19] <icinga-miraheze>	 PROBLEM - cp36 Nginx Backend for mwtask151 on cp36 is CRITICAL: connect to address localhost and port 8162: Connection refused
[16:50:25] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 98%
[16:50:29] <icinga-miraheze>	 PROBLEM - kalons-reverie.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:50:30] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 38.46.223.205/cpweb
[16:50:35] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received
[16:50:59] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[16:51:19] <icinga-miraheze>	 RECOVERY - cp36 Nginx Backend for mwtask151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8162
[16:52:25] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 3%
[16:52:29] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4113 bytes in 0.049 second response time
[16:52:30] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:52:59] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[16:59:59] <icinga-miraheze>	 PROBLEM - lads.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lads.wiki' expires in 14 day(s) (Thu 06 Mar 2025 04:34:20 PM GMT +0000).
[17:00:11] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/d72b266c6e6f5422ece7b76e6a65009d07421289
[17:00:11] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 03d72b266 Bot: Update SSL cert for lads.wiki
[17:03:33] <MacFanBot>	 [02mw-config] 07Reception123 created 03Reception123-patch-1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/5154f055b864
[17:03:33] <MacFanBot>	 02mw-config/03Reception123-patch-1 07Reception123 035154f05 Un-restrict CirrusSearch in ManageWiki…
[17:03:45] <MacFanBot>	 [02mw-config] 07Reception123 opened pull request #5831: Un-restrict CirrusSearch in ManageWiki (03master...03Reception123-patch-1) 13https://github.com/miraheze/mw-config/pull/5831
[17:03:50] <MirahezeRelay>	 <reception123> Free CirrusSearch!
[17:04:43] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[17:13:00] <MirahezeRelay>	 <pixldev, replying to reception123> What did they do to Cirrus
[17:18:15] <icinga-miraheze>	 RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Sun 09 Mar 2025 08:42:53 AM GMT +0000.
[17:18:33] <icinga-miraheze>	 RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Sun 09 Mar 2025 11:38:22 AM GMT +0000.
[17:18:33] <icinga-miraheze>	 RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Mon 21 Apr 2025 12:52:28 AM GMT +0000.
[17:18:35] <icinga-miraheze>	 RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Mon 07 Apr 2025 02:27:02 AM GMT +0000.
[17:18:36] <icinga-miraheze>	 RECOVERY - rodzinka.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rodzinka.wiki' will expire on Mon 28 Apr 2025 07:14:12 PM GMT +0000.
[17:18:56] <icinga-miraheze>	 RECOVERY - exmormon.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'exmormon.wiki' will expire on Thu 24 Apr 2025 01:56:43 PM GMT +0000.
[17:19:01] <icinga-miraheze>	 RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Sat 08 Mar 2025 12:02:35 AM GMT +0000.
[17:19:18] <icinga-miraheze>	 RECOVERY - kalons-reverie.com - LetsEncrypt on sslhost is OK: OK - Certificate 'kalons-reverie.com' will expire on Wed 26 Mar 2025 09:12:22 AM GMT +0000.
[17:22:16] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/MirahezeMagic/maintenance/replaceTextEligible.php --wiki=pandorastalewiki (END - exit=0)
[17:22:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:27:24] <MirahezeRelay>	 <reception123, replying to pixldev> they kept it restricted in managewiki!
[17:27:27] <MirahezeRelay>	 <reception123> by they I mean _we_ 😄
[17:29:27] <icinga-miraheze>	 RECOVERY - lads.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lads.wiki' will expire on Tue 20 May 2025 04:01:34 PM GMT +0000.
[17:34:10] <MirahezeRelay>	 <songngu.xyz, replying to reception123> SIUUUUU
[18:03:01] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[18:07:46] <MirahezeRelay>	 <pixldev, replying to reception123> Insert Obama meme here
[18:09:00] <MirahezeRelay>	 <songngu.xyz> https://tenor.com/view/obama-gif-18675569
[18:10:34] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.78, 13.89, 6.93
[18:12:34] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 7.04, 11.62, 7.00
[18:16:50] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.87, 16.67, 7.90
[18:26:50] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 7.84, 20.74, 15.78
[18:28:35] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 26.63, 15.34, 9.29
[18:28:50] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 1.68, 14.11, 13.95
[18:32:35] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 16.92, 16.29, 11.23
[18:36:35] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.75, 23.63, 15.45
[18:46:35] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 10.24, 22.46, 19.86
[18:48:34] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 23.50, 20.50
[18:50:06] <MacFan4000>	 hmm, high load on task171
[18:50:58] <MirahezeRelay>	 <agentisai> All mwtask* servers seem to cycle through high load every day
[18:51:08] <MirahezeRelay>	 <agentisai> might have to do with recent haproxy changes
[18:51:47] <MacFan4000>	 top suggests that the most intensive processes currently are php-fpm
[18:52:26] <MirahezeRelay>	 <agentisai> might be kafka paging /rpc/runSingleJob.php?
[18:57:08] <MacFan4000>	 Feb 19 18:56:22 mwtask171 php8.2-fpm[1215588]: [WARNING] [pool www] child 1215600, script '/srv/mediawiki/rpc/RunSingleJob.php' (request: "POST /rpc/RunSingleJob.php") executing too slow (19.700962 sec), logging
[18:57:16] <MacFan4000>	 thats exactly it @agentisai
[18:58:37] <MirahezeRelay>	 <agentisai> oh wow
[18:58:46] <MirahezeRelay>	 <agentisai> It might indeed be a haproxy thing then
[18:58:53] <MirahezeRelay>	 <agentisai> cc @paladox ^
[18:59:04] <MirahezeRelay>	 <agentisai> this seems to have started around the time of your latest haproxy change
[18:59:14] <MirahezeRelay>	 <paladox> it's required for SemanticMediaWiki
[18:59:29] <MirahezeRelay>	 <agentisai> that’s unfortunate
[18:59:39] <MirahezeRelay>	 <paladox> until https://github.com/SemanticMediaWiki/SemanticMediaWiki/commit/f931cc50631e97538f032bcf8a14af31657a77a5 is deployed
[18:59:40] <MacFanBot>	 [GitHub] [SemanticMediaWiki/SemanticMediaWiki] paladox: ChangePropagationDispatchJob: Don't presume job will be run on same server (#6021)… | 59 changes in 3 files | Authored 4 days, 3 hours ago | Committed 4 days, 3 hours ago
[18:59:47] <MirahezeRelay>	 <agentisai> ahhhh
[19:00:13] <MirahezeRelay>	 <agentisai> oh wait, that got merged?
[19:00:34] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 3.79, 16.62, 20.06
[19:04:16] <MirahezeRelay>	 <paladox> yes
[19:04:22] <MacFan4000>	 We install it via composer, not sure if that commit is in the latest dev release
[19:04:34] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 30.28, 23.69, 22.10
[19:05:06] <MirahezeRelay>	 <paladox> it is.
[19:05:14] <MirahezeRelay>	 <paladox> we use the master branch
[19:05:30] <MacFan4000>	 https://github.com/miraheze/mediawiki-repos/blob/master/composer.local.json#L3
[19:05:30] <MacFanBot>	 [GitHub] [miraheze/mediawiki-repos] composer.local.json @ master | L3:                 "mediawiki/semantic-media-wiki": "5.x-dev",
[19:07:42] <MirahezeRelay>	 <paladox> which is the master branch
[19:14:34] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 10.42, 23.41, 23.93
[19:18:35] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.40, 23.59, 23.56
[19:30:35] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 6.02, 19.13, 23.14
[19:30:44] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=jbcstudioswiki cooleritemasylum_pages_full.xml --no-updates (END - exit=0)
[19:30:45] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=jbcstudioswiki (START)
[19:30:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:30:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:34:35] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 8.33, 14.57, 20.33
[19:38:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 34.12, 22.95, 12.82
[19:49:20] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.71, 20.15, 12.94
[19:53:11] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.42, 22.05, 15.37
[19:57:01] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.67, 23.11, 17.21
[20:04:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.05, 23.54, 23.78
[20:08:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 23.77, 23.70
[20:10:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.06, 22.84, 23.34
[20:12:14] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=jbcstudioswiki (END - exit=0)
[20:12:15] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=jbcstudioswiki --update (END - exit=0)
[20:12:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:12:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:16:43] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 4.23, 12.60, 18.98
[20:45:19] <MirahezeRelay>	 <cosmicalpha, replying to paladox> isn't master-dev the master branch?
[20:45:56] <MirahezeRelay>	 <paladox> https://packagist.org/packages/mediawiki/semantic-media-wiki#dev-master
[20:46:16] <MirahezeRelay>	 <paladox> composer doesn't like use the word dev-master, instead you have to use the alias i think
[20:46:24] <MirahezeRelay>	 <paladox> which is 5.x-dev
[20:46:50] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 4.11, 17.10, 22.21
[20:47:03] <MirahezeRelay>	 <cosmicalpha> I installed it with dev-master before
[20:47:53] <MirahezeRelay>	 <cosmicalpha> https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/docs/INSTALL.md?plain=1#L112 shows to use dev-master also
[20:47:53] <MacFanBot>	 [GitHub] [SemanticMediaWiki/SemanticMediaWiki] docs/INSTALL.md @ master | L112: * master: `"mediawiki/semantic-media-wiki": "dev-master"`
[20:48:50] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 1.97, 12.00, 19.72
[20:49:18] <MirahezeRelay>	 <paladox> it wouldn't work for me
[20:49:31] <MirahezeRelay>	 <paladox> was failing with an error that i forgot what it was now as it was so long ago
[20:58:59] <icinga-miraheze>	 PROBLEM - allthetropes.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'allthetropes.org' expires in 14 day(s) (Thu 06 Mar 2025 08:38:36 PM GMT +0000).
[20:59:14] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/9dc43f0f2902d35ffc3e1e93650f345d0129d3cc
[20:59:14] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 039dc43f0 Bot: Update SSL cert for allthetropes.org
[21:20:03] <icinga-miraheze>	 PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 14 day(s) (Thu 06 Mar 2025 09:11:56 PM GMT +0000).
[21:20:14] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/e8a274abe40e79a5f1f4601f405d7f687d66945f
[21:20:14] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 03e8a274a Bot: Update SSL cert for speleo.wiki
[21:28:54] <icinga-miraheze>	 RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 20 May 2025 08:00:38 PM GMT +0000.
[21:31:07] <icinga-miraheze>	 PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thelonsdalebattalion.co.uk' expires in 14 day(s) (Thu 06 Mar 2025 09:28:33 PM GMT +0000).
[21:31:22] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/8a2eeefe315b52461722ff5f87096b6059c90ea2
[21:31:22] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 038a2eeef Bot: Update SSL cert for thelonsdalebattalion.co.uk
[21:38:30] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[21:38:39] <icinga-miraheze>	 PROBLEM - creativecommons.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:39] <icinga-miraheze>	 PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:40] <icinga-miraheze>	 PROBLEM - pokemonwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:46] <icinga-miraheze>	 PROBLEM - bobobay.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:48] <icinga-miraheze>	 PROBLEM - dccomicswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:50] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds
[21:38:50] <icinga-miraheze>	 PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:52] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw154 mw163 mw164 mw173 mw174 mw183 mw184 mediawiki
[21:38:55] <icinga-miraheze>	 PROBLEM - geeu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:59] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb
[21:39:13] <icinga-miraheze>	 PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:39:23] <icinga-miraheze>	 PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:39:41] <icinga-miraheze>	 PROBLEM - fnafmw.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:40:30] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:40:44] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4114 bytes in 0.070 second response time
[21:40:52] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 29 backends are healthy
[21:40:59] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[22:00:02] <icinga-miraheze>	 PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'marinebiodiversitymatrix.org' expires in 14 day(s) (Thu 06 Mar 2025 09:33:11 PM GMT +0000).
[22:00:13] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/64d0f1eab618eabb4e460cb819ff8bfc9798e6c3
[22:00:13] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 0364d0f1e Bot: Update SSL cert for marinebiodiversitymatrix.org
[22:00:18] <icinga-miraheze>	 RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Tue 20 May 2025 08:32:46 PM GMT +0000.
[22:07:21] <icinga-miraheze>	 RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Tue 18 Mar 2025 03:39:57 PM GMT +0000.
[22:07:22] <icinga-miraheze>	 RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Tue 11 Mar 2025 12:09:09 PM GMT +0000.
[22:07:25] <icinga-miraheze>	 RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Sun 09 Mar 2025 06:07:09 PM GMT +0000.
[22:07:47] <icinga-miraheze>	 RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 08 Mar 2025 03:26:18 PM GMT +0000.
[22:07:55] <icinga-miraheze>	 RECOVERY - geeu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'geeu.org' will expire on Sun 27 Apr 2025 02:05:19 PM GMT +0000.
[22:08:38] <icinga-miraheze>	 RECOVERY - bobobay.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bobobay.wiki' will expire on Mon 28 Apr 2025 01:38:22 PM GMT +0000.
[22:08:47] <icinga-miraheze>	 RECOVERY - dccomicswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dccomicswiki.com' will expire on Sun 18 May 2025 12:10:49 PM GMT +0000.
[22:09:12] <icinga-miraheze>	 RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sat 08 Mar 2025 01:09:01 PM GMT +0000.
[22:09:22] <icinga-miraheze>	 RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Sat 08 Mar 2025 03:13:41 PM GMT +0000.
[22:09:27] <icinga-miraheze>	 RECOVERY - fnafmw.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fnafmw.wiki' will expire on Thu 15 May 2025 11:15:17 PM GMT +0000.
[22:12:30] <MacFanBot>	 [02CreateWiki] 07BlankEclair approved pull request #669 13https://github.com/miraheze/CreateWiki/pull/669#pullrequestreview-2628094429
[22:18:03] <icinga-miraheze>	 RECOVERY - speleo.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'speleo.wiki' will expire on Tue 20 May 2025 08:21:38 PM GMT +0000.
[22:18:17] <MacFanBot>	 [02mw-config] 07Universal-Omega approved pull request #5829 13https://github.com/miraheze/mw-config/pull/5829#pullrequestreview-2628103244
[22:18:36] <MacFanBot>	 [02mw-config] 07Universal-Omega merged 07songnguxyz's pull request #5829: Update lhmnwiki's configs (07miraheze:03master...07songnguxyz:03patch-4) 13https://github.com/miraheze/mw-config/pull/5829
[22:18:36] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/ae005aa0176c245c5d796d90e30cfc3b5d6e6ae6
[22:18:37] <MacFanBot>	 02mw-config/03master 07Vũ Minh Nguyên 03ae005aa Update lhmnwiki's configs (#5829)…
[22:19:39] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[22:29:46] <icinga-miraheze>	 RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 20 May 2025 09:01:38 PM GMT +0000.