[00:00:02] !log [blankeclair@mwtask181] runRefreshLinksChunked: [900/2761] Handling kosmoswiki (start id: 0) [00:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:16] evil skin:cosmos: ^ [00:04:45] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 59111MiB (13% inode=99%); [00:07:26] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 58633MiB (12% inode=99%); [00:32:05] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [00:32:06] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=lostechwiki 4001 --e 6000 (running for 02:38) [00:32:07] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 160001 --e 162000 (running for 00:39) [00:32:08] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=sagan4alphawiki 12001 --e 14000 (running for 10:19) [00:32:09] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=lilypondwiki 1 --e 2000 (running for 02:45) [00:36:51] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.73, 6.87, 5.97 [00:40:50] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.02, 7.53, 6.40 [00:44:49] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.83, 7.86, 6.80 [00:56:45] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.78, 7.17 [00:58:45] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.96, 7.75, 7.23 [01:08:42] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.39, 7.87, 7.46 [01:10:44] !log [blankeclair@mwtask181] runRefreshLinksChunked: [1000/2761] Handling llofriuwiki (start id: 0) [01:10:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:41] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.68, 7.68, 7.49 [01:14:40] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.14, 8.15, 7.68 [01:16:40] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.75, 7.69, 7.57 [01:30:36] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.29, 7.77, 7.63 [01:32:07] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [01:32:08] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mandaepediawiki 1 --e 2000 (running for 05:27) [01:32:09] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 182001 --e 184000 (running for 01:18) [01:32:10] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=sagan4alphawiki 14001 --e 16000 (running for 40:08) [01:32:11] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 1 --e 2000 (running for 00:56) [01:34:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.91, 7.54, 7.60 [01:52:30] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.40, 5.89, 6.70 [02:05:00] !log [blankeclair@mwtask151] runRefreshLinksChunked: [100/1788] Handling mapwiki (start id: 0) [02:05:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:23] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.71, 7.25, 6.95 [02:11:23] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.40, 6.72, 6.78 [02:20:41] !log [blankeclair@mwtask171] runRefreshLinksChunked: [500/1403] Handling sanctisregnumwiki (start id: 0) [02:20:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:10] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [02:32:11] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mbta06hideoutwiki 14001 --e 16000 (running for 00:20) [02:32:12] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 212001 --e 214000 (running for 03:39) [02:32:13] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=sdiywikiwiki 2001 --e 4000 (running for 01:27) [02:32:14] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 4001 --e 6000 (running for 01:54) [02:47:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:05:47] !log [blankeclair@mwtask171] runRefreshLinksChunked: [600/1403] Handling simplepediacomputerwikiwiki (start id: 0) [03:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:43] oh yippee [03:15:18] clear faster! [03:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:32:12] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [03:32:13] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=memorydeltawiki 4001 --e 6000 (running for 01:25) [03:32:14] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 254001 --e 256000 (running for 01:14) [03:32:15] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=sleepyburglarfmwiki 1 --e 2000 (running for 01:15) [03:32:16] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 8001 --e 10000 (running for 06:11) [04:32:14] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [04:32:15] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=microtonalwiki 1 --e 2000 (running for 05:44) [04:32:16] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 16:46) [04:32:17] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=smutstonewiki 22001 --e 24000 (running for 02:31) [04:32:18] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 10001 --e 12000 (running for 58:07) [05:32:16] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [05:32:17] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=minecraftwiki 6001 --e 8000 (running for 03:58) [05:32:18] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 01:16:48) [05:32:19] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=sonaponawiki 4001 --e 6000 (running for 04:19) [05:32:20] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 10001 --e 12000 (running for 01:58:09) [06:24:25] why do we have a minecraft fork again? [06:24:42] PROBLEM - kingdomway.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kingdomway.wiki' expires in 14 day(s) (Mon 14 Apr 2025 05:53:16 AM GMT +0000). [06:24:45] > This wiki has been closed because Stewards or Trust and Safety have determined that it violates Miraheze's Content Policy or other global policies. Bureaucrats should have received either an email or talk page message with more information, including how to appeal. [06:24:46] oh [06:24:57] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/25b7ce8e5bac572e5dc37b1b9f01dab9c5017b99 [06:24:57] 02ssl/03master 07WikiTideSSLBot 0325b7ce8 Bot: Update SSL cert for kingdomway.wiki [06:31:06] cause funny [06:32:18] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [06:32:19] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=minecraftwiki 50001 --e 52000 (running for 01:01) [06:32:20] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 02:16:51) [06:32:21] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=soupandsorcerywiki 2001 --e 4000 (running for 22:30) [06:32:22] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 10001 --e 12000 (running for 02:58:12) [06:53:24] RECOVERY - kingdomway.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomway.wiki' will expire on Sat 28 Jun 2025 05:26:21 AM GMT +0000. [07:24:13] !log [blankeclair@mwtask151] runRefreshLinksChunked: [200/1788] Handling miperuwiki (start id: 0) [07:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:21] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [07:32:22] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=miraheziumwiki 1 --e 2000 (running for 08:03) [07:32:23] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 03:16:53) [07:32:24] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=speculativeevolutionwiki 2001 --e 4000 (running for 05:14) [07:32:25] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 12001 --e 14000 (running for 38:29) [08:32:23] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [08:32:24] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mysrblxwiki 1 --e 2000 (running for 05:16) [08:32:25] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 04:16:55) [08:32:26] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=sprixieincwiki 2001 --e 4000 (running for 11:20) [08:32:27] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 14001 --e 16000 (running for 23:18) [08:33:47] !log [blankeclair@mwtask171] runRefreshLinksChunked: [700/1403] Handling srredbaronwiki (start id: 0) [08:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:55] waow so smooth [08:57:44] !log [blankeclair@mwtask151] runRefreshLinksChunked: [300/1788] Handling narcissismresearchwiki (start id: 0) [08:57:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:57] i gotta put my parents in there asap [09:22:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.08, 16.31, 9.47 [09:24:31] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.08, 12.77, 9.01 [09:25:47] BlankEclair: what [09:26:54] RhinosF1: hat? [09:26:55] what? [09:27:21] BlankEclair: 09:57:57 i gotta put my parents in there asap [09:27:30] am i wrong- [09:27:44] BlankEclair: put your parents in where [09:27:50] narcissismresearchwiki [09:27:55] (if it wasn't clear, it's a /j) [09:28:02] Right [09:32:25] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [09:32:26] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ncountrymusicwiki 1 --e 2000 (running for 24:26) [09:32:27] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 05:16:58) [09:32:28] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=starfleetlogisticswiki 8001 --e 10000 (running for 06:26) [09:32:29] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 14001 --e 16000 (running for 01:23:20) [10:29:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:32:28] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [10:32:29] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nexuswiki 6001 --e 8000 (running for 04:39) [10:32:30] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 06:17:00) [10:32:31] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=stockcarracingwiki 4001 --e 6000 (running for 04:31) [10:32:32] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 16001 --e 18000 (running for 31:24) [10:42:55] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f76d68f9ffde [10:42:55] 02puppet/03paladox-patch-1 07paladox 03f76d68f changeprop201: increase webVideoTranscode* concurrency to 2 [10:43:10] [02puppet] 07paladox opened pull request #4277: changeprop201: increase webVideoTranscode* concurrency to 2 (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4277 [10:43:17] [02puppet] 07coderabbitai[bot] commented on pull request #4277: --- […] 13https://github.com/miraheze/puppet/pull/4277#issuecomment-2764497210 [10:43:27] [02puppet] 07paladox merged pull request #4277: changeprop201: increase webVideoTranscode* concurrency to 2 (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4277 [10:43:28] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/24b9d532c97d9b4216cd5c9b76ede24b35ac34a3 [10:43:28] 02puppet/03master 07paladox 0324b9d53 changeprop201: increase webVideoTranscode* concurrency to 2 (#4277) [10:43:28] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03f76d68f 13https://github.com/miraheze/puppet/commit/f76d68f [11:32:31] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [11:32:32] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nltramswiki 4001 --e 6000 (running for 02:21) [11:32:33] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 07:17:03) [11:32:34] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=stockcarracingwiki 8001 --e 10000 (running for 12:21) [11:32:35] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 16001 --e 18000 (running for 01:31:27) [12:12:38] PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [12:15:16] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:00] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:16:04] PROBLEM - mwtask161 jobrunner-high.wikitide.net HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10001 milliseconds [12:16:13] PROBLEM - mwtask161 SSH on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:43] PROBLEM - mwtask161 jobrunner.wikitide.net HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10002 milliseconds [12:16:53] PROBLEM - mwtask161 videoscaler.wikitide.net HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [12:16:55] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 105.36, 66.52, 32.41 [12:18:02] PROBLEM - mwtask161 mathoid on mwtask161 is CRITICAL: connect to address 10.0.16.157 and port 10044: Connection refused [12:18:13] RECOVERY - mwtask161 SSH on mwtask161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0) [12:18:38] RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:18:48] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:19:34] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:26] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49937MiB (10% inode=99%); [12:22:02] RECOVERY - mwtask161 mathoid on mwtask161 is OK: TCP OK - 0.000 second response time on 10.0.16.157 port 10044 [12:22:02] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.051 second response time [12:22:15] RECOVERY - mwtask161 jobrunner-high.wikitide.net HTTPS on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.011 second response time [12:22:52] RECOVERY - mwtask161 jobrunner.wikitide.net HTTPS on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.011 second response time [12:23:05] RECOVERY - mwtask161 videoscaler.wikitide.net HTTPS on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.011 second response time [12:23:15] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [12:23:35] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.530 second response time [12:23:35] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 6.24, 1.68, 0.58 [12:23:36] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:24:38] PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [12:27:26] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 34.09, 21.67, 12.31 [12:32:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.75, 21.67, 14.07 [12:32:34] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [12:32:35] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nltramswiki 30001 --e 32000 (running for 02:24) [12:32:36] mwtask161: not running [12:32:37] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=stockcarracingwiki 14001 --e 16000 (running for 00:55) [12:32:38] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 16001 --e 18000 (running for 02:31:30) [12:38:31] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.57, 23.19, 16.93 [12:43:33] PROBLEM - guia.esporo.net - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:52:03] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.55, 17.06, 13.43 [12:54:01] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.86, 20.45, 15.13 [12:55:26] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 19.07, 21.57, 23.88 [12:56:00] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.25, 20.72, 15.85 [12:59:26] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 26.98, 22.49, 23.59 [12:59:57] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 17.56, 19.99, 16.73 [13:03:26] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 14.43, 22.57, 23.71 [13:05:26] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 30.30, 25.43, 24.59 [13:07:51] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 35.00, 25.91, 19.63 [13:09:04] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.92, 19.09, 14.84 [13:09:26] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 11.70, 19.59, 22.58 [13:11:03] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 17.25, 18.50, 15.16 [13:13:26] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 14.19, 15.82, 20.26 [13:32:38] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [13:32:39] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nltramswiki 58001 (running for 40:56) [13:32:40] mwtask161: not running [13:32:41] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=stockcarracingwiki 20001 --e 22000 (running for 03:59) [13:32:42] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 18001 --e 20000 (running for 18:30) [13:36:11] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f1da2e810328 [13:36:11] 02puppet/03paladox-patch-1 07paladox 03f1da2e8 changeprop201: increase max.poll.interval.ms to 2h for webVideoTranscode… [13:36:14] [02puppet] 07paladox opened pull request #4278: changeprop201: increase max.poll.interval.ms to 2h for webVideoTranscode (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4278 [13:36:20] [02puppet] 07coderabbitai[bot] commented on pull request #4278: --- […] 13https://github.com/miraheze/puppet/pull/4278#issuecomment-2764567265 [13:36:22] [02puppet] 07paladox merged pull request #4278: changeprop201: increase max.poll.interval.ms to 2h for webVideoTranscode (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4278 [13:36:24] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/f6d52a7d7d4435ea4ca447343f2416d6eb111de1 [13:36:26] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03f1da2e8 13https://github.com/miraheze/puppet/commit/f1da2e8 [13:36:28] 02puppet/03master 07paladox 03f6d52a7 changeprop201: increase max.poll.interval.ms to 2h for webVideoTranscode (#4278)… [13:42:41] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 18.91, 14.48 [13:44:39] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.93, 20.07, 15.48 [13:46:38] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 22.26, 16.83 [13:48:37] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.61, 22.13, 17.42 [13:50:35] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 14.12, 19.65, 17.12 [14:05:26] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 19.79, 15.03 [14:13:11] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 12.02, 13.43, 22.81 [14:17:11] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 6.24, 9.57, 19.11 [14:23:26] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 19.34, 23.23, 22.13 [14:27:11] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 18.92, 18.77 [14:29:11] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.64, 20.30, 19.32 [14:31:11] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 16.00, 18.69, 18.86 [14:31:26] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 14.86, 18.37, 20.07 [14:32:40] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [14:32:41] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nltramswiki 58001 (running for 01:40:59) [14:32:42] mwtask161: not running [14:32:43] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=studyingfictionwiki 1 --e 2000 (running for 00:50) [14:32:44] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 30001 --e 32000 (running for 03:46) [15:16:03] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 44.01, 26.27, 17.59 [15:23:26] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.74, 18.83, 14.41 [15:26:50] PROBLEM - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:27:30] PROBLEM - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:27:30] PROBLEM - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:27:47] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:28:23] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.48, 20.73, 14.11 [15:28:55] RECOVERY - mwtask181 jobrunner-high.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.469 second response time [15:29:26] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [15:30:16] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:29] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:56] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.071 second response time [15:32:15] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.484 second response time [15:32:26] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0) [15:32:46] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [15:32:47] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nopediawiki 4001 --e 6000 (running for 02:51) [15:32:48] mwtask161: not running [15:32:49] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=subcultureswiki 8001 (running for 02:58) [15:32:50] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 34001 --e 36000 (running for 49:52) [15:33:26] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044 [15:33:31] RECOVERY - mwtask181 videoscaler.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.010 second response time [15:33:46] RECOVERY - mwtask181 jobrunner.wikitide.net HTTPS on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.012 second response time [15:34:08] PROBLEM - mwtask171 jobrunner-high.wikitide.net HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:35:50] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 8.74, 17.86, 22.68 [15:36:05] RECOVERY - mwtask171 jobrunner-high.wikitide.net HTTPS on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.199 second response time [15:38:42] !log [paladox@mwtask181] root@mwtask181:/srv/mediawiki/1.43/extensions/TimedMediaHandler/maintenance]# sudo -u www-data php requeueTranscodes.php --wiki avidwiki --force [15:38:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:39:47] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 11.47, 14.02, 19.98 [15:41:33] RECOVERY - guia.esporo.net - Cloudflare on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Sat 24 May 2025 03:21:39 PM GMT +0000. [15:42:23] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 11.78, 21.15, 20.30 [15:44:23] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 13.25, 18.34, 19.35 [15:49:39] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 27.29, 21.80, 20.36 [15:51:38] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.50, 22.63, 20.90 [15:55:35] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 24.10, 21.78 [15:57:34] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 22.66, 23.44, 21.81 [15:58:21] [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/935941fe5e37 [15:58:21] 02mw-config/03paladox-patch-4 07paladox 03935941f Increase wgFFmpegThreads to 8… [15:58:25] [02mw-config] 07paladox opened pull request #5886: Increase wgFFmpegThreads to 8 (03master...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/5886 [15:58:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 10.03, 14.13, 23.85 [15:59:17] [02mw-config] 07paladox merged pull request #5886: Increase wgFFmpegThreads to 8 (03master...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/5886 [15:59:19] [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/5cc556376e3b370f9bb5ab6152ac9bb07e5af935 [15:59:21] [02mw-config] 07paladox 04deleted 03paladox-patch-4 at 03935941f 13https://github.com/miraheze/mw-config/commit/935941f [15:59:22] miraheze/mw-config - paladox the build passed. [15:59:22] 02mw-config/03master 07paladox 035cc5563 Increase wgFFmpegThreads to 8 (#5886)… [15:59:33] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:59:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:06] miraheze/mw-config - paladox the build passed. [16:00:11] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 38s [16:00:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:01:31] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 8.86, 15.66, 19.06 [16:02:31] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 30.75, 22.37, 24.85 [16:07:53] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.18, 6.36, 5.10 [16:09:53] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 5.37, 6.11, 5.17 [16:10:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 11.93, 20.61, 23.61 [16:11:23] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.06, 19.61, 18.53 [16:13:22] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 9.53, 15.83, 17.29 [16:14:31] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.13, 21.86, 23.14 [16:17:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:18:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.58, 21.92, 23.16 [16:21:15] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 35.63, 23.49, 19.39 [16:23:53] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.83, 7.53, 6.42 [16:24:31] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.80, 12.80, 18.73 [16:31:53] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 9.12, 7.89, 6.89 [16:32:50] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [16:32:51] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nopediawiki 16001 --e 18000 (running for 03:40) [16:32:52] mwtask161: not running [16:32:53] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=surgitewiki 1 --e 2000 (running for 11:55) [16:32:54] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 36001 --e 38000 (running for 17:05) [16:41:54] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.54, 7.95, 7.47 [16:43:53] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 7.78, 8.05, 7.58 [16:48:08] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/fc4b791a6bc82f6c1d9e1795aed127422155f64b [16:48:08] 02ssl/03master 07MacFan4000 03fc4b791 add wiki.prison-mc.com [16:48:26] !log [blankeclair@mwtask171] runRefreshLinksChunked: [800/1403] Handling taleofverideawiki (start id: 0) [16:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:31] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 41.67, 28.33, 20.59 [16:53:53] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.58, 7.92, 7.79 [16:55:54] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.85, 8.43, 8.00 [16:59:53] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 6.71, 7.78, 7.86 [17:01:53] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.51, 7.88, 7.87 [17:29:51] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 18.76, 12.08 [17:31:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 17.18, 18.20, 12.71 [17:32:53] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [17:32:54] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=nopediawiki 28001 --e 30000 (running for 08:59) [17:32:55] mwtask161: not running [17:32:56] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=tanukipediawiki 8001 --e 10000 (running for 02:44) [17:32:57] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 36001 --e 38000 (running for 01:17:08) [17:45:59] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 11.47, 16.27, 23.53 [17:51:52] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.98, 12.58, 19.57 [18:13:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 30.42, 20.93, 13.37 [18:19:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 17.28, 21.67, 16.68 [18:21:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 15.86, 19.79, 16.60 [18:23:36] !log [blankeclair@mwtask151] runRefreshLinksChunked: [400/1788] Handling nvd123wiki (start id: 0) [18:23:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:42] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.63, 7.22, 7.99 [18:25:42] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 7.49, 7.49, 8.01 [18:27:42] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 5.96, 6.90, 7.73 [18:32:56] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [18:32:57] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=oblasiawiki 1 --e 2000 (running for 01:39) [18:32:58] mwtask161: not running [18:32:59] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=tanukipediawiki 20001 --e 22000 (running for 02:49) [18:33:00] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 36001 --e 38000 (running for 02:17:11) [18:33:42] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.38, 7.35, 7.60 [18:35:50] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 14.25, 16.62, 22.80 [18:37:42] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.02, 7.29, 7.54 [18:41:42] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 7.61, 7.59 [18:41:44] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 13.41, 14.48, 19.93 [18:43:42] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.69, 7.62, 7.60 [18:45:38] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.86, 21.09, 21.28 [18:57:42] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 8.50, 7.65, 7.51 [19:02:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:03:42] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 5.01, 6.93, 7.34 [19:09:42] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.16, 6.02, 6.76 [19:13:13] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.35, 20.26, 14.53 [19:25:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 8.19, 10.87, 23.50 [19:33:00] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [19:33:01] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=orainmetawiki 1 --e 2000 (running for 07:39) [19:33:02] mwtask161: not running [19:33:03] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=ted52wiki 1 --e 2000 (running for 07:18) [19:33:04] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=makoentertainmentwiki 38001 --e 40000 (running for 45:59) [19:35:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 12.39, 13.47, 19.27 [19:37:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:10:13] !log [blankeclair@mwtask171] runRefreshLinksChunked: [900/1403] Handling thegreatwarwiki (start id: 0) [20:10:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:33:03] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [20:33:04] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=osdevwiki 1 --e 2000 (running for 05:57) [20:33:05] mwtask161: not running [20:33:06] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=thevaultwiki 2001 (running for 00:00) [20:33:07] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=mappingwiki 1 --e 2000 (running for 01:51) [20:38:39] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:51] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.701 second response time [20:48:26] !log [blankeclair@mwtask171] runRefreshLinksChunked: [1000/1403] Handling tomwikiwiki (start id: 0) [20:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:01] PROBLEM - mwtask151 jobrunner-high.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:02:02] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:04:00] RECOVERY - mwtask151 jobrunner-high.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.127 second response time [21:04:02] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 0.742 second response time [21:04:11] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 28.65, 17.95, 13.60 [21:04:30] !log [blankeclair@mwtask181] runRefreshLinksChunked: [1100/2761] Handling mcmodhofwiki (start id: 0) [21:04:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:08] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 12.11, 16.13, 13.53 [21:17:57] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 29.51, 19.55, 14.79 [21:21:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.64, 23.74, 17.79 [21:23:39] so that ATT run was likely taken down w/ mwtask161 [21:24:57] !log [blankeclair@mwtask161] runRefreshLinksChunked: [0/2] Handling allthetropeswiki (start id: 326000) [21:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:25:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.54, 19.95, 17.50 [21:33:07] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [21:33:08] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=pacopediawiki 1 --e 2000 (running for 05:08) [21:33:09] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 08:08) [21:33:10] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=totaltenniswiki 4001 --e 6000 (running for 03:35) [21:33:11] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=meaastawiki 2001 --e 4000 (running for 06:44) [21:35:59] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:35:59] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:47] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 47.03, 32.07, 23.18 [21:37:46] PROBLEM - mwtask151 videoscaler.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:37:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 20.50, 19.42, 15.65 [21:37:59] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3972 bytes in 1.717 second response time [21:38:00] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.055 second response time [21:39:17] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49887MiB (10% inode=99%); [21:39:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 18.33, 19.23, 16.06 [21:39:51] RECOVERY - mwtask151 videoscaler.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.552 second response time [21:52:20] [Grafana] FIRING: The mediawiki job queue has more than 20000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:10:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 10.04, 13.56, 23.70 [22:16:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.20, 10.44, 19.26 [22:26:31] PROBLEM - mwtask151 jobrunner.wikitide.net HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:28:40] RECOVERY - mwtask151 jobrunner.wikitide.net HTTPS on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.922 second response time [22:33:10] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [22:33:11] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=palacetesdobrasilwiki 1 --e 2000 (running for 51:58) [22:33:12] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 01:08:11) [22:33:13] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=totaltenniswiki 12001 --e 14000 (running for 00:21) [22:33:14] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=megamanwiki 64001 --e 66000 (running for 00:42) [22:36:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.48, 17.44, 16.45 [22:38:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.13, 21.77, 18.16 [22:40:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.81, 21.26, 18.45 [22:46:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 22.34, 19.82 [22:51:07] !log [blankeclair@mwtask151] runRefreshLinksChunked: [500/1788] Handling pandorastalewiki (start id: 0) [22:51:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:10] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:49] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.98, 18.32, 13.41 [22:56:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.29, 23.69, 22.63 [22:57:26] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.179 second response time [22:57:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.56, 18.93, 14.21 [22:59:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 11.12, 16.37, 13.89 [23:00:58] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.03, 17.28, 20.26 [23:18:58] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 19.80, 17.96 [23:33:14] * MirahezeLSBot is checking the other mwtask* servers on refreshLinks [23:33:15] mwtask151: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=pgwiki 1 --e 2000 (running for 15:32) [23:33:16] mwtask161: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=allthetropeswiki 326001 (running for 02:08:15) [23:33:17] mwtask171: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=tristarwiki 1 --e 2000 (running for 00:27) [23:33:18] mwtask181: sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php refreshLinks --wiki=metawiki 1 --e 2000 (running for 04:55) [23:35:57] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 6.54, 11.64, 23.77 [23:37:49] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.25, 18.18, 13.57 [23:39:49] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 18.79, 17.56, 13.86 [23:41:57] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 10.74, 11.17, 19.72