[00:07:27] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57200MiB (12% inode=99%); [00:07:33] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57024MiB (12% inode=99%); [00:08:13] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56204MiB (12% inode=99%); [01:06:13] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php createAndPromote --wiki=candelabremwiki Skye --sysop --force (END - exit=0) [01:48:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw191 mw203 [01:48:27] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 61.49, 25.16, 10.50 [01:49:32] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw171 mw172 mw163 [01:49:48] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw172 mw163 [01:49:52] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw172 mw163 [01:50:17] uhoh [01:51:52] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [01:52:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [01:52:34] you cant just die for like a second like that wdym [01:53:30] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [01:53:48] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [01:56:03] Now if it the whole farm goes down… [01:56:27] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.96, 10.15, 10.30 [01:58:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.81, 7.03, 9.13 [06:04:17] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.75, 19.19, 14.48 [06:06:17] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.46, 19.08, 15.01 [06:20:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756531220000&orgId=1&to=1756534856679 [06:32:06] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.08, 20.50, 19.32 [06:40:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756536056682[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532300000&orgId=1&to=1756535900000 [06:42:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.98, 20.11, 19.79 [06:47:58] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.65, 21.04, 20.29 [08:00:13] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49836MiB (10% inode=99%); [08:17:16] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.97, 16.22, 19.11 [08:31:09] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.26, 20.08, 19.37 [08:57:33] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49935MiB (10% inode=99%); [09:10:50] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.05, 19.61, 20.18 [09:19:27] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49863MiB (10% inode=99%); [09:30:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756546256708[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756542620000&orgId=1&to=1756546256708 [09:47:12] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:47:44] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:47:57] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:48:34] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:02] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:49:43] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:49:43] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:21] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:51:22] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:51:36] PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:51:36] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:52:01] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:52:06] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:52:37] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:52:50] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:53:19] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.856 second response time [09:53:23] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.592 second response time [09:53:26] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:35] RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.619 second response time [09:53:49] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.388 second response time [09:53:49] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.325 second response time [09:53:55] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time [09:54:02] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.270 second response time [09:54:41] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 5.210 second response time [09:54:45] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.080 second response time [09:55:17] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 4.697 second response time [09:55:25] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.585 second response time [09:55:45] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.753 second response time [09:56:05] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.139 second response time [09:56:20] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.302 second response time [09:56:53] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.792 second response time [09:57:26] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.112 second response time [10:00:43] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:01:15] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:45] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.619 second response time [10:03:16] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.024 second response time [10:07:52] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:59] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.986 second response time [10:12:52] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:13:03] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:13:21] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:13:33] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:34] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:13:43] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:53] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:15:15] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.009 second response time [10:16:01] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:16:04] PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:16:17] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:16:59] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.998 second response time [10:17:38] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.991 second response time [10:17:46] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 6.174 second response time [10:19:17] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.288 second response time [10:21:08] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:21:31] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:22:35] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:22:53] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:23:01] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [10:23:09] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.976 second response time [10:23:12] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:12] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:24:11] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.265 second response time [10:24:12] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.103 second response time [10:26:12] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:26:40] RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.589 second response time [10:26:58] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.365 second response time [10:27:50] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 8.563 second response time [10:28:46] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 3.249 second response time [10:28:48] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time [10:29:15] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.755 second response time [10:29:22] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.965 second response time [10:30:30] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 9.234 second response time [10:35:28] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:35:40] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:35:46] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:35:46] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:35:54] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:36:32] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:28] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.304 second response time [10:40:43] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:41:11] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:41:51] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.878 second response time [10:41:55] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.363 second response time [10:42:00] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.343 second response time [10:42:19] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:42:37] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time [10:43:06] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:43:28] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:37] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:43:44] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:44:10] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:46:10] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:46:16] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:46:24] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:47:33] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.301 second response time [10:48:12] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time [10:48:29] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.286 second response time [10:49:28] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 5.120 second response time [10:49:43] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.176 second response time [10:49:44] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.055 second response time [10:50:10] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.453 second response time [10:50:22] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.722 second response time [10:50:34] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 3.343 second response time [10:50:39] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.007 second response time [10:50:50] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:51:11] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.116 second response time [10:51:26] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:35] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:51:55] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:51:56] PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:54:03] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [10:54:15] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [10:55:57] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.271 second response time [10:56:01] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.728 second response time [10:56:06] RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.544 second response time [10:56:56] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.253 second response time [10:56:56] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:57:46] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 7.053 second response time [10:57:48] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.585 second response time [10:58:56] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [11:09:08] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:09:13] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [11:10:01] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:10:14] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:10:23] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:10:41] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:44] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:11:12] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.190 second response time [11:11:14] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 8.778 second response time [11:12:00] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 2.685 second response time [11:12:15] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 7.611 second response time [11:12:19] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.270 second response time [11:12:42] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 4375 bytes in 5.324 second response time [11:12:45] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.608 second response time [11:15:38] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:36] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:16:36] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [11:17:03] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:17:26] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:20:42] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [11:20:43] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.332 second response time [11:21:15] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.796 second response time [11:21:24] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.349 second response time [11:21:50] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.338 second response time [11:35:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756553756729[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756549880000&orgId=1&to=1756553480000 [11:45:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756554356728[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756550480000&orgId=1&to=1756554356728[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https [11:45:57] .wikitide.net/d/GtxbP1Xnk?from=1756549880000&orgId=1&to=1756553480000 [13:09:29] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=falcomwiki /home/reception/kiseki --search-recursively --summary=Imported from https://kiseki.fandom.com (START) [13:09:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:38:50] [02mw-config] 07Reception123 merged pull request #6077: remove centralauth-unmerge from stewards (03main...03Reception123-patch-2) 13https://github.com/miraheze/mw-config/pull/6077 [14:38:50] [02mw-config] 07Reception123 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/965b51c5300919f4100bbf8b2087bb400f462ee1 [14:38:50] 02mw-config/03main 07Reception123 03965b51c remove centralauth-unmerge from stewards (#6077)… [14:38:51] [02mw-config] 07Reception123 04deleted 03Reception123-patch-2 at 03e365ac2 13https://api.github.com/repos/miraheze/mw-config/commit/e365ac2 [14:39:46] miraheze/mw-config - Reception123 the build passed. [14:39:53] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:39:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:32] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 38s [14:40:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:20:47] [02mw-config] 07Reception123 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/f9d19b3b45a656a054b2f0c61d0a43b798eaba1e [15:20:47] 02mw-config/03main 07Reception123 03f9d19b3 remove self from RequestCD requests … [15:21:45] miraheze/mw-config - Reception123 the build passed. [15:25:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/9a39bf83726a14239db5538b59ebd9cde331727c [15:25:29] 02ssl/03main 07WikiTideBot 039a39bf8 Bot: Auto-update domain lists [15:25:57] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756567556763[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756563680000&orgId=1&to=1756567280000 [15:35:36] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/2806595ec741e59f4b7131ed65cc9037e64f8e74 [15:35:36] 02ssl/03main 07WikiTideBot 032806595 Bot: Auto-update domain lists [15:40:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756568456767[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756564760000&orgId=1&to=1756568456767 [16:07:48] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importImages.php --wiki=falcomwiki /home/reception/kiseki --search-recursively --summary=Imported from https://kiseki.fandom.com (END - exit=0) [17:29:41] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.37, 19.83, 18.03 [17:37:41] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 16.04, 19.49, 18.94 [18:12:03] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.14, 20.29, 19.21 [18:13:09] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=apstversewiki --new=apstracktwiki --user=Skye (END - exit=0) [18:13:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.10, 19.75, 19.14 [18:17:30] ooh finally something with leftovers [18:31:25] just deleted images on the old one so bye bye [18:36:43] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=apstracktwiki (END - exit=0) [18:57:34] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=millenniafulminationwiki --new=tuagwiki --user=Skye (END - exit=0) [18:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:48] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=tuagwiki (END - exit=0) [19:40:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756582856800[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756564760000&orgId=1&to=1756582856800 [19:43:08] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.90, 11.99, 8.51 [19:45:04] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.72, 8.45, 7.63 [19:56:42] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 21.52, 13.55, 9.25 [19:58:57] [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6076: Update justinrainbow/json-schema requirement from 6.4.2 to 6.5.1 (03main...03dependabot/composer/justinrainbow/json-schema-6.5.1) 13https://github.com/miraheze/mw-config/pull/6076 [19:58:58] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/829c73517d6ce0a2eaa223ed10ac159528a54534 [19:58:58] 02mw-config/03main 07dependabot[bot] 03829c735 Update justinrainbow/json-schema requirement from 6.4.2 to 6.5.1 (#6076)… [19:58:58] [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/justinrainbow/json-schema-6.5.1 at 035784467 13https://api.github.com/repos/miraheze/mw-config/commit/5784467 [19:59:54] miraheze/mw-config - Universal-Omega the build passed. [19:59:56] [02mw-config] 07dependabot[bot] 04force-pushed 03dependabot/composer/phpunit/phpunit-12.3.7 from 03f3571ab to 035deddf4 13https://api.github.com/repos/miraheze/mw-config/commits/dependabot/composer/phpunit/phpunit-12.3.7 [19:59:56] 02mw-config/03dependabot/composer/phpunit/phpunit-12.3.7 07dependabot[bot] 035deddf4 Update phpunit/phpunit requirement from 12.3.6 to 12.3.7… [20:00:35] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.35, 11.47, 9.43 [20:00:47] miraheze/mw-config - dependabot[bot] the build passed. [20:01:21] [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6075: Update phpunit/phpunit requirement from 12.3.6 to 12.3.7 (03main...03dependabot/composer/phpunit/phpunit-12.3.7) 13https://github.com/miraheze/mw-config/pull/6075 [20:01:21] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/9032b47e4195bde1a5fe5c9c7b6db6c3d9c454cd [20:01:23] 02mw-config/03main 07dependabot[bot] 039032b47 Update phpunit/phpunit requirement from 12.3.6 to 12.3.7 (#6075)… [20:01:25] [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/phpunit/phpunit-12.3.7 at 035deddf4 13https://api.github.com/repos/miraheze/mw-config/commit/5deddf4 [20:02:19] miraheze/mw-config - Universal-Omega the build passed. [20:02:32] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.40, 14.95, 10.96 [20:03:52] [02mw-config] 07Universal-Omega merged pull request #6074: Add an isBeta() method to MirahezeFunctions (03main...03isBeta) 13https://github.com/miraheze/mw-config/pull/6074 [20:03:52] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b95e8ee3e6c1d2f351d155a3687bfa00dac3f338 [20:03:52] 02mw-config/03main 07CosmicAlpha 03b95e8ee Add an isBeta() method to MirahezeFunctions (#6074)… [20:03:53] [02mw-config] 07Universal-Omega 04deleted 03isBeta at 03f7907ea 13https://api.github.com/repos/miraheze/mw-config/commit/f7907ea [20:04:45] miraheze/mw-config - Universal-Omega the build passed. [20:04:51] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:04:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:23] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw183 [20:05:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:33] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:05:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:18] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw202 [20:06:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:23] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:06:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:57] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 34s [20:07:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:27] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.98, 3.48, 11.12 [20:34:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.30, 2.73, 9.92 [20:55:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756587356813[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756583540000&orgId=1&to=1756587140000 [20:56:17] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.18, 20.13, 18.93 [20:58:17] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.20, 20.07, 19.06 [21:18:17] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.79, 20.73, 20.17 [21:22:17] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.80, 20.37, 20.15 [21:45:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756590356820[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756586660000&orgId=1&to=1756590356820 [23:05:36] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a4017c70721cbf75ebff4749ce54029e1b113ba6 [23:05:36] 02ssl/03main 07WikiTideBot 03a4017c7 Bot: Auto-update domain lists