[00:00:08] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to mwtask181 [00:00:09] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to mwtask181 - SUCCESS in 0s [00:00:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:38] miraheze/ErrorPages - SomeMWDev the build passed. [00:00:42] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to mwtask181 [00:00:43] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to mwtask181 - SUCCESS in 0s [00:00:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:47] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to mw183 [00:00:48] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to mw183 - SUCCESS in 1s [00:00:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:38] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/1265a7b9cc7f66d10b164444b747c3167e5e07e2 [00:01:40] 02statichelp/03main 07WikiTideBot 031265a7b Bot: Auto-update Tech namespace pages 2025-11-17 00:01:36 [00:18:01] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49660MiB (10% inode=99%); [00:20:01] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56090MiB (12% inode=99%); [00:26:43] [02ErrorPages] 07SomeMWDev 04force-pushed 03sync-managewiki-cache from 035bf8ada to 0317a2389 13https://api.github.com/repos/miraheze/ErrorPages/commits/sync-managewiki-cache [00:27:17] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to mwtask181 [00:27:18] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to mwtask181 - SUCCESS in 0s [00:27:21] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to mw183 [00:27:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:23] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to mw183 - SUCCESS in 1s [00:27:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:34] miraheze/ErrorPages - SomeMWDev the build passed. [00:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:44] [02mw-config] 07SomeMWDev created 03deleted-wiki-handling (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0a03cb6f9a16 [00:48:44] 02mw-config/03deleted-wiki-handling 07SomeRandomDeveloper 030a03cb6 Call wfHandleDeletedWiki in BeforeInitialize [00:48:56] [02mw-config] 07SomeMWDev opened pull request #6182: Call wfHandleDeletedWiki in BeforeInitialize (03main...03deleted-wiki-handling) 13https://github.com/miraheze/mw-config/pull/6182 [00:49:51] miraheze/mw-config - SomeMWDev the build passed. [00:50:02] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [00:50:03] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:50:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:22] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CreateWiki:DeleteWiki --wiki=loginwikibeta --delete --deletewiki t14400test1wikibeta (END - exit=0) [00:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:47] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to mwtask181 [00:54:48] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True} to mwtask181 - SUCCESS in 0s [00:54:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:53] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to mw183 [00:54:54] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True} to mw183 - SUCCESS in 1s [00:54:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:06] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to mwtask181 [00:58:07] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True} to mwtask181 - SUCCESS in 0s [00:58:09] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True} to mw183 [00:58:11] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True} to mw183 - SUCCESS in 1s [00:58:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:02] [02ErrorPages] 07SomeMWDev pushed 1 new commit to 03sync-managewiki-cache 13https://github.com/miraheze/ErrorPages/commit/4d445a824b177ab355b380ca230b61be8ca51851 [00:59:03] 02ErrorPages/03sync-managewiki-cache 07SomeRandomDeveloper 034d445a8 Don't call allowGlobalInstance [00:59:46] miraheze/ErrorPages - SomeMWDev the build passed. [01:00:53] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'errorpages': True} to mwtask181 [01:00:54] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'errorpages': True} to mwtask181 - SUCCESS in 0s [01:00:57] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'errorpages': True} to mw183 [01:00:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:59] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'errorpages': True} to mw183 - SUCCESS in 2s [01:01:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:28] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'errorpages': True} to mwtask181 [01:03:29] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'errorpages': True} to mwtask181 - SUCCESS in 0s [01:03:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:33] !log [somerandomdeveloper@mwtask181] starting deploy of {'config': True, 'errorpages': True} to mw183 [01:03:35] !log [somerandomdeveloper@mwtask181] finished deploy of {'config': True, 'errorpages': True} to mw183 - SUCCESS in 2s [01:03:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:39] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [01:36:39] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [01:54:01] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:54:03] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:06] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:08] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:54:08] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:09] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:54:09] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [01:54:12] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:15] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:19] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:23] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:54:24] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:54:24] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:24] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [01:54:26] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:54:26] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:54:27] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 91.89, 42.30, 17.40 [01:54:34] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:36] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:40] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:54:40] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:54:46] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:54:48] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:54:57] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:54:58] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [01:54:59] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:55:00] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:55:03] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:55:04] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:55:04] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [01:55:23] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:26] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:55:28] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:55:33] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [01:55:37] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:55:37] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [01:55:38] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:55:39] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:55:41] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:41] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:55:43] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:55:48] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:55:53] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:54] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:55:58] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [01:57:15] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: [01:58:37] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.122 second response time [01:58:37] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.567 second response time [01:58:39] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [01:58:42] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.046 second response time [01:58:48] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.189 second response time [01:58:53] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.147 second response time [01:58:54] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [01:58:57] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [01:58:58] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.182 second response time [01:59:03] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.048 second response time [01:59:03] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 4.174 second response time [01:59:05] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [01:59:09] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.065 second response time [01:59:12] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 6.4%Current connections: 64 [01:59:12] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.201 second response time [01:59:15] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.527 second response time [01:59:21] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.063 second response time [01:59:26] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4328 bytes in 0.069 second response time [01:59:30] kill all problems [01:59:30] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time [01:59:31] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time [01:59:31] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [01:59:33] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 24% [01:59:33] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.059 second response time [01:59:35] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [01:59:37] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [01:59:37] Let's recovery [01:59:40] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.048 second response time [01:59:43] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.169 second response time [01:59:43] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.058 second response time [01:59:46] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.059 second response time [01:59:58] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [01:59:59] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.067 second response time [02:00:01] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [02:00:06] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.062 second response time [02:00:09] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [02:00:09] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [02:00:09] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time [02:00:10] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.060 second response time [02:00:12] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.050 second response time [02:00:19] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.054 second response time [02:00:25] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [02:00:29] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.055 second response time [02:00:32] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.214 second response time [02:00:32] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.066 second response time [02:00:36] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.073 second response time [02:00:40] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.062 second response time [02:00:42] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [02:06:39] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [02:22:01] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49735MiB (10% inode=99%); [02:32:26] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.53, 1.95, 11.71 [02:36:26] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.37, 1.70, 9.38 [02:47:36] Hello? [02:49:31] So for a week on All the Tropes, on large pages, the history has not been working correctly [02:50:21] It keeps having everything in a single straight line, which one of the moderators said was due to a slow connection [02:51:29] I know it isin't anything on my end, because I cleared the cache, and checked the internet at home and its all fine, so I know it is something wrong on the site itself [03:08:02] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.14, 16.48, 10.05 [03:10:02] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.98, 19.16, 11.84 [03:12:02] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 20.37, 13.14 [03:12:45] I don't understand [03:15:03] Link to example page? [03:15:40] https://allthetropes.org/wiki/File:Screenshot_2025-11-12_173450.png [03:15:56] On every single page, it is like that [03:16:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.95, 20.40, 14.98 [03:16:47] Its been like that for a week, and I know there isin't an issue on my end [03:18:19] I cleared the cache and the internet where I am is fine [03:24:15] So I know it is something involving the site itself [03:25:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/960a73f55df9b25c9e09ce945494e23ba14509bc [03:25:18] 02ssl/03main 07WikiTideBot 03960a73f Bot: Auto-update domain lists [03:30:02] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.41, 19.85, 15.74 [03:44:02] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 16.61, 22.74, 21.02 [03:46:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.00, 18.56, 19.69 [03:51:25] You're missing some CSS. I'm not sure why. The usual advice is to switch browser/device to see what happens. Create a [[Phorge]] task if the problem persists and include a screenshot of the "console" and "network" tab of the browser's developer tools. [03:51:26]

[03:57:51] Anyhow i have to go [03:58:04] Goodbye [08:35:17] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/46bc9f897ea2de2d9001871fdbf9f69ae87329d9 [08:35:17] 02ssl/03main 07WikiTideBot 0346bc9f8 Bot: Auto-update domain lists [11:42:44] [02mw-config] 07SomeMWDev merged pull request #6182: Call wfHandleDeletedWiki in BeforeInitialize (03main...03deleted-wiki-handling) 13https://github.com/miraheze/mw-config/pull/6182 [11:42:44] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a7f656521f36247842b0a0fd23def47d522f74ef [11:42:46] 02mw-config/03main 07SomeRandomDeveloper 03a7f6565 Call wfHandleDeletedWiki in BeforeInitialize (#6182)… [11:42:48] [02mw-config] 07SomeMWDev 04deleted 03deleted-wiki-handling at 030a03cb6 13https://api.github.com/repos/miraheze/mw-config/commit/0a03cb6 [11:42:50] [02ErrorPages] 07SomeMWDev merged pull request #54: Sync ManageWiki cache for deleted wikis (03main...03sync-managewiki-cache) 13https://github.com/miraheze/ErrorPages/pull/54 [11:42:51] [02ErrorPages] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/ErrorPages/commit/36aba4f862f687adcfae5805e10905b74ca55ca7 [11:42:52] 02ErrorPages/03main 07SomeRandomDeveloper 0336aba4f Sync ManageWiki cache for deleted wikis (#54)… [11:42:54] [02ErrorPages] 07SomeMWDev 04deleted 03sync-managewiki-cache at 034d445a8 13https://api.github.com/repos/miraheze/ErrorPages/commit/4d445a8 [11:43:06] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config,errorpages', 'config': True, 'errorpages': True} to test151 [11:43:07] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config,errorpages', 'config': True, 'errorpages': True} to test151 - SUCCESS in 1s [11:43:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:17] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config,errorpages', 'config': True, 'errorpages': True} to all [11:43:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:33] miraheze/ErrorPages - SomeMWDev the build passed. [11:43:40] miraheze/mw-config - SomeMWDev the build passed. [11:44:01] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config,errorpages', 'config': True, 'errorpages': True} to all - SUCCESS in 44s [11:44:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:20:50] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/a31bb6f00e5dde01a37a8ea048a8150b31ada6c5 [12:20:50] 02CreateWiki/03main 07translatewiki.net 03a31bb6f Localisation updates from https://translatewiki.net. [12:20:52] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/063687096f8870a7cdcd7a007c05ccb296d5bf08 [12:20:52] 02DiscordNotifications/03main 07translatewiki.net 030636870 Localisation updates from https://translatewiki.net. [12:20:54] [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/13e76c3bda5dac81b69b84291bbfaa917d612944 [12:20:56] [02DataDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/6214ce921d1114704e0d3382a959f8447767e604 [12:20:58] [02IncidentReporting] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/b4a5152d3da0c2f07cc63c3d5d7cbe81faffc3a1 [12:20:59] 02ImportDump/03main 07translatewiki.net 0313e76c3 Localisation updates from https://translatewiki.net. [12:21:01] [02ErrorPages] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ErrorPages/commit/76f85c4d95d1f70d8d9774b2b566a0e80f9f34b4 [12:21:03] 02DataDump/03main 07translatewiki.net 036214ce9 Localisation updates from https://translatewiki.net. [12:21:05] [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/71fd513c3f10dac314a344c70a7f309ab7b28f87 [12:21:07] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/a4aa4ec4ab1299f8bfef7074722c04c71db630aa [12:21:08] 02IncidentReporting/03main 07translatewiki.net 03b4a5152 Localisation updates from https://translatewiki.net. [12:21:10] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/1315233283142b5d4d574105e752a775777ee5c3 [12:21:12] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/33ab387188938930373daac46226b6902058366e [12:21:14] 02ErrorPages/03main 07translatewiki.net 0376f85c4 Localisation updates from https://translatewiki.net. [12:21:16] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/7b28ab89739f9d9f46836e99c8d60b251ea4e81d [12:21:18] 02landing/03main 07translatewiki.net 0371fd513 Localisation updates from https://translatewiki.net. [12:21:19] 02MatomoAnalytics/03main 07translatewiki.net 03a4aa4ec Localisation updates from https://translatewiki.net. [12:21:21] 02MirahezeMagic/03main 07translatewiki.net 031315233 Localisation updates from https://translatewiki.net. [12:21:22] 02ManageWiki/03main 07translatewiki.net 0333ab387 Localisation updates from https://translatewiki.net. [12:21:24] 02WikiDiscover/03main 07translatewiki.net 037b28ab8 Localisation updates from https://translatewiki.net. [12:27:03] miraheze/DataDump - translatewiki the build passed. [12:29:41] miraheze/IncidentReporting - translatewiki the build passed. [12:30:14] miraheze/ErrorPages - translatewiki the build passed. [12:30:26] miraheze/landing - translatewiki the build passed. [12:34:22] miraheze/MatomoAnalytics - translatewiki the build passed. [12:35:21] miraheze/WikiDiscover - translatewiki the build passed. [12:35:52] miraheze/ManageWiki - translatewiki the build passed. [12:36:53] miraheze/MirahezeMagic - translatewiki the build passed. [12:40:23] miraheze/CreateWiki - translatewiki the build passed. [12:41:28] miraheze/ImportDump - translatewiki the build passed. [12:45:13] miraheze/DiscordNotifications - translatewiki the build passed. [12:59:33] [02mw-config] 07SomeMWDev created 03T14392 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/5a41d6f2068c [12:59:33] 02mw-config/03T14392 07SomeRandomDeveloper 035a41d6f T14392: Add $wgPageImagesOpenGraph to ManageWiki [12:59:50] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [12:59:51] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [12:59:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:59:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:00:01] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27177MiB (5% inode=99%); [13:02:30] [02mw-config] 07SomeMWDev opened pull request #6183: T14392: Add $wgPageImagesOpenGraph to ManageWiki (03main...03T14392) 13https://github.com/miraheze/mw-config/pull/6183 [13:03:23] miraheze/mw-config - SomeMWDev the build passed. [13:03:31] [02mw-config] 07SomeMWDev merged pull request #6183: T14392: Add $wgPageImagesOpenGraph to ManageWiki (03main...03T14392) 13https://github.com/miraheze/mw-config/pull/6183 [13:03:33] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/3f811a0732209f987e7bba7deb29ff6e4b1bc208 [13:03:35] [02mw-config] 07SomeMWDev 04deleted 03T14392 at 035a41d6f 13https://api.github.com/repos/miraheze/mw-config/commit/5a41d6f [13:03:36] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [13:03:37] 02mw-config/03main 07SomeRandomDeveloper 033f811a0 T14392: Add $wgPageImagesOpenGraph to ManageWiki (#6183) [13:03:37] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [13:03:39] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [13:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:04:01] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [13:04:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:04:25] miraheze/mw-config - SomeMWDev the build passed. [13:17:09] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49900MiB (10% inode=99%); [13:43:20] [02CreateWiki] 07SomeMWDev merged 07dependabot[bot]'s pull request #791: Bump eslint-config-wikimedia from 0.32.0 to 0.32.1 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.1) 13https://github.com/miraheze/CreateWiki/pull/791 [13:43:20] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/191b0032ac2c13829c5538d68128c37acb860ff0 [13:43:20] 02CreateWiki/03main 07dependabot[bot] 03191b003 Bump eslint-config-wikimedia from 0.32.0 to 0.32.1 (#791)… [13:43:21] [02CreateWiki] 07SomeMWDev 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.1 at 032e8300d 13https://api.github.com/repos/miraheze/CreateWiki/commit/2e8300d [13:43:42] [02ManageWiki] 07SomeMWDev merged 07dependabot[bot]'s pull request #751: Bump eslint-config-wikimedia from 0.32.0 to 0.32.1 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.1) 13https://github.com/miraheze/ManageWiki/pull/751 [13:43:43] [02ManageWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/5d52d61c35aa2224ccf63bfe9e02b7042f9fcfa4 [13:43:43] 02ManageWiki/03main 07dependabot[bot] 035d52d61 Bump eslint-config-wikimedia from 0.32.0 to 0.32.1 (#751)… [13:43:44] [02ManageWiki] 07SomeMWDev 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.1 at 03bfb5a51 13https://api.github.com/repos/miraheze/ManageWiki/commit/bfb5a51 [13:50:03] miraheze/ManageWiki - SomeMWDev the build passed. [13:55:16] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49823MiB (10% inode=99%); [14:04:44] miraheze/CreateWiki - SomeMWDev the build passed. [15:46:08] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/656169787669 [15:46:08] 02puppet/03paladox-patch-6 07paladox 036561697 mariadb: Increase thread cache size from 100 to 300… [15:46:13] [02puppet] 07paladox opened pull request #4607: mariadb: Increase thread cache size from 100 to 300 (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4607 [15:46:20] [02puppet] 07coderabbitai[bot] commented on pull request #4607:

[…] 13https://github.com/miraheze/puppet/pull/4607#issuecomment-3542556766 [15:46:22] [02puppet] 07paladox merged pull request #4607: mariadb: Increase thread cache size from 100 to 300 (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4607 [15:46:22] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/17f34bd61483cc933435455a7827902be3961eff [15:46:24] 02puppet/03main 07paladox 0317f34bd mariadb: Increase thread cache size from 100 to 300 (#4607)… [15:46:26] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 036561697 13https://api.github.com/repos/miraheze/puppet/commit/6561697 [15:46:39] !log db*: set global thread_cache_size = 300; [15:46:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:30] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/3cfe11b7784b [16:24:30] 02puppet/03paladox-patch-6 07paladox 033cfe11b mariadb: Increase max_connect_errors to 1 billion… [16:24:33] [02puppet] 07paladox opened pull request #4608: mariadb: Increase max_connect_errors to 1 billion (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4608 [16:24:39] [02puppet] 07coderabbitai[bot] commented on pull request #4608:

[…] 13https://github.com/miraheze/puppet/pull/4608#issuecomment-3542766100 [16:24:42] [02puppet] 07paladox merged pull request #4608: mariadb: Increase max_connect_errors to 1 billion (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4608 [16:24:42] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b83bd702d0b4911e810b92d1f15c977e6d1a081e [16:24:42] 02puppet/03main 07paladox 03b83bd70 mariadb: Increase max_connect_errors to 1 billion (#4608)… [16:24:44] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 033cfe11b 13https://api.github.com/repos/miraheze/puppet/commit/3cfe11b [16:25:02] !log db*: set global max_connect_errors = 1000000000; [16:25:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:34:05] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2366bd1fcfb6 [16:34:05] 02puppet/03paladox-patch-6 07paladox 032366bd1 mariadb: Set use-stat-tables to preferably… [16:34:11] [02puppet] 07paladox opened pull request #4609: mariadb: Set use-stat-tables to preferably (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4609 [16:34:20] [02puppet] 07coderabbitai[bot] commented on pull request #4609:

[…] 13https://github.com/miraheze/puppet/pull/4609#issuecomment-3542813893 [16:34:56] !log db*: SET GLOBAL use_stat_tables = 'preferably'; [16:35:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:16] [02puppet] 07paladox merged pull request #4609: mariadb: Set use-stat-tables to preferably (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4609 [16:35:16] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c1381a1f78daba3648bf1e10ae756165b1c12da0 [16:35:17] 02puppet/03main 07paladox 03c1381a1 mariadb: Set use-stat-tables to preferably (#4609)… [16:35:18] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 032366bd1 13https://api.github.com/repos/miraheze/puppet/commit/2366bd1 [16:51:33] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:35] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:36] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:37] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:51:37] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:43] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:51:43] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:46] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:49] No [16:51:52] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:52] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:51:52] No [16:51:53] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:51:57] You're not supposed to do that [16:51:58] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [16:51:59] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:52:01] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:52:03] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:06] what did you do [16:52:06] urgh, what's going on this time? [16:52:08] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [16:52:08] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:09] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [16:52:09] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [16:52:12] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:15] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:19] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:23] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:52:24] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [16:52:24] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:25] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.016 second response time [16:52:26] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:52:27] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 119.12, 57.42, 23.36 [16:52:34] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:36] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:40] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:52:40] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:52:50] idt the changes paladox did on puppet 15min ago should be doing this [16:52:58] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:59] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:53:07] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:08] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:09] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:53:13] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:53:14] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:14] yeah, I don't really se how it could be related [16:53:14] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:23] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:26] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:53:27] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:53:33] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [16:53:37] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [16:55:02] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:55:15] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: [16:55:37] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.070 second response time [16:55:37] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.353 second response time [16:55:38] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [16:55:39] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.067 second response time [16:55:42] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.069 second response time [16:55:48] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.341 second response time [16:55:48] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.061 second response time [16:55:54] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.066 second response time [16:56:01] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.344 second response time [16:56:03] that's an evil message [16:56:07]