[00:01:22] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/377021367d1d425d593fa0eb855816491cfc515d [00:01:22] 02statichelp/03main 07WikiTideBot 033770213 Bot: Auto-update Tech namespace pages 2025-10-31 00:01:20 [00:12:22] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73684MiB (16% inode=99%); [00:14:12] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 73262MiB (16% inode=99%); [00:14:32] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74055MiB (16% inode=99%); [00:15:30] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b4a78a6b749792f2c73b6ef024d6ef5ea795c644 [00:15:30] 02ssl/03main 07WikiTideBot 03b4a78a6 Bot: Auto-update domain lists [04:35:27] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c5872fb6adbe20aa0807e536352c174d680cf2e3 [04:35:27] 02ssl/03main 07WikiTideBot 03c5872fb Bot: Auto-update domain lists [08:47:27] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:47:31] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:31] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:34] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:47:36] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:37] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:37] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:38] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:42] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:43] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:46] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:49] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:47:53] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:47:59] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:48:07] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:11] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:12] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:48:14] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:17] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:48:17] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:23] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:48:34] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:35] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:37] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:48:42] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:48:43] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:48:51] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [08:48:53] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.017 second response time [08:48:55] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:48:56] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:48:58] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:02] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:03] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:06] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:49:07] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.014 second response time [08:49:07] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [08:49:08] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:49:12] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:12] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:49:12] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 282.39, 122.68, 49.78 [08:49:22] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:49:33] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [08:49:52] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [08:50:25] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:51:37] PROBLEM - db161 MariaDB on db161 is UNKNOWN: [08:52:22] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: [08:52:25] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 7.906 second response time [08:52:32] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.004 second response time [08:52:41] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 6.144 second response time [08:52:43] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 6.251 second response time [08:53:03] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.581 second response time [08:53:21] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:53:33] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.723 second response time [08:53:34] PROBLEM - db161 MariaDB on db161 is CRITICAL: Too many connections [08:53:45] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 7.966 second response time [08:57:01] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:06] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:57:56] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:07] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:58:49] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:59:09] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:59:17] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:05:33] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:09:58] RECOVERY - db161 APT on db161 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [09:10:03] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [09:10:15] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.177 second response time [09:10:16] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.165 second response time [09:10:17] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.361 second response time [09:10:17] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.580 second response time [09:10:19] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.062 second response time [09:10:22] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [09:10:26] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [09:10:28] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [09:10:29] !log kill MariaDB on db161 so hopefully it stops being stuck, no time to debug [09:10:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:39] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.065 second response time [09:10:47] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [09:11:17] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 60 Threads: 452 Questions: 1435 Slow queries: 0 Opens: 123 Open tables: 117 Queries per second avg: 23.916 [09:12:24] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 45.2%Current connections: 452 [09:14:06] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:14:08] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:14:11] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:14:14] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:14:14] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:14:20] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:14:21] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:14:23] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:14:33] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:14:42] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [09:16:48] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:16:58] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:33] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:25:33] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:27:40] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.074 second response time [09:27:42] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 5.155 second response time [09:27:42] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.625 second response time [09:27:42] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 6.652 second response time [09:27:43] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 6.707 second response time [09:27:43] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.087 second response time [09:27:45] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 8.415 second response time [09:27:45] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.717 second response time [09:27:46] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.377 second response time [09:27:48] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 4.678 second response time [09:27:51] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 3.212 second response time [09:27:51] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 3.217 second response time [09:27:54] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 2.320 second response time [09:27:59] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 43 minutes ago with 0 failures [09:28:03] RECOVERY - db161 APT on db161 is OK: APT OK: 127 packages available for upgrade (0 critical updates). [09:28:07] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.070 second response time [09:28:10] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.058 second response time [09:28:11] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.063 second response time [09:28:12] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [09:28:12] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.197 second response time [09:28:14] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.056 second response time [09:28:18] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.071 second response time [09:28:34] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.169 second response time [09:28:37] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.060 second response time [09:28:43] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [09:28:43] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [09:28:55] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4327 bytes in 0.059 second response time [09:28:57] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time [09:29:01] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [09:29:06] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.048 second response time [09:29:07] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [09:29:07] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [09:29:10] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.204 second response time [09:29:12] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.062 second response time [09:29:14] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.189 second response time [09:29:14] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [09:29:15] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.061 second response time [09:29:16] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.070 second response time [09:29:21] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.058 second response time [09:29:21] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [09:29:26] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.057 second response time [09:29:28] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.057 second response time [09:29:33] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:29:52] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 17% [09:35:33] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [10:29:44] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 31.77, 12.54, 7.17 [10:31:11] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 3.65, 3.54, 11.15 [10:31:44] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.07, 10.03, 6.92 [10:35:11] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 2.97, 3.68, 9.50 [12:47:44] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 61360MiB (10% inode=99%); [13:13:44] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 33687MiB (5% inode=99%); [13:50:32] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49915MiB (10% inode=99%); [14:09:44] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 167769MiB (28% inode=99%); [14:14:22] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49817MiB (10% inode=99%); [14:28:12] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49789MiB (10% inode=99%); [16:55:44] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.16, 16.21, 8.19 [17:01:44] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.50, 11.98, 10.01 [17:03:44] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.81, 8.27, 8.88 [17:55:06] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 45% [17:59:01] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 28% [19:00:22] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.05, 10.21, 7.13 [19:02:18] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.95, 8.21, 6.76 [19:15:56] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.46, 10.35, 8.24 [19:17:53] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.51, 8.27, 7.74 [22:14:55] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.44/extensions/Mermaid,1.45/extensions/Mermaid'} to test151 [22:14:56] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.44/extensions/Mermaid,1.45/extensions/Mermaid'} to test151 - SUCCESS in 0s [22:14:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:03] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/Mermaid'} to all [22:17:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:17:27] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/Mermaid'} to all - SUCCESS in 24s [22:17:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:20] !log removed local patch for Mermaid 1.44 compatibility (T14250) [22:18:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:41] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'TimedMediaHandler'} to test151 [23:18:43] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'TimedMediaHandler'} to test151 - SUCCESS in 2s [23:18:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:32] !log [somerandomdeveloper@test151] starting deploy of {'world': True, 'versions': '1.44'} to test151 [23:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:57] !log [somerandomdeveloper@test151] finished deploy of {'world': True, 'versions': '1.44'} to test151 - SUCCESS in 25s [23:20:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:13] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CSS'} to test151 [23:25:14] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CSS'} to test151 - SUCCESS in 1s [23:25:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:22] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CSS'} to all [23:25:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:00] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CSS'} to all - SUCCESS in 37s [23:26:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log