[00:00:54] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/f18eba1a5ff21e116a455f43c34eaf2f823cb09e [00:00:54] 02statichelp/03main 07WikiTideBot 03f18eba1 Bot: Auto-update Tech namespace pages 2026-07-04 00:00:51 [00:39:07] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 86185MiB (19.0% inode=100%); [00:48:16] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw161 mw193 mw201 mw203 [00:49:27] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 35.32, 21.56, 9.87 [00:52:16] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [00:57:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.90, 8.87, 9.38 [04:22:05] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 10.97, 9.48, 6.60 [04:28:05] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 17.33, 11.99, 8.42 [04:30:05] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.25, 11.57, 8.72 [04:32:05] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 6.15, 9.67, 8.37 [04:44:05] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.72, 10.23, 8.87 [04:46:05] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 5.77, 8.45, 8.39 [05:32:23] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.22, 9.82, 5.48 [05:34:23] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 6.22, 8.50, 5.55 [05:34:34] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.94, 16.68, 12.43 [05:35:00] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.33, 17.19, 12.53 [05:36:34] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.38, 15.91, 12.66 [05:37:00] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 13.99, 15.43, 12.43 [09:18:48] [02puppet] 07RhinosF1 approved pull request #4874 13https://github.com/miraheze/puppet/pull/4874#pullrequestreview-4629262275 [09:19:02] [02puppet] 07RhinosF1 merged 07lihaohong6's pull request #4874: T15558: move robots.php from puppet to mw-config (07miraheze:03main...07lihaohong6:03robots2) 13https://github.com/miraheze/puppet/pull/4874 [09:19:02] [02puppet] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/babc3a13135589f4283fb281b095d585336f8205 [09:19:02] 02puppet/03main 07Peter Li 03babc3a1 T15558: move robots.php from puppet to mw-config (#4874)… [09:23:25] !log rm /srv/mediawiki/robots.php on mw and test [09:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:07] [02puppet] 07lihaohong6 closed pull request #4841: T15231: experimental robots.php adjustment for non-English support (07miraheze:03main...07lihaohong6:03robots) 13https://github.com/miraheze/puppet/pull/4841 [09:25:07] [02puppet] 07lihaohong6 commented on pull request #4841: robots.php got moved to mw-config. This PR is no longer applicable. 13https://github.com/miraheze/puppet/pull/4841#issuecomment-4881455441 [09:30:27] PROBLEM - mw151 SSH on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:31:56] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.71, 28.28, 16.63 [09:32:27] RECOVERY - mw151 SSH on mw151 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [09:33:25] miraheze/mw-config - lihaohong6 the build has errored. [09:33:54] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 10.00, 21.08, 15.39 [09:34:07] !log [petramagna@test151] starting deploy of {'config': True} to test151 [09:34:08] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:34:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:35:52] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 6.51, 16.39, 14.35 [09:35:56] !log [petramagna@test151] starting deploy of {'config': True} to test151 [09:35:57] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:35:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:36:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:36:35] miraheze/mw-config - lihaohong6 the build has errored. [09:36:47] !log [petramagna@test151] starting deploy of {'config': True} to test151 [09:36:48] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:36:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:39:39] !log [petramagna@test151] starting deploy of {'config': True} to test151 [09:39:40] !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:39:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:39:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:41:22] [02mw-config] 07lihaohong6 readied pull request #6381: T15231: add placeholder for non-English links in robots.txt (07miraheze:03main...07lihaohong6:03robots) 13https://github.com/miraheze/mw-config/pull/6381 [09:45:50] miraheze/mw-config - lihaohong6 the build passed. [09:56:56] miraheze/mw-config - lihaohong6 the build passed. [10:03:10] miraheze/mw-config - lihaohong6 the build passed. [10:04:46] [02mw-config] 07lihaohong6 retitled PR #6381: "T15231: add placeholder for non-English links in robots.txt" ➜ "T15231: exclude localized special page titles in robots.txt" 13https://github.com/miraheze/mw-config/pull/6381 [10:06:41] miraheze/mw-config - lihaohong6 the build passed. [11:19:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.14, 18.79, 14.65 [11:21:38] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.91, 18.62, 15.11 [11:25:11] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:15] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:15] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:25:17] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [11:25:17] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:20] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:25:22] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:23] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:25:29] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:25:40] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:25:42] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:44] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:48] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:25:53] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:25:55] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:25:56] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:26:03] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:26:04] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [11:26:04] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:26:16] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw163 mw191 mw193 mw201 [11:26:21] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:26:23] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 144.36, 78.15, 33.75 [11:26:26] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw163 mw191 mw193 mw201 [11:26:36] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw191 mw193 mw201 [11:26:42] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:26:43] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:26:47] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [11:26:48] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [11:26:49] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 3 backends are down. mw161 mw191 mw201 [11:26:50] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [11:26:52] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:26:55] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:26:56] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:26:58] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:27:01] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:27:09] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:27:20] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:44] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:49] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:51] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 8.816 second response time [11:28:52] [02python-functions] 07renovate[bot] created 03renovate/all-deps (+1 new commit) 13https://github.com/miraheze/python-functions/commit/cba69f127850 [11:28:52] 02python-functions/03renovate/all-deps 07renovate[bot] 03cba69f1 chore(deps): update dependency coverage to v7.15.0 [11:28:53] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.654 second response time [11:29:06] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.265 second response time [11:29:14] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1485 bytes in 9.236 second response time [11:29:24] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.247 second response time [11:29:26] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 7.287 second response time [11:29:50] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.758 second response time [11:29:50] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 8.462 second response time [11:29:53] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.811 second response time [11:29:53] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 9.666 second response time [11:30:16] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [11:32:49] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:32:58] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [11:33:07] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:33:10] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:19] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [11:33:25] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:33:42] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:33:42] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:33:44] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:33:46] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:34:16] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw161 mw163 mw191 mw201 [11:34:43] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:35:34] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 8.676 second response time [11:35:49] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:53] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [11:37:44] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [11:37:51] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.067 second response time [11:38:16] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 9.379 second response time [11:38:36] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [11:38:50] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 59% [11:38:51] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 5.217 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [11:40:50] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [11:41:49] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:42:30] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:42:36] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw161 mw163 mw191 mw201 [11:43:30] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.395 second response time [11:43:32] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.112 second response time [11:43:35] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.951 second response time [11:45:14] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:47:05] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:25] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [11:47:27] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [11:47:27] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [11:47:43] Depooling db161. Maybe those DNS errors are related to the outage? [11:47:44] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s [11:47:44] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 3.634 second response time [11:47:48] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 4.428 second response time [11:47:49] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.705 second response time [11:47:50] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 3.547 second response time [11:47:54] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.032 second response time [11:47:59] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 3.725 second response time [11:48:00] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 4.451 second response time [11:48:02] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.031 second response time [11:48:07] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.446 second response time [11:48:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:24] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.074 second response time [11:48:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:25] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 1.518 second response time [11:48:26] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [11:48:36] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [11:48:36] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.229 second response time [11:48:42] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.083 second response time [11:48:49] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [11:48:50] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 8% [11:48:57] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.109 second response time [11:49:00] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.075 second response time [11:49:00] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.254 second response time [11:49:05] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [11:49:06] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.071 second response time [11:49:14] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.076 second response time [11:49:14] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.246 second response time [11:49:16] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.246 second response time [11:49:19] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 3.016 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [11:49:19] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.335 second response time [11:49:23] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.244 second response time [11:49:23] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.264 second response time [11:49:23] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.233 second response time [11:49:24] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1518 bytes in 0.090 second response time [11:49:24] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.075 second response time [11:49:34] @Infrastructure Specialists I did https://discord.com/channels/407504499280707585/808001911868489748/1522931914372284558. Won't be around soon so someone else needs to take over when things calm down. [11:49:37] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.086 second response time [11:49:37] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.080 second response time [11:49:41] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.368 second response time [11:49:42] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.074 second response time [11:50:16] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [11:55:58] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:19] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [12:19:29] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:21:34] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 8.405 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [12:29:10] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:29:35] @Infrastructure Specialists see mm [12:33:20] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 7.735 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [12:37:45] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:46:08] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 5.498 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [12:48:26] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [12:50:33] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:51:19] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [12:52:59] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [12:53:19] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [12:55:48] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 132 packages available for upgrade (18 critical updates). [12:57:39] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [12:58:49] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 1.030 second response time. db161.fsslc.wtnet returns 10.0.16.128 [12:59:18] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [12:59:23] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:01:59] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 94 packages available for upgrade (15 critical updates). [13:02:08] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:03:02] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 133 packages available for upgrade (19 critical updates). [13:03:14] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:04:17] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:05:12] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (17 critical updates). [13:05:57] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:06:15] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:07:00] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:09:06] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 92 packages available for upgrade (15 critical updates). [13:09:53] PROBLEM - db161 conntrack_table_size on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:10:46] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:11:37] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 7.742 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [13:11:38] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:11:51] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:12:06] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:12:17] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:13:29] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 131 packages available for upgrade (18 critical updates). [13:14:47] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:15:14] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:15:45] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 93 packages available for upgrade (15 critical updates). [13:16:02] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:16:19] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 130 packages available for upgrade (18 critical updates). [13:17:45] RECOVERY - db161 conntrack_table_size on db161 is OK: OK: nf_conntrack is 0 % full [13:24:24] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 7.340 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [13:28:49] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:47:32] PROBLEM - db161 conntrack_table_size on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:39] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 2.967 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [13:51:59] PROBLEM - db161 SSH on db161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:04] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:52:54] RECOVERY - db161 conntrack_table_size on db161 is OK: OK: nf_conntrack is 0 % full [13:54:02] RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [14:02:31] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 5.215 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [14:13:15] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:24:08] PROBLEM - db161 SSH on db161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:08] RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [14:27:58] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 7.668 seconds response time. db161.fsslc.wtnet returns 10.0.16.128 [14:30:57] !log sudo qm reset 128 (cloud16 for db161) [14:31:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:24] RECOVERY - db161 APT on db161 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [14:32:26] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.20, 0.48, 0.18 [14:32:50] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:33:18] !log [petramagna@mwtask181] starting deploy of {'config': True} to all [14:33:19] !log [petramagna@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [14:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:21] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [14:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:42] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s [14:33:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:39:13] PROBLEM - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki_job_manage-inactive-wikis [17:12:15] [1/2] https://cdn.discordapp.com/attachments/808001911868489748/1523013584160952328/image.jpg?ex=6a4a906e&is=6a493eee&hm=5c9f00704c30845ceed3ec75da7376d34e76a2ad1ca58684a8a3036d73e6cc7c& [17:12:15] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1523013584530047076/image.jpg?ex=6a4a906e&is=6a493eee&hm=591fc2531af21be9b77196d90622d871b8069e1b5fde279c541fd2ae7de02034& [17:31:50] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png] [17:31:55] PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl [17:31:58] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [17:39:55] RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl [17:39:58] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [17:59:50] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:44] !log [void@puppet181] Upgraded packages on matomo151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [19:17:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:37] !log [void@puppet181] Upgraded packages on phorge171: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [19:18:40] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [19:18:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:04] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 77 packages available for upgrade (0 critical updates). [19:20:30] !log [void@puppet181] Upgraded packages on mw153: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:20:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:53] RECOVERY - mw153 APT on mw153 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:22:41] RECOVERY - mw152 APT on mw152 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:22:54] !log [void@puppet181] Upgraded packages on mw152: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:22:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:51] RECOVERY - mw162 APT on mw162 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:25:16] !log [void@puppet181] Upgraded packages on mw162: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:12] RECOVERY - mw171 APT on mw171 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:27:11] !log [void@puppet181] Upgraded packages on mw171: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:27:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:15] !log [void@puppet181] Upgraded packages on mw161: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:29:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:31] RECOVERY - mw161 APT on mw161 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:31:17] RECOVERY - mw151 APT on mw151 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:31:44] !log [void@puppet181] Upgraded packages on mw151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:31:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:58] !log [void@puppet181] Upgraded packages on mwtask151: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:33:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:01] RECOVERY - mw191 APT on mw191 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:34:17] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:35:16] !log [void@puppet181] Upgraded packages on mw191: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:35:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:18] !log [void@puppet181] Upgraded packages on mon181: php8.4-opcache, libapache2-mod-php8.4, php8.4-xml, php8.4-soap, php8.4-readline, php8.4-pgsql, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, php8.4-common, and php8.4 [19:36:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:53] RECOVERY - mon181 APT on mon181 is OK: APT OK: 13 packages available for upgrade (0 critical updates). [19:37:28] !log [void@puppet181] Upgraded packages on mwtask161: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:04] RECOVERY - mw172 APT on mw172 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:38:16] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:39:55] !log [void@puppet181] Upgraded packages on mw172: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:39:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:03] RECOVERY - mw163 APT on mw163 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:41:55] [02RequestCustomDomain] 07MacFan4000 created 031.46-patch from 03main (+0 new commit) 13https://github.com/miraheze/RequestCustomDomain/compare/1.46-patch [19:41:57] !log [void@puppet181] Upgraded packages on mw163: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:42:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:37] RECOVERY - mw201 APT on mw201 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:42:55] [02RequestCustomDomain] 07MacFan4000 pushed 1 new commit to 031.46-patch 13https://github.com/miraheze/RequestCustomDomain/commit/880f7da058a1d22e6f0cf1c859fc26228f4d1afe [19:42:56] 02RequestCustomDomain/031.46-patch 07MacFan4000 03880f7da Update RequestCustomDomainCFAddJob.php [19:43:29] [02RequestCustomDomain] 07MacFan4000 pushed 1 new commit to 031.46-patch 13https://github.com/miraheze/RequestCustomDomain/commit/912bc514559565d315526b0b0ebf3ece3987ca37 [19:43:30] 02RequestCustomDomain/031.46-patch 07MacFan4000 03912bc51 Update RequestManager.php [19:44:20] !log [void@puppet181] Upgraded packages on mw201: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:44:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:44:31] [02RequestCustomDomain] 07MacFan4000 opened pull request #114: 1.46 patch (03main...031.46-patch) 13https://github.com/miraheze/RequestCustomDomain/pull/114 [19:45:06] [02RequestCustomDomain] 07MacFan4000 pushed 1 new commit to 031.46-patch 13https://github.com/miraheze/RequestCustomDomain/commit/d15d286b5732876f53e90b612ec5418450ab5173 [19:45:07] 02RequestCustomDomain/031.46-patch 07MacFan4000 03d15d286 Update extension.json [19:45:08] RECOVERY - mw192 APT on mw192 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:46:10] !log [void@puppet181] Upgraded packages on mw192: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:46:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:28] [02RequestCustomDomain] 07github-actions[bot] pushed 1 new commit to 031.46-patch 13https://github.com/miraheze/RequestCustomDomain/commit/f45491a31d40f5cbd547856c7a40597d80f22939 [19:46:28] 02RequestCustomDomain/031.46-patch 07github-actions 03f45491a CI: lint code to MediaWiki standards… [19:47:47] miraheze/RequestCustomDomain - MacFan4000 the build has errored. [19:48:05] !log [void@puppet181] Upgraded packages on mw181: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:48:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:22] RECOVERY - mw181 APT on mw181 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:49:08] miraheze/RequestCustomDomain - MacFan4000 the build has errored. [19:49:50] !log [void@puppet181] Upgraded packages on mw203: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:49:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:11] RECOVERY - mw203 APT on mw203 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:51:25] [02RequestCustomDomain] 07MacFan4000 un-readied pull request #114: 1.46 patch (03main...031.46-patch) 13https://github.com/miraheze/RequestCustomDomain/pull/114 [19:52:28] RECOVERY - mw182 APT on mw182 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:52:35] !log [void@puppet181] Upgraded packages on mw182: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:52:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:12] RECOVERY - mw202 APT on mw202 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:53:36] miraheze/RequestCustomDomain - github-actions[bot] the build has errored. [19:55:18] !log [void@puppet181] Upgraded packages on mw202: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:55:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:35] !log [void@puppet181] Upgraded packages on mwtask171: php8.4-bcmath, php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-cli, and php8.4-common [19:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:24] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 113 packages available for upgrade (0 critical updates). [19:58:02] RECOVERY - mw193 APT on mw193 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [19:58:33] !log [void@puppet181] Upgraded packages on mw193: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [19:58:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:36] RECOVERY - mw173 APT on mw173 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [20:00:13] !log [void@puppet181] Upgraded packages on mw173: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:05] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [20:01:29] !log [void@puppet181] Upgraded packages on mwtask181: php8.4-bcmath, php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-cli, and php8.4-common [20:01:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:20] RECOVERY - mw183 APT on mw183 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [20:03:40] !log [void@puppet181] Upgraded packages on mw183: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-cli, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-bcmath, and php8.4-common [20:03:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:35] !log [void@puppet181] Upgraded packages on reports171: php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-curl, php8.4-cli, and php8.4-common [20:04:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:47] !log [void@puppet181] Upgraded packages on test151: php8.4-bcmath, php8.4-zip, php8.4-xml, php8.4-readline, php8.4-opcache, php8.4-phpdbg, php8.4-mysql, php8.4-mbstring, php8.4-ldap, php8.4-intl, php8.4-gmp, php8.4-gd, php8.4-fpm, php8.4-dev, php8.4-dba, php8.4-curl, php8.4-bz2, php8.4-cli, and php8.4-common [20:05:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:53] RECOVERY - reports171 APT on reports171 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [20:06:22] RECOVERY - test151 APT on test151 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [20:07:09] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Package[php8.4-opcache],Package[php8.4-cli],Package[php8.4-fpm] [20:11:07] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45236MiB (10.0% inode=100%); [20:35:09] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:39:49] !log [void@mwtask181] restart mediawiki_job_manage-inactive-wikis [20:39:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:13] RECOVERY - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis [23:13:50] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45280MiB (10.0% inode=100%); [23:20:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45360MiB (10.0% inode=100%);