[00:00:59] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/a0f693eb53ac91e2fb6bff68aa36cc23340c7aeb [00:01:01] 02statichelp/03main 07WikiTideBot 03a0f693e Bot: Auto-update Tech namespace pages 2026-07-13 00:00:57 [00:44:05] !log [void@cloud21] clear ipmi sel (inlet temperature spike) [00:44:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:32] RECOVERY - cloud21 IPMI Sensors on cloud21 is OK: IPMI Status: OK [00:49:00] [02mw-config] 07The-Voidwalker merged 07KockaAdmiralac's pull request #6374: Toggle for replacing CodeEditor with CodeMirror (07miraheze:03main...07KockaAdmiralac:03codemirror) 13https://github.com/miraheze/mw-config/pull/6374 [00:49:00] [02mw-config] 07The-Voidwalker pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/fb9bb5bab4c516a2ecfdaf77ff94d5fc2db5b60e [00:49:00] 02mw-config/03main 07Luka Simić 03fb9bb5b Toggle for replacing CodeEditor with CodeMirror (#6374)… [00:49:14] !log [void@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:49:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:40] !log [void@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [00:49:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:11] miraheze/mw-config - The-Voidwalker the build passed. [02:51:28] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.55, 13.44, 11.98 [02:55:21] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 8.56, 11.83, 11.73 [03:01:05] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/59a97b70ed3d767c5dbf6d9ed2d18f6c6db49e59 [03:01:07] 02puppet/03main 07paladox 0359a97b7 Update Graylog package version to 7.1.5-1 [03:02:35] PROBLEM - db171 Check unit status of sql-backup on db171 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [03:11:54] [02mw-config] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/8992f1853c0b [03:11:54] 02mw-config/03paladox-patch-6 07paladox 038992f18 Add 'blog_root' content model to SimpleBlogPage [03:11:58] [02mw-config] 07paladox opened pull request #6462: Add 'blog_root' content model to SimpleBlogPage (03main...03paladox-patch-6) 13https://github.com/miraheze/mw-config/pull/6462 [03:13:00] miraheze/mw-config - paladox the build passed. [03:13:44] [02mw-config] 07paladox merged pull request #6462: Add 'blog_root' content model to SimpleBlogPage (03main...03paladox-patch-6) 13https://github.com/miraheze/mw-config/pull/6462 [03:13:46] [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/f5342308c6f529130d63d9a73d2df5516076fa37 [03:13:48] [02mw-config] 07paladox 04deleted 03paladox-patch-6 at 038992f18 13https://api.github.com/repos/miraheze/mw-config/commit/8992f18 [03:13:49] 02mw-config/03main 07paladox 03f534230 Add 'blog_root' content model to SimpleBlogPage (#6462) [03:14:07] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:14:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:22] !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [03:14:25] !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 3s [03:14:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:32] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [03:14:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:51] miraheze/mw-config - paladox the build passed. [03:42:23] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw202 mw203 [03:42:38] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw202 mw203 [03:42:45] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw202 mw203 [03:42:47] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw202 mw203 [03:42:49] PROBLEM - mw203 PowerDNS Recursor on mw203 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:42:49] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:42:53] PROBLEM - mw202 SSH on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:43:05] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 48.15, 29.89, 16.16 [03:44:05] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 29.53, 29.00, 16.56 [03:44:19] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [03:44:36] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [03:44:41] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [03:44:44] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.075 second response time [03:44:47] RECOVERY - mw203 PowerDNS Recursor on mw203 is OK: DNS OK: 0.046 seconds response time. mw203.fsslc.wtnet returns 10.0.20.165 [03:44:47] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [03:44:51] RECOVERY - mw202 SSH on mw202 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [03:45:04] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 12.43, 22.25, 14.96 [03:46:00] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 9.53, 21.76, 15.37 [03:47:03] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 7.02, 16.95, 13.90 [03:47:55] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 7.05, 16.91, 14.34 [03:57:10] !log [void@db171] restart sql-backup.service [03:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:58:35] RECOVERY - db171 Check unit status of sql-backup on db171 is OK: OK: Status of the systemd unit sql-backup [04:02:37] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 13.62, 11.75, 9.11 [04:27:50] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 5 backends are down. mw152 mw171 mw181 mw182 mw203 [04:28:04] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 3 backends are down. mw152 mw182 mw153 [04:28:14] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw152 mw162 mw183 mw193 [04:28:46] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 7 backends are down. mw151 mw161 mw162 mw153 mw163 mw192 mw193 [04:31:15] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 46% [04:33:59] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [04:35:15] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 10% [04:35:50] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [04:35:58] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.306 second response time [04:36:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [04:36:14] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [04:36:46] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [04:54:37] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.62, 1.92, 12.00 [04:58:37] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.71, 1.24, 9.41 [06:01:59] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 26.26, 15.07, 8.83 [06:02:20] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 15.44, 9.56 [06:06:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw201 mw202 [06:06:14] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 8 backends are down. mw152 mw172 mw181 mw182 mw153 mw183 mw192 mw201 [06:06:16] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [06:06:20] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 4.08, 11.85, 9.76 [06:06:46] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 mw153 mw163 mw183 mw191 mw192 mw201 mw202 mw203 [06:07:15] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 57% [06:07:16] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:07:50] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 mw153 mw163 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [06:07:58] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:08:05] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [06:08:20] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.56, 8.27, 8.69 [06:08:21] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:09:11] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.061 second response time [06:09:15] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 14% [06:10:06] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.456 second response time [06:10:17] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.309 second response time [06:12:14] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 7.859 second response time [06:12:14] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [06:12:20] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 61.25, 35.69, 19.62 [06:13:50] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [06:14:04] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [06:14:18] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [06:14:46] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [06:22:20] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 1.29, 6.24, 11.49 [06:23:59] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.26, 4.39, 11.84 [06:24:20] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.55, 4.32, 10.15 [06:27:59] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.80, 3.05, 9.59 [06:33:59] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 51.98, 20.86, 13.99 [06:36:04] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:36:14] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw193 mw201 mw202 mw203 [06:36:46] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw182 [06:36:52] Depooling db161 [06:37:11] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [06:37:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:29] !log Depooling db161 [06:37:32] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 20s [06:37:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:02] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.061 second response time [06:38:14] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [06:38:46] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [06:43:23] !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all [06:43:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:31] !log Adjust Cloudflare rules and repool db161 [06:43:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:41] !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 18s [06:43:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:45:59] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.81, 6.08, 11.33 [06:47:59] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.27, 4.44, 10.09 [22:12:39] [02CreateWiki] 07dependabot[bot] created 03dependabot/npm_and_yarn/svgo-4.0.2 (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/4d335a72b7d4 [22:12:40] 02CreateWiki/03dependabot/npm_and_yarn/svgo-4.0.2 07dependabot[bot] 034d335a7 Bump svgo from 4.0.1 to 4.0.2… [22:12:40] [02CreateWiki] 07dependabot[bot] added the label 'dependencies' to pull request #829 (Bump svgo from 4.0.1 to 4.0.2) 13https://github.com/miraheze/CreateWiki/pull/829 [22:12:41] [02CreateWiki] 07dependabot[bot] opened pull request #829: Bump svgo from 4.0.1 to 4.0.2 (03main...03dependabot/npm_and_yarn/svgo-4.0.2) 13https://github.com/miraheze/CreateWiki/pull/829 [22:12:42] [02CreateWiki] 07dependabot[bot] added the label 'javascript' to pull request #829 (Bump svgo from 4.0.1 to 4.0.2) 13https://github.com/miraheze/CreateWiki/pull/829 [22:12:44] [02CreateWiki] 07dependabot[bot] added the label 'dependencies' to pull request #829 (Bump svgo from 4.0.1 to 4.0.2) 13https://github.com/miraheze/CreateWiki/pull/829 [22:12:46] [02CreateWiki] 07dependabot[bot] added the label 'javascript' to pull request #829 (Bump svgo from 4.0.1 to 4.0.2) 13https://github.com/miraheze/CreateWiki/pull/829 [22:19:45] miraheze/CreateWiki - dependabot[bot] the build passed. [22:23:25] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:ReplaceTextEligible --wiki=wikibitkiwiki (END - exit=0) [22:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:05] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': '1.45'} to all [23:51:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:27] !log [paladox@test151] starting deploy of {'world': True, 'versions': ['1.45', '1.46']} to test151 [23:51:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:55] !log upgrade SemanticMediaWiki to tip of master branch as of today on mw and test [23:51:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:52:23] !log [paladox@test151] finished deploy of {'world': True, 'versions': ['1.45', '1.46']} to test151 - SUCCESS in 55s [23:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:44] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': '1.45'} to all - SUCCESS in 219s [23:54:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log