[00:01:26] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/202d57e8c40b510f9ac9bd78f97b2ac90cb5b170 [00:01:26] 02statichelp/03main 07WikiTideBot 03202d57e Bot: Auto-update Tech namespace pages 2025-09-21 00:01:24 [00:11:35] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54332MiB (11% inode=99%); [00:12:09] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 54773MiB (12% inode=99%); [00:13:27] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 54060MiB (11% inode=99%); [01:00:33] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.59, 10.60, 6.52 [01:02:30] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.06, 7.41, 5.84 [03:04:09] PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 1899MiB (6% inode=80%); [03:06:09] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4047MiB (13% inode=80%); [03:07:27] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49783MiB (10% inode=99%); [03:13:35] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49847MiB (10% inode=99%); [03:19:39] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [03:30:09] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49901MiB (10% inode=99%); [08:01:39] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [09:11:40] [02mw-config] 07c-oreills left a file comment in pull request #6113 03817fc77: @pskyechology I've cherry picked in the commit from #6112 and then removed the override [as you requested](https://github.com/miraheze/mw-config/pull/6112/files#r2356921423). […] 13https://github.com/miraheze/mw-config/pull/6113#discussion_r2366139385 [09:12:08] [02mw-config] 07c-oreills closed pull request #6112: T14285: add NS_MAIN to wgMFNamespacesWithoutCollapsibleSections for twistwoodtaleswiki (07miraheze:03main...07c-oreills:03main) 13https://github.com/miraheze/mw-config/pull/6112 [09:12:09] [02mw-config] 07c-oreills commented on pull request #6112: Closed in favor of #6113 13https://github.com/miraheze/mw-config/pull/6112#issuecomment-3315863566 [13:42:50] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1950MiB (10% inode=89%); [14:23:31] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.11, 7.73, 4.37 [14:25:28] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 9.44, 8.67, 5.12 [14:27:08] [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c1b1b738e21f6e94f3838a90e2a9cb7846fa3ce3 [14:27:09] 02ssl/03main 07Skye 03c1b1b73 Redirect left4dead.miraheze.org to left4deadwiki.com [14:53:11] [02puppet] 07paladox created 03revert-php-buffer (+2 new commits) 13https://github.com/miraheze/puppet/compare/6d4900b05632^...f1a767180b65 [14:53:11] 02puppet/03revert-php-buffer 07paladox 036d4900b Revert "mediawiki: set fastcgi_buffering to off in nginx template (#4330)"… [14:53:11] 02puppet/03revert-php-buffer 07paladox 03f1a7671 Revert "mediawiki: set fastcgi_request_buffering to off (#4322)"… [14:53:36] [02puppet] 07paladox opened pull request #4531: Revert turning fastcgi_buffering and fastcgi_request_buffering off (03main...03revert-php-buffer) 13https://github.com/miraheze/puppet/pull/4531 [14:53:43] [02puppet] 07coderabbitai[bot] commented on pull request #4531:
[…] 13https://github.com/miraheze/puppet/pull/4531#issuecomment-3316045560 [15:02:48] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6452MiB (35% inode=89%); [15:11:57] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [15:12:04] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 7111 bytes in 3.099 second response time [15:12:05] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:05] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [15:12:07] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [15:12:08] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:12:15] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: No route to host in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connectio [15:12:15] line 66Fatal error: Uncaught mysqli_sql_exception: No route to host in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 7111 bytes in 3.101 second response time [15:12:27] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:27] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:31] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [15:12:35] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [15:12:36] PROBLEM - db171 PowerDNS Recursor on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:12:36] PROBLEM - db171 Check unit status of sql-weekly-mhglobal-backup on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:12:40] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4851 bytes in 5.350 second response time [15:12:43] PROBLEM - db171 Disk Space on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:12:43] PROBLEM - db171 Current Load on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:12:44] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [15:12:48] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4855 bytes in 3.094 second response time [15:12:48] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 7111 bytes in 3.113 second response time [15:12:49] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4855 bytes in 3.094 second response time [15:12:50] PROBLEM - db171 NTP time on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:12:50] PROBLEM - db171 Check unit status of sql-backup on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:12:53] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4855 bytes in 3.095 second response time [15:12:57] PROBLEM - db171 conntrack_table_size on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [15:13:00] PROBLEM - Host db171 is DOWN: CRITICAL - Host Unreachable (10.0.17.119) [15:13:00] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [15:13:05] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 11 backends are down. mw152 mw161 mw171 mw172 mw181 mw182 mw173 mw183 mw193 mw201 mw203 [15:13:12] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4851 bytes in 5.588 second response time [15:13:14] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4851 bytes in 1.429 second response time [15:13:17] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 11 backends are down. mw152 mw161 mw172 mw181 mw182 mw173 mw183 mw191 mw192 mw201 mw203 [15:13:17] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [15:13:19] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:21] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:13:23] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4851 bytes in 2.949 second response time [15:13:35] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:38] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 7107 bytes in 1.360 second response time [15:13:38] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 10 backends are down. mw151 mw152 mw161 mw162 mw182 mw153 mw163 mw191 mw192 mw203 [15:13:41] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4855 bytes in 8.442 second response time [15:13:46] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4851 bytes in 2.882 second response time [15:13:52] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 11 backends are down. mw151 mw161 mw162 mw171 mw172 mw182 mw153 mw163 mw191 mw192 mw202 [15:14:01] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:09] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [15:14:14] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 9.458 second response time [15:14:41] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 6.230 second response time [15:15:03] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 7.225 second response time [15:15:21] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 1.595 second response time [15:16:06] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [15:16:46] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:17:02] RECOVERY - Host db171 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [15:17:16] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.375 second response time [15:17:17] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.865 second response time [15:17:19] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:17:20] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.258 second response time [15:17:25] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [15:17:39] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.204 second response time [15:17:39] [02mw-config] 07RhinosF1 created 03RhinosF1-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/56b2755061f4 [15:17:40] 02mw-config/03RhinosF1-patch-2 07RhinosF1 0356b2755 depool c3 for outage [15:17:40] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.109 second response time [15:17:43] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.173 second response time [15:17:45] [02mw-config] 07RhinosF1 opened pull request #6120: depool c3 for outage (03main...03RhinosF1-patch-2) 13https://github.com/miraheze/mw-config/pull/6120 [15:17:52] [02mw-config] 07RhinosF1 merged pull request #6120: depool c3 for outage (03main...03RhinosF1-patch-2) 13https://github.com/miraheze/mw-config/pull/6120 [15:17:52] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/288d7bcee1940325d2e4766dcd31c235a34cc23a [15:17:52] 02mw-config/03main 07RhinosF1 03288d7bc depool c3 for outage (#6120) [15:17:53] [02mw-config] 07RhinosF1 04deleted 03RhinosF1-patch-2 at 0356b2755 13https://api.github.com/repos/miraheze/mw-config/commit/56b2755 [15:17:53] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.817 second response time [15:17:57] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.034 second response time [15:18:03] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [15:18:09] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 63.7%Current connections: 637 [15:18:12] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 1.654 second response time [15:18:13] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [15:18:13] !log [paladox@mwtask181] starting deploy of {'config': True} to all [15:18:14] PROBLEM - db171 Puppet on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:18:18] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.342 second response time [15:18:23] !log [paladox@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw152 [15:18:27] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [15:18:29] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:18:30] !log [rhinos@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [15:18:31] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [15:18:44] RECOVERY - db171 Check unit status of sql-weekly-mhglobal-backup on db171 is OK: OK: Status of the systemd unit sql-weekly-mhglobal-backup [15:18:44] RECOVERY - db171 Disk Space on db171 is OK: DISK OK - free space: / 403770MiB (45% inode=98%); [15:18:44] RECOVERY - db171 PowerDNS Recursor on db171 is OK: DNS OK: 2.127 seconds response time. db171.fsslc.wtnet returns 10.0.17.119 [15:18:45] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.064 second response time [15:18:45] miraheze/mw-config - RhinosF1 the build passed. [15:18:48] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [15:18:49] RECOVERY - db171 NTP time on db171 is OK: NTP OK: Offset -0.005464226007 secs [15:18:53] miraheze/mw-config - RhinosF1 the build passed. [15:18:54] RECOVERY - db171 Check unit status of sql-backup on db171 is OK: OK: Status of the systemd unit sql-backup [15:19:02] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [15:19:04] RECOVERY - db171 conntrack_table_size on db171 is OK: OK: nf_conntrack is 0 % full [15:19:14] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.452 second response time [15:19:20] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 1.592 second response time [15:20:00] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 48% [15:20:17] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 109s [15:20:17] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [15:20:30] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.068 second response time [15:20:44] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4216 bytes in 0.070 second response time [15:21:00] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [15:21:06] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.081 second response time [15:21:14] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.087 second response time [15:21:14] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [15:21:17] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [15:21:17] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.095 second response time [15:21:19] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.056 second response time [15:21:23] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [15:21:36] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.207 second response time [15:21:39] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 22s [15:21:43] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [15:21:57] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 10% [15:22:06] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [15:22:12] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [15:22:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:28] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.174 second response time [15:22:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:30] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [15:22:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:41] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [15:22:41] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time [15:22:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:58] [02puppet] 07paladox merged pull request #4531: Revert turning fastcgi_buffering and fastcgi_request_buffering off (03main...03revert-php-buffer) 13https://github.com/miraheze/puppet/pull/4531 [15:22:58] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/daa27d02b78991356b78073cee125720495f4a69 [15:22:58] 02puppet/03main 07paladox 03daa27d0 Revert turning fastcgi_buffering and fastcgi_request_buffering off (#4531) [15:22:59] [02puppet] 07paladox 04deleted 03revert-php-buffer at 03f1a7671 13https://api.github.com/repos/miraheze/puppet/commit/f1a7671 [15:23:01] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [15:23:03] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.160 second response time [15:23:09] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.165 second response time [15:23:12] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time [15:23:12] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [15:23:19] [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/fe5f0d2f2116 [15:23:19] 02puppet/03paladox-patch-3 07paladox 03fe5f0d2 mediawiki: set mediawiki::php::fpm::fpm_workers_multiplier to 2.5 [15:23:23] [02puppet] 07paladox opened pull request #4532: mediawiki: set mediawiki::php::fpm::fpm_workers_multiplier to 2.5 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4532 [15:23:30] [02puppet] 07coderabbitai[bot] commented on pull request #4532:
[…] 13https://github.com/miraheze/puppet/pull/4532#issuecomment-3316064402 [15:24:44] !log [rhinos@mwtask181] starting deploy of {'pull': 'config'} to mwtask181 [15:24:45] !log [rhinos@mwtask181] finished deploy of {'pull': 'config'} to mwtask181 - SUCCESS in 1s [15:24:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:27] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:25:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:49] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [15:26:02] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [15:27:33] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.963 second response time [15:27:49] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.090 second response time [15:27:57] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.100 second response time [15:28:02] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.091 second response time [15:28:14] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.082 second response time [15:28:16] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.083 second response time [15:28:20] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.088 second response time [15:28:22] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.096 second response time [15:28:26] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.083 second response time [15:28:31] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.078 second response time [15:28:56] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.084 second response time [15:29:01] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.097 second response time [15:29:01] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.076 second response time [15:29:02] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.089 second response time [15:29:02] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.087 second response time [15:29:24] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.097 second response time [15:29:26] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.083 second response time [15:29:27] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.094 second response time [15:29:28] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.097 second response time [15:29:36] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.077 second response time [15:29:38] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.087 second response time [15:29:39] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.097 second response time [15:32:18] [02mw-config] 07RhinosF1 created 03revert-6120-RhinosF1-patch-2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/98975bf73671 [15:32:18] 02mw-config/03revert-6120-RhinosF1-patch-2 07RhinosF1 0398975bf Revert "depool c3 for outage (#6120)"… [15:32:23] [02mw-config] 07RhinosF1 opened pull request #6121: Revert "depool c3 for outage" (03main...03revert-6120-RhinosF1-patch-2) 13https://github.com/miraheze/mw-config/pull/6121 [15:32:34] [02mw-config] 07RhinosF1 merged pull request #6121: Revert "depool c3 for outage" (03main...03revert-6120-RhinosF1-patch-2) 13https://github.com/miraheze/mw-config/pull/6121 [15:32:36] [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/fcad1db388f5536a8b67e5cc18d11bdbe1d19c75 [15:32:38] [02mw-config] 07RhinosF1 04deleted 03revert-6120-RhinosF1-patch-2 at 0398975bf 13https://api.github.com/repos/miraheze/mw-config/commit/98975bf [15:32:39] 02mw-config/03main 07RhinosF1 03fcad1db Revert "depool c3 for outage" (#6121)… [15:32:40] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [15:32:49] !log [rhinos@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:32:57] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [15:33:12] !log [rhinos@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [15:33:19] miraheze/mw-config - RhinosF1 the build passed. [15:33:22] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [15:33:25] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [15:33:26] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [15:33:26] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [15:33:32] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [15:33:33] miraheze/mw-config - RhinosF1 the build passed. [15:33:33] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.182 second response time [15:33:33] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.325 second response time [15:33:43] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [15:33:46] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.237 second response time [15:33:55] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.241 second response time [15:34:02] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [15:34:11] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.204 second response time [15:34:11] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [15:34:14] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.227 second response time [15:34:18] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.255 second response time [15:34:31] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.214 second response time [15:34:51] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [15:34:58] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time [15:35:00] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.161 second response time [15:35:01] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [15:35:09] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.244 second response time [15:42:02] !log setting php-fpm childs @ 3  12 (mw cores) on mw to experiment [15:42:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:44] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [15:46:17] ^ expected [15:50:40] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [16:03:41] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.34, 7.27, 11.91 [16:05:40] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 6.21, 9.44, 12.35 [16:07:40] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.13, 6.44, 10.89 [16:09:40] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.61, 4.49, 9.64 [16:13:40] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 5.48, 11.18, 11.75 [16:13:44] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:17:40] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.92, 5.44, 9.25 [16:31:06] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a47f0ebb6611b9f0984beca88a5372e1303bc7c4 [16:31:06] 02mw-config/03main 07Skye 03a47f0eb Add wgMinervaShowCategories override for osmaniawiki… [16:31:35] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [16:31:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:57] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [16:32:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:32:05] miraheze/mw-config - pskyechology the build passed. [16:53:53] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27172MiB (5% inode=99%); [16:59:39] !log revert [16:59:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:31] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27174MiB (5% inode=99%); [17:12:49] [02puppet] 07paladox created 03paladox-patch-5 (+1 new commit) 13https://github.com/miraheze/puppet/commit/5d7d97a4ed3d [17:12:50] 02puppet/03paladox-patch-5 07paladox 035d7d97a prometheus: gather mediawiki php http port 9181 metrics [17:12:53] [02puppet] 07paladox opened pull request #4533: prometheus: gather mediawiki php http port 9181 metrics (03main...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4533 [17:12:59] [02puppet] 07coderabbitai[bot] commented on pull request #4533:
[…] 13https://github.com/miraheze/puppet/pull/4533#issuecomment-3316129118 [17:15:44] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/ccf1dbec3f39716502071123689785b8bc153fa1 [17:15:45] 02puppet/03paladox-patch-5 07paladox 03ccf1dbe Update prometheus.pp [17:16:22] [02puppet] 07paladox merged pull request #4533: prometheus: gather mediawiki php http port 9181 metrics (03main...03paladox-patch-5) 13https://github.com/miraheze/puppet/pull/4533 [17:16:22] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4585d03c1fcc9b9d9d384949d028d27f1104dd1b [17:16:22] 02puppet/03main 07paladox 034585d03 prometheus: gather mediawiki php http port 9181 metrics (#4533) [17:16:23] [02puppet] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-5 13https://github.com/miraheze/puppet/commit/1bfcb0cd2e75e09486c063f05e6efb7bb9df95c0 [17:16:24] 02puppet/03paladox-patch-5 07github-actions 031bfcb0c CI: lint puppet code to standards… [17:26:52] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27256MiB (5% inode=99%); [17:31:21] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/a6096b3d39c5 [17:31:21] 02puppet/03paladox-patch-6 07paladox 03a6096b3 prometheus: Add missing php_mediawiki_job var to scrape_extra [17:31:24] [02puppet] 07paladox opened pull request #4534: prometheus: Add missing php_mediawiki_job var to scrape_extra (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4534 [17:31:29] [02puppet] 07paladox merged pull request #4534: prometheus: Add missing php_mediawiki_job var to scrape_extra (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4534 [17:31:31] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/54476aff9180849fc8782a80c75cf7672c16fd82 [17:31:33] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03a6096b3 13https://api.github.com/repos/miraheze/puppet/commit/a6096b3 [17:31:35] 02puppet/03main 07paladox 0354476af prometheus: Add missing php_mediawiki_job var to scrape_extra (#4534) [17:31:37] [02puppet] 07coderabbitai[bot] commented on pull request #4534:
[…] 13https://github.com/miraheze/puppet/pull/4534#issuecomment-3316140357 [17:57:34] [02mw-config] 07pskyechology left a file comment in pull request #6113 03817fc77: It is the intended way and is documented in [CONTRIBUTING.md](https://github.com/miraheze/mw-config/blob/main/.github/CONTRIBUTING.md) 13https://github.com/miraheze/mw-config/pull/6113#discussion_r2366328757 [17:59:04] [02mw-config] 07pskyechology merged 07c-oreills's pull request #6113: T14285: Add wgMFNamespacesWithoutCollapsibleSections to ManageWikiNamespaces.php (07miraheze:03main...07c-oreills:03wgMFNamespacesWithoutCollapsibleSectionsManageWikiNamespaces) 13https://github.com/miraheze/mw-config/pull/6113 [17:59:04] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e11311d5efed431e10f37f17b06c96ec6bbec609 [17:59:04] 02mw-config/03main 07Christy O'Reilly 03e11311d T14285: Add wgMFNamespacesWithoutCollapsibleSections to ManageWikiNamespaces.php (#6113)… [17:59:15] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:59:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:30] !log [skye@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw193 [17:59:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:56] miraheze/mw-config - pskyechology the build passed. [18:01:04] LMAO IT GOT GRAFANA'D [18:01:08] lol [18:01:44] !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [18:01:45] [02mw-config] 07c-oreills commented on pull request #6113: @pskyechology thanks for you help! 😊 13https://github.com/miraheze/mw-config/pull/6113#issuecomment-3316158905 [18:01:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:05] !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 21s [18:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:01] !log [skye@mwtask171] Starting import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (START) [18:24:02] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (START) [18:24:03] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (END - exit=1) [18:24:04] !log [skye@mwtask171] Finished import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (END - exit=1) [18:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:21] i forgor to chmod [18:24:57] !log [skye@mwtask171] Starting import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (START) [18:24:58] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (START) [18:24:59] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (END - exit=1) [18:25:00] !log [skye@mwtask171] Finished import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (END - exit=1) [18:25:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:47] ok now I dont know [18:36:25] i forgor to give others the read right for the folder it was in lol [18:36:44] !log [skye@mwtask171] Starting import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (START) [18:36:45] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (START) [18:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:36:50] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (END - exit=1) [18:36:51] !log [skye@mwtask171] Finished import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (END - exit=1) [18:36:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:36:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:21] unknown error in miraheze-swift wtf [18:38:07] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=thechurchofthestatuewiki (END - exit=0) [18:38:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:38:47] exit 0 yeah right [18:39:19] [1/18] ``` [18:39:19] [2/18] Making sure 'mwstore://miraheze-swift/local-public' exists...making public...failed. [18:39:19] [3/18] Array [18:39:20] [4/18] ( [18:39:20] [5/18] [0] => Wikimedia\Message\MessageValue Object [18:39:20] [6/18] ( [18:39:20] [7/18] [key:Wikimedia\Message\MessageValue:private] => backend-fail-internal [18:39:21] [8/18] [params:Wikimedia\Message\MessageValue:private] => Array [18:39:21] [9/18] ( [18:39:21] [10/18] [0] => Wikimedia\Message\ScalarParam Object [18:39:22] [11/18] ( [18:39:22] [12/18] [type:protected] => text [18:39:23] [13/18] [value:protected] => miraheze-swift [18:39:23] [14/18] ) [18:39:24] [15/18] ) [18:39:24] [16/18] ) [18:39:25] [17/18] ) [18:39:25] [18/18] ``` [18:59:25] @rhinosf1 do you know why swift might be sad for `thechurchofthestatuewiki`? everything just gives An unknown error occurred in storage backend "miraheze-swift". [19:00:21] You tried the script to set permissions? [19:00:34] ah right lemme try the python script [19:00:34] setContainerAccess ? [19:00:55] Oh it's right above [19:01:04] You could tell me to learn to read @pskyechology [19:01:40] @paladox can you help @pskyechology ? [19:02:46] sorry I thought my taxes did that already [19:03:04] doesn't seem the container was created [19:03:37] makes sense, the resets came with a bunch of shenanigans [19:07:32] https://github.com/miraheze/CreateWiki/blob/fedf87fb18cc30ec7eb86b9be82020800c3a1059/maintenance/SetContainersAccess.php#L61 should have created it? [19:07:32] [GitHub] [miraheze/CreateWiki] maintenance/SetContainersAccess.php @ fedf87fb18cc30ec7eb86b9be82020800c3a1059 | L61:  $status = $backend->prepare( [ 'dir' => $dir ] + $secure ); [19:07:34] strange [19:08:23] wtf [19:10:33] oh is it still being deleted? [19:10:45] [1/2] > Sep 21 19:09:45 swiftproxy161 proxy-server: 2607:fea8:4ad9:2100:15e:a88:f0c6:c0d5 ::ffff:127.0.0.1 21/Sep/2025/19/09/45 GET /v1/AUTH_mw/miraheze-thechurchofthestatuewiki-local-thumb/2/2c/Obt_light_back.png/2560px-Obt_light_back.png HTTP/1.0 401 - Mozilla/5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit/537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome/137.0.0.0%20Saf [19:10:46] [2/2] ari/537.36%20OPR/121.0.0.0%20%28Edition%20std-2%29 - - 131 - tx17422f19b1de4a78aae3c-0068d04d79 - 0.0034 - - 1758481785.656185627 1758481785.659542561 0 [19:10:59] i can't list the container [19:11:29] [1/2] > ::ffff:127.0.0.1 21/Sep/2025/19/09/45 GET /v1/AUTH_mw/miraheze-thechurchofthestatuewiki-local-thumb/2/2c/Obt_light_back.png/2560px-Obt_light_back.png HTTP/1.0 401 - Mozilla/5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit/537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome/137.0.0.0%20Safari/537.36%20OPR/121.0.0.0%20%28Edition%20std-2%29 - - 131 - tx17422f19b1de4a78aae [19:11:29] [2/2] 3c-0068d04d79 - 0.0034 - - 1758481785.656185627 1758481785.659542561 0 [19:12:50] shouldn't be, but I wouldn't be surprised if the resetwiki script messed up somewhere as it seems to have done elsewhere [19:13:01] hmm [19:14:22] oh it's just someone fetching it but it 401ing [19:14:31] idk why it's failing [19:15:02] @rhinosf1 / @cosmicalpha any idea? [19:16:24] I thought you were the swift SME [19:16:53] i am i just have never had this problem occur where a container can't be created but it also isn't being listed. [19:17:02] so i was wondering if you ever experenced it [19:17:09] Nope [19:20:32] i can create it with `swift post` [19:20:53] sure, the images are backed up so there's no risk [19:21:16] [1/2] in the log i see something like `::ffff:127.0.0.1 21/Sep/2025/19/16/21 POST /v1/AUTH_mw/miraheze-thechurchofthestatuewiki-local-public HTTP/1.0 404 - wikimedia/multi-http-client%20v1.1 AUTH_tk062d0bb14... - 70 - tx0ca166ca807442f0a4920-0068d04f05 - 0.0115 - - 1758482181.671931028 1758482181.683395863 - [19:21:17] [2/2] ` that is created via the script [19:23:09] oh it needs to be a PUT [19:37:39] oh huh prepare is returning true? [19:37:40] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [19:42:17] @pskyechology fixed [19:42:29] woohoo! cheers for helping out <3 [19:42:41] !log hacked SetContainersAccess.php script to clean thechurchofthestatuewiki and then re-ran it to create contains. [19:42:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:43:34] i absolutely have no clue how that fixes it [19:44:26] @pskyechology shouldn't the rename script check that it successfully renamed swift before? [19:44:57] wasn't rename, was reset [19:45:07] oh [19:45:40] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:47:31] !log [skye@mwtask171] Starting import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (START) [19:47:32] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (START) [19:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:38] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=thechurchofthestatuewiki --sleep=0 '--comment=Reupload pre-reset images' --search-recursively -- ./thechurchofthestatuewiki (END - exit=0) [19:48:39] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=thechurchofthestatuewiki --update (START) [19:48:40] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=thechurchofthestatuewiki --update (END - exit=0) [19:48:41] !log [skye@mwtask171] Finished import for thechurchofthestatuewiki (XML: None; Images: ./thechurchofthestatuewiki) (END - exit=0) [19:48:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:07] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=armeniapastpresentwiki --new=armeniacawiki --user=Skye (END - exit=0) [19:59:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:30] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=armeniacawiki (END - exit=0) [20:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:36] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=trotyenergyresearchfaciliywiki --new=trotywikiwiki --user=Skye (END - exit=0) [20:04:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:08] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=trotywikiwiki (END - exit=0) [20:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:43] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=lastdayofrobloxiawiki --new=pwnedgewiki --user=Skye (END - exit=0) [20:09:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:26] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=pwnedgewiki (END - exit=0) [20:11:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:17] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.60, 9.80, 5.95 [20:16:43] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=senpuuwiki --new=sempuwiki --user=Skye (END - exit=0) [20:16:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.87, 7.01, 5.37 [20:24:15] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=sempuwiki (END - exit=0) [20:24:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:21] [02mw-config] 07c-oreills commented on pull request #6113: It seems like this hasn't had the intended effect. I enabled it for the main namespace on my wiki and am still getting sections collapsing on mobile for e.g. [the main page](https://www.twistwoodtales.wiki/wiki/Main_Page). […] 13https://github.com/miraheze/mw-config/pull/6113#issuecomment-3316246528 [20:44:54] !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [20:44:55] !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [20:44:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:11] [02mw-config] 07pskyechology commented on pull request #6113: @c-oreills You're most likely facing caching issues, as the main namespace does not have collapsible sections on my end. Try in incognito? 13https://github.com/miraheze/mw-config/pull/6113#issuecomment-3316255029 [20:55:44] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php purgeParserCache --wiki=thechurchofthestatuewiki --age=300 (END - exit=0) [20:55:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:56] PROBLEM - db161 Disk Space on db161 is WARNING: DISK WARNING - free space: / 93855MiB (10% inode=98%); [22:10:26] [02RottenLinks] 07Universal-Omega created 03REL1_43 from 03main (+0 new commit) 13https://github.com/miraheze/RottenLinks/compare/REL1_43 [22:10:49] [02RottenLinks] 07Universal-Omega 04deleted 03REL1_43 at 0353016b7 13https://api.github.com/repos/miraheze/RottenLinks/commit/53016b7 [22:11:17] [02RottenLinks] 07Universal-Omega created 03REL1_43 at 0309b632b (+0 new commit) 13https://github.com/miraheze/RottenLinks/compare/REL1_43 [22:16:42] [02RottenLinks] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/d6cfbcb5320abc568868823a4f0c1723e13de6b1 [22:16:44] 02RottenLinks/03main 07CosmicAlpha 03d6cfbcb Replace class aliases and use ISQLPlatform::ALL_ROWS [22:18:59] [02RottenLinks] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/9e7907faf9c148bbc91c0e1cd33e885bac871735 [22:18:59] 02RottenLinks/03main 07CosmicAlpha 039e7907f Fix date [22:23:58] miraheze/RottenLinks - Universal-Omega the build passed.