[00:07:27] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56777MiB (12% inode=99%); [00:07:33] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57197MiB (12% inode=99%); [00:08:13] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56466MiB (12% inode=99%); [00:35:01] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw191 [00:36:59] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [00:48:27] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 7.55, 12.93, 8.42 [00:50:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.77, 8.92, 7.49 [03:12:27] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [03:12:56] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756072610000&orgId=1&to=1756091510000 [06:39:37] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:39:45] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [06:39:50] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:51] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [06:39:54] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:39:55] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:39:57] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.016 second response time [06:39:57] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:40:01] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [06:40:05] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [06:40:07] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:40:09] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:40:10] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:18] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [06:40:19] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:40:20] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:40:21] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:25] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [06:40:26] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:40:26] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:40:29] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:40:31] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:40:36] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [06:40:40] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:40:41] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [06:40:41] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:40:41] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:40:59] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:41:02] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:03] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:05] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:41:07] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:41:09] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:20] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:41:20] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:41:21] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 223.24, 95.88, 38.06 [06:41:21] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:35] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:41:38] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:41:39] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:41] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:48] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [06:41:52] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:52] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [06:43:09] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:44:18] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 7.462 second response time [06:44:18] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.172 second response time [06:44:19] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.937 second response time [06:44:21] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: [06:44:21] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.060 second response time [06:44:21] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 6.530 second response time [06:44:22] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time [06:44:30] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.173 second response time [06:44:32] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.066 second response time [06:44:32] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.057 second response time [06:44:33] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.070 second response time [06:44:33] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [06:44:34] PROBLEM - db181 MariaDB on db181 is UNKNOWN: [06:44:35] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:44:40] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.228 second response time [06:44:42] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.064 second response time [06:44:47] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.075 second response time [06:44:50] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.059 second response time [06:44:52] PROBLEM - db181 Puppet on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:44:59] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.060 second response time [06:45:02] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.065 second response time [06:45:03] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [06:45:05] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [06:45:11] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.063 second response time [06:45:14] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [06:45:20] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.062 second response time [06:45:25] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.066 second response time [06:45:27] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time [06:45:33] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4216 bytes in 0.065 second response time [06:45:36] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.059 second response time [06:45:42] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [06:45:42] !log reboot db181 [06:45:43] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.069 second response time [06:45:44] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.161 second response time [06:45:47] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.199 second response time [06:45:48] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [06:45:49] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [06:45:52] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [06:45:52] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.159 second response time [06:45:57] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [06:45:59] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.059 second response time [06:45:59] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.058 second response time [06:46:01] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.165 second response time [06:46:03] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [06:46:08] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [06:46:11] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:46:20] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.069 second response time [06:46:32] PROBLEM - db181 MariaDB on db181 is CRITICAL: Can't connect to server on 'db181.fsslc.wtnet' (115) [06:46:41] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [06:47:01] PROBLEM - db181 ferm_active on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:47:12] PROBLEM - db181 NTP time on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:47:15] PROBLEM - db181 SSH on db181 is CRITICAL: connect to address 10.0.18.102 and port 22: Connection refused [06:47:27] PROBLEM - db181 Check unit status of sql-backup on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:47:59] PROBLEM - db181 conntrack_table_size on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:48:13] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:48:48] PROBLEM - db181 Disk Space on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [06:52:08] RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:52:13] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.072 seconds response time. db181.fsslc.wtnet returns 10.0.18.102 [06:52:30] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [06:52:48] RECOVERY - db181 Disk Space on db181 is OK: DISK OK - free space: / 439254MiB (49% inode=98%); [06:52:52] RECOVERY - db181 APT on db181 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [06:53:00] RECOVERY - db181 ferm_active on db181 is OK: OK ferm input default policy is set [06:53:02] RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset -0.0004938542843 secs [06:53:20] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.85, 0.52, 0.18 [06:53:27] RECOVERY - db181 Check unit status of sql-backup on db181 is OK: OK: Status of the systemd unit sql-backup [06:54:04] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 1%Current connections: 10 [06:54:05] RECOVERY - db181 conntrack_table_size on db181 is OK: OK: nf_conntrack is 0 % full [06:54:22] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 134 Threads: 14 Questions: 205204 Slow queries: 0 Opens: 8254 Open tables: 8198 Queries per second avg: 1531.373 [06:56:08] !log reboot db181 [06:56:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:13] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49842MiB (10% inode=99%); [08:00:27] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [08:01:27] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49864MiB (10% inode=99%); [08:31:33] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49947MiB (10% inode=99%); [11:45:33] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6174a4aa67bf53666a53ea41ffdb32bfbe2fefe3 [11:45:33] 02ssl/03main 07WikiTideBot 036174a4a Bot: Auto-update domain lists [12:12:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756120310000&orgId=1&to=1756123940011 [12:23:17] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/3a49619bc5eb11d70da20e1c479506424f40f37f [12:23:17] 02DiscordNotifications/03main 07translatewiki.net 033a49619 Localisation updates from https://translatewiki.net. [12:23:18] [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/91d1f19a960a77fb4242b82aeeb1b50b8d889dde [12:23:20] 02ImportDump/03main 07translatewiki.net 0391d1f19 Localisation updates from https://translatewiki.net. [12:23:22] [02IncidentReporting] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/1a56926fc444ac50f3b3cb4b87c64c1464e4c25d [12:23:24] [02RequestCustomDomain] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RequestCustomDomain/commit/d0e4157d2f3845ebd8efef0fa89121bd38334d7f [12:23:25] 02RequestCustomDomain/03main 07translatewiki.net 03d0e4157 Localisation updates from https://translatewiki.net. [12:23:27] 02IncidentReporting/03main 07translatewiki.net 031a56926 Localisation updates from https://translatewiki.net. [12:23:29] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/74bb5437d0e0d74ed843d1c08fc125412aa15b7a [12:23:31] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/430be689279c6ef039785e43c5808ad071269ea4 [12:23:33] 02ManageWiki/03main 07translatewiki.net 0374bb543 Localisation updates from https://translatewiki.net. [12:23:34] 02MirahezeMagic/03main 07translatewiki.net 03430be68 Localisation updates from https://translatewiki.net. [12:25:37] miraheze/DiscordNotifications - translatewiki the build has errored. [12:25:42] miraheze/IncidentReporting - translatewiki the build has errored. [12:25:49] miraheze/ImportDump - translatewiki the build has errored. [12:25:53] miraheze/RequestCustomDomain - translatewiki the build has errored. [12:25:59] miraheze/ManageWiki - translatewiki the build has errored. [12:26:06] miraheze/MirahezeMagic - translatewiki the build has errored. [12:42:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756120310000&orgId=1&to=1756125710000 [15:45:35] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/9fdfbaf4d3d761ee26ec24ae4b53790318b1214a [15:45:35] 02ssl/03main 07WikiTideBot 039fdfbaf Bot: Auto-update domain lists [15:45:59] miraheze/MirahezeMagic - translatewiki the build passed. [15:55:33] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/ef1e06e1faf7f22fb86ded45b29cd3874ee7b45c [15:55:34] 02ssl/03main 07WikiTideBot 03ef1e06e Bot: Auto-update domain lists [16:15:47] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/30cf7f15c8e042f9d8309fdd62c2e0d06d5ca9a1 [16:15:47] 02ssl/03main 07WikiTideBot 0330cf7f1 Bot: Auto-update domain lists [16:27:20] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756135610000&orgId=1&to=1756139240011 [16:43:22] PROBLEM - farthestfrontier.wiki - Cloudflare on sslhost is WARNING: WARNING - Certificate 'farthestfrontier.wiki' expires in 28 day(s) (Tue 23 Sep 2025 04:23:30 PM GMT +0000). [16:53:19] [02MirahezeMagic] 07Universal-Omega created 03cleanup (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/79dc88cf9dfc [16:53:20] 02MirahezeMagic/03cleanup 07CosmicAlpha 0379dc88c Some cleanup [16:53:26] [02MirahezeMagic] 07Universal-Omega opened pull request #590: Some cleanup (03main...03cleanup) 13https://github.com/miraheze/MirahezeMagic/pull/590 [16:53:34] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #590:

[…] 13https://github.com/miraheze/MirahezeMagic/pull/590#issuecomment-3221010055 [16:54:12] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/957cd50398c4795cf1bb71e71640a95e765efee2 [16:54:12] 02MirahezeMagic/03cleanup 07CosmicAlpha 03957cd50 Update [16:55:48] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/391a4bdd0d6cce66853063ab9b465b91ef5b8108 [16:55:48] 02MirahezeMagic/03cleanup 07CosmicAlpha 03391a4bd - [16:56:47] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/21e9b1e9dbec68be3ef43792f83796cb42ecff94 [16:56:47] 02MirahezeMagic/03cleanup 07CosmicAlpha 0321e9b1e - [17:00:51] miraheze/MirahezeMagic - Universal-Omega the build has errored. [17:15:16] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/50bfbf719809e2f159b99d361817b3af0aea5c6b [17:15:16] 02MirahezeMagic/03cleanup 07CosmicAlpha 0350bfbf7 Update [17:15:45] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/f24161c0e4c5663930089a4ed900384d7c29190d [17:15:45] 02MirahezeMagic/03cleanup 07CosmicAlpha 03f24161c Remove [17:16:08] miraheze/MirahezeMagic - Universal-Omega the build has errored. [17:19:31] miraheze/MirahezeMagic - Universal-Omega the build has errored. [17:21:50] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/d40f0e1ccff96af991f857e0ad76ea46b69d02a1 [17:21:50] 02MirahezeMagic/03cleanup 07CosmicAlpha 03d40f0e1 Fix [17:29:36] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=scribblepediawiki dump.xml --no-updates (START) [17:29:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:26] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03cleanup 13https://github.com/miraheze/MirahezeMagic/commit/d3dec09e719268541ab23541978f6b275c6bbbe1 [17:32:26] 02MirahezeMagic/03cleanup 07CosmicAlpha 03d3dec09 Update [17:45:09] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:51:45] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=scribblepediawiki dump.xml --no-updates (END - exit=0) [17:51:46] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=scribblepediawiki (START) [17:52:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:43] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:19:05] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:21:50] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:22:11] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:23:06] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:27:35] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:30:24] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:31:12] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:32:43] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:34:27] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:34:28] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 70 packages available for upgrade (9 critical updates). [18:34:29] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:34:46] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:35:30] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/d2d60b4220c8077fef8edea9ecec862154345113 [18:35:30] 02ssl/03main 07WikiTideBot 03d2d60b4 Bot: Auto-update domain lists [18:36:18] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:37:04] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:40:18] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:40:27] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:42:03] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:42:10] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:42:58] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:43:35] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:44:25] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [18:45:13] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 87 packages available for upgrade (9 critical updates). [19:18:09] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1966MiB (10% inode=90%); [19:21:51] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=scribblepediawiki (END - exit=0) [19:21:53] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=scribblepediawiki --update (END - exit=0) [19:22:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756135610000&orgId=1&to=1756149710000 [19:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:09] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6680MiB (37% inode=89%); [21:31:08] !log [void@puppet181] Upgraded packages on mw151: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:31:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:26] !log [void@puppet181] Upgraded packages on mw163: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:31] thank you void <3 [21:31:44] !log [void@puppet181] Upgraded packages on mw193: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:31:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:01] RECOVERY - mw151 APT on mw151 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:32:03] !log [void@puppet181] Upgraded packages on mw153: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:32:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:22] !log [void@puppet181] Upgraded packages on mw203: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:32:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:32:39] !log [void@puppet181] Upgraded packages on mw201: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:00] !log [void@puppet181] Upgraded packages on mw202: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:33:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:05] RECOVERY - mw193 APT on mw193 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:33:10] RECOVERY - mw153 APT on mw153 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:33:12] RECOVERY - mw163 APT on mw163 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:33:23] !log [void@puppet181] Upgraded packages on mw182: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:35] RECOVERY - mw203 APT on mw203 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:33:40] !log [void@puppet181] Upgraded packages on mwtask181: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:33:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:54] RECOVERY - mw201 APT on mw201 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:34:07] !log [void@puppet181] Upgraded packages on mw162: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:34:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:23] RECOVERY - mw202 APT on mw202 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:34:23] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:34:26] !log [void@puppet181] Upgraded packages on test151: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:34:28] RECOVERY - test151 APT on test151 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [21:34:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:47] !log [void@puppet181] Upgraded packages on mw172: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:34:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:03] RECOVERY - mw182 APT on mw182 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:35:06] !log [void@puppet181] Upgraded packages on mw161: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:35:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:26] !log [void@puppet181] Upgraded packages on mw173: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:35:27] RECOVERY - mw162 APT on mw162 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:35:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:44] !log [void@puppet181] Upgraded packages on mw171: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:35:46] RECOVERY - mw171 APT on mw171 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:35:57] RECOVERY - mw173 APT on mw173 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:36:03] RECOVERY - mw172 APT on mw172 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:36:06] !log [void@puppet181] Upgraded packages on mwtask161: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:36:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:22] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:36:29] !log [void@puppet181] Upgraded packages on mwtask171: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:36:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:46] !log [void@puppet181] Upgraded packages on mw152: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:36:57] RECOVERY - mw161 APT on mw161 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:37:08] !log [void@puppet181] Upgraded packages on mw181: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:37:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:32] !log [void@puppet181] Upgraded packages on mwtask151: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:57] !log [void@puppet181] Upgraded packages on mw191: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:38:00] RECOVERY - mw181 APT on mw181 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:10] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:38:20] !log [void@puppet181] Upgraded packages on mw192: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:38:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:38:35] RECOVERY - mw152 APT on mw152 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:38:40] !log [void@puppet181] Upgraded packages on mw183: libswscale6, libavdevice59, libavformat59, libavfilter8, libavcodec59, libavutil57, libpostproc56, libswresample4, and ffmpeg [21:38:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:10] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:39:34] RECOVERY - mw191 APT on mw191 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:39:51] RECOVERY - mw192 APT on mw192 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:40:11] RECOVERY - mw183 APT on mw183 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [21:52:12] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03mw-hooks 13https://github.com/miraheze/MirahezeMagic/commit/5ba04c6542a085f75341f7f9fee3dddcd1c53c39 [21:52:12] 02MirahezeMagic/03mw-hooks 07CosmicAlpha 035ba04c6 Fix using primary domain [21:53:08] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03mw-hooks 13https://github.com/miraheze/MirahezeMagic/commit/772b4d4e91137be023bb3279d607a0a890f02adc [21:53:08] 02MirahezeMagic/03mw-hooks 07CosmicAlpha 03772b4d4 Add use [21:55:19] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03mw-hooks 13https://github.com/miraheze/MirahezeMagic/commit/b14861302f037dbe8a2d02c31b0db1721c2095e2 [21:55:19] 02MirahezeMagic/03mw-hooks 07CosmicAlpha 03b148613 Fix [22:08:01] miraheze/MirahezeMagic - Universal-Omega the build passed. [22:48:51] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=reviwiki --startOver (END - exit=0) [22:49:23] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=reviwiki (END - exit=0) [22:49:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:33] PROBLEM - farthestfrontier.wiki - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds