[01:47:00] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 11.83, 4.98 [01:47:07] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:55] did the scrapers find phorge again [01:48:24] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:48:51] PROBLEM - phorge171 APT on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:49:23] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:31] PROBLEM - phorge171 PowerDNS Recursor on phorge171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:51:43] PROBLEM - phorge171 SSH on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:43] RECOVERY - phorge171 SSH on phorge171 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [01:55:26] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 18207 bytes in 3.562 second response time [01:55:32] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17668 bytes in 2.997 second response time [01:55:52] who hurt you girl [01:56:55] PROBLEM - phorge171 conntrack_table_size on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:59:25] any @Infrastructure Specialists around to give phorge a nudge? [01:59:51] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:55] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:00] PROBLEM - phorge171 SSH on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:03] 👀 [02:01:57] PROBLEM - phorge171 ferm_active on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:00] PROBLEM - phorge171 Check unit status of phorge_images-backup on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:06] PROBLEM - phorge171 Check unit status of phorge_db-backup on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:02:24] PROBLEM - phorge171 Check unit status of phd on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:07:33] RECOVERY - phorge171 Check unit status of phorge_images-backup on phorge171 is OK: OK: Status of the systemd unit phorge_images-backup [02:07:34] RECOVERY - phorge171 Check unit status of phorge_db-backup on phorge171 is OK: OK: Status of the systemd unit phorge_db-backup [02:07:34] RECOVERY - phorge171 Check unit status of phd on phorge171 is OK: OK: Status of the systemd unit phd [02:07:34] RECOVERY - phorge171 ferm_active on phorge171 is OK: OK ferm input default policy is set [02:07:34] RECOVERY - phorge171 PowerDNS Recursor on phorge171 is OK: DNS OK: 0.978 seconds response time. phorge171.fsslc.wtnet returns 10.0.17.124 [02:07:58] RECOVERY - phorge171 conntrack_table_size on phorge171 is OK: OK: nf_conntrack is 0 % full [02:08:04] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 53 minutes ago with 0 failures [02:08:06] RECOVERY - phorge171 SSH on phorge171 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [02:08:07] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 18207 bytes in 0.350 second response time [02:12:02] PROBLEM - phorge171 PowerDNS Recursor on phorge171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:12:32] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:41] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 81 packages available for upgrade (0 critical updates). [02:20:41] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 18207 bytes in 0.075 second response time [02:20:41] RECOVERY - phorge171 PowerDNS Recursor on phorge171 is OK: DNS OK: 0.047 seconds response time. phorge171.fsslc.wtnet returns 10.0.17.124 [02:20:41] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17668 bytes in 0.034 second response time [02:21:04] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.54, 0.33, 0.14 [02:21:36] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet last ran 1 hour ago [02:21:38] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:36:58] !log [void@cloud17] reboot phorge171 [02:37:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:07] !log [skye@mwtask171] Starting import for aniphobiawiki (XML: None; Images: .) (START) [03:00:08] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=aniphobiawiki --sleep=0 '--comment=Importing files from https://aniphobia.fandom.com/ ([[phorge:T15474|T15474]])' -- . (START) [03:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:27] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=aniphobiawiki --sleep=0 '--comment=Importing files from https://aniphobia.fandom.com/ ([[phorge:T15474|T15474]])' -- . (END - exit=0) [03:00:28] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=aniphobiawiki --update (START) [03:00:29] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php initSiteStats --wiki=aniphobiawiki --update (END - exit=0) [03:00:30] !log [skye@mwtask171] Finished import for aniphobiawiki (XML: None; Images: .) (END - exit=0) [03:00:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:45:34] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.56, 6.54, 5.35 [07:46:45] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.61, 6.68, 5.60 [07:47:33] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.46, 6.07, 5.35 [07:48:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 3.34, 5.57, 5.34 [10:31:23] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780479050000&orgId=1&to=1780482683351 [10:35:38] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.19, 5.63, 4.95 [10:37:37] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.60, 5.72, 5.07 [11:21:23] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780481870000&orgId=1&to=1780485470000 [11:59:23] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780484330000&orgId=1&to=1780487963352 [12:14:23] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780484990000&orgId=1&to=1780488590000 [12:27:29] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 96.68 ms [12:27:32] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 49.17 ms [12:29:01] PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 96.63 ms [12:29:32] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.34 ms [12:29:34] RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [12:31:05] RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.49 ms [12:41:55] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 96.74 ms [12:42:49] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 96.67 ms [12:43:31] PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 96.38 ms [12:45:01] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 96.52 ms [12:45:06] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.45],Exec[git_pull_3d2png] [12:46:25] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [12:47:53] PROBLEM - os201 Puppet on os201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:48:04] PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media [12:48:15] PROBLEM - ping6 on cloud21 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 96.70 ms [12:48:20] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 1.416 second response time [12:48:47] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 96.68 ms [12:49:00] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 96.67 ms [12:49:51] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.12, 6.86, 5.90 [12:49:58] PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 96.84 ms [12:50:04] RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media [12:50:46] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:51:51] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.50, 6.41, 5.86 [12:52:22] RECOVERY - ping6 on cloud21 is OK: PING OK - Packet loss = 0%, RTA = 96.92 ms [12:53:45] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [12:54:03] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:55:05] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 48.52 ms [12:55:57] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 96.59 ms [12:56:04] PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media [12:56:14] RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 97.22 ms [12:56:32] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 96.69 ms [12:59:47] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.45, 6.52, 5.93 [12:59:56] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms [13:00:02] RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [13:00:04] RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media [13:00:45] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.16, 6.47, 5.99 [13:01:14] RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [13:03:46] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.96, 6.28, 5.99 [13:04:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.49, 6.39, 6.07 [13:06:12] PROBLEM - ping6 on cloud20 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 97.10 ms [13:06:50] PROBLEM - ping6 on cloud19 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 96.72 ms [13:07:16] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 96.68 ms [13:08:13] RECOVERY - ping6 on cloud20 is OK: PING OK - Packet loss = 0%, RTA = 82.49 ms [13:08:45] PROBLEM - os202 Current Load on os202 is CRITICAL: LOAD CRITICAL - total load average: 8.28, 7.01, 6.36 [13:08:53] RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 44.93 ms [13:09:14] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 25.09 ms [13:10:45] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.64, 7.15, 6.50 [13:11:43] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.18, 7.44, 6.60 [13:12:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.48, 6.74, 6.44 [13:13:08] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:13:42] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.37, 5.83, 6.12 [13:15:53] RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:19:23] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780489130000&orgId=1&to=1780492763351 [13:19:40] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.73, 7.31, 6.66 [13:20:45] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 6.85, 6.66, 6.35 [13:20:45] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:21:39] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.22, 6.61, 6.49 [13:21:52] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:22:03] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:22:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.16, 6.28, 6.24 [13:49:23] [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780490810000&orgId=1&to=1780494563353[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780489130000&orgId=1&to=1780494563353 [14:16:34] can we murder smw already [15:19:09] I can agree [15:19:19] I didn't realise it put stuff in the jobqueue tbh [15:19:24] but I can imagine [15:19:50] I don't even know how many jobs are in the jobqueue on WO [15:59:21] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=evnovawiki dump.xml --no-updates (START) [15:59:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/20be8bd881c54d5433a829376bedc7716e649417 [16:00:20] 02ssl/03main 07WikiTideBot 0320be8bd Bot: Auto-update domain lists [16:18:45] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.43, 5.91, 4.97 [16:20:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.50, 5.77, 5.03 [16:29:23] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780490810000&orgId=1&to=1780504010000[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780489130000&orgId=1&to=1780504163378 [16:29:43] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/importDump.php --wiki=evnovawiki dump.xml --no-updates (END - exit=0) [16:29:44] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/rebuildall.php --wiki=evnovawiki (START) [16:29:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:45] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.46, 6.11, 5.20 [16:48:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.62, 5.83, 5.20 [16:56:55] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/rebuildall.php --wiki=evnovawiki (END - exit=0) [16:56:57] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/initSiteStats.php --wiki=evnovawiki --update (END - exit=0) [16:56:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:32] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.23, 6.66, 5.55 [16:58:45] PROBLEM - os202 Current Load on os202 is CRITICAL: LOAD CRITICAL - total load average: 9.70, 7.65, 6.17 [17:00:45] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 4.88, 6.77, 6.05 [17:01:30] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.61, 5.73, 5.51 [17:03:05] [02ssl] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/fba9325c706a494a770545bf65409fa18a10b441 [17:03:05] 02ssl/03main 07RhinosF1 03fba9325 Delete outlasterwiki redirect from redirects.yaml… [17:26:21] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.32, 6.51, 5.46 [17:27:10] PROBLEM - os202 Current Load on os202 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 6.71, 5.73 [17:28:21] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.26, 6.25, 5.51 [17:29:07] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 3.79, 5.51, 5.41 [17:37:00] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.21, 6.38, 5.74 [17:38:54] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 6.89, 6.32, 5.74 [17:38:59] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.11, 6.25, 5.77 [17:40:50] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.59, 6.38, 5.83 [17:45:22] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=fantendowiki . --comment='Importing images from fantendo.fandom.com ([[phorge:T15427|T15427]])' --skip-dupes (END - exit=256) [17:45:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:46:35] 256?? tf happened [17:48:13] nah its a happy ending [18:54:10] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.35, 6.07, 5.15 [18:56:08] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 2.68, 4.59, 4.72 [19:27:42] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.67, 6.50, 5.89 [19:29:41] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.02, 5.39, 5.55 [19:45:27] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.32, 6.31, 5.55 [19:47:26] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.71, 5.86, 5.49 [19:58:44] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.67, 6.95, 6.04 [20:00:13] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 9.00, 7.47, 6.30 [20:02:12] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.64, 7.22, 6.36 [20:04:10] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.37, 6.74, 6.30 [20:04:44] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.20, 6.77, 6.31 [20:10:36] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=fantendowiki . --comment='Importing gifs from fantendo.fandom.com ([[phorge:T15427|T15427]])' --skip-dupes (START) [20:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:24] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importImages --wiki=fantendowiki . --comment='Importing gifs from fantendo.fandom.com ([[phorge:T15427|T15427]])' --skip-dupes (END - exit=0) [20:13:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:15] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php updateSpecialPages --wiki=fantendowiki --only=MediaStatistics (END - exit=0) [20:15:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:30] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.13, 6.21, 5.97 [20:17:59] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.42, 6.64, 6.07 [20:19:26] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 4.81, 5.57, 5.76 [20:19:57] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.47, 5.61, 5.75 [20:29:23] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780489130000&orgId=1&to=1780518563415 [21:24:23] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780518230000&orgId=1&to=1780521830000 [21:43:23] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780519370000&orgId=1&to=1780523003350 [22:03:23] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780520450000&orgId=1&to=1780524050000 [22:19:53] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=evnovawiki --delete --ns=0 --ns=1 --ns=2 --ns=3 --ns=4 --ns=5 --ns=6 --ns=7 --ns=8 --ns=9 --ns=10 --ns=11 --ns=12 --ns=13 --ns=14 --ns=15 --all (START) [22:19:54] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=evnovawiki --delete --ns=0 --ns=1 --ns=2 --ns=3 --ns=4 --ns=5 --ns=6 --ns=7 --ns=8 --ns=9 --ns=10 --ns=11 --ns=12 --ns=13 --ns=14 --ns=15 --all (END - exit=256) [22:19:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:08] awh [22:20:23] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1780521590000&orgId=1&to=1780525223353 [22:42:56] !log for i in {0..15}; do mwscript nukeNS evnovawiki --delete --ns=${i} --all --y --no-log; done [22:43:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:24] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=evnovawiki --delete --ns=828 --all (START) [22:43:26] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=evnovawiki --delete --ns=828 --all (END - exit=0) [22:43:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:30] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=evnovawiki --delete --ns=829 --all (START) [22:43:32] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php nukeNS --wiki=evnovawiki --delete --ns=829 --all (END - exit=0) [22:43:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:43:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:46:08] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php purgeOldText --wiki=evnovawiki --purge (END - exit=0) [22:46:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:57] db reset lite cause caching a bitch [22:49:52] !log [skye@mwtask171] evnovawiki: DELETE FROM text; [22:49:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:33] what about parser cache? pretty sure that's got some db stuff too [23:19:40] [1/2] people smarter than me have said db resets dont clear out objectstash which results in issues like https://issue-tracker.miraheze.org/T15452#309084 [23:19:40] [2/2] see https://issue-tracker.miraheze.org/T14878#298267 [23:20:24] at the time of creation of the latter task it only happened occasionally but now seems to happen with every single wiki [23:21:12] several tables also dont get properly recreated making https://meta.miraheze.org/wiki/Tech:Fixing_slot_roles_and_content_models and https://meta.miraheze.org/wiki/Tech:Matomo#Wikis_using_a_different_SiteID hard requirements for a functional wiki [23:26:22] !log [skye@mwtask171] Starting import for evnovawiki (XML: evn_pages_full.xml; Images: images) (START) [23:26:23] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php importDump --wiki=evnovawiki --no-updates --username-prefix=fandom:evn -- evn_pages_full.xml (START) [23:26:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:22] yeah every time I reset a wiki I have to run `fix_container_permissions.py` and fix matomo. And often I have to fix content models and slot roles