[00:03:24] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 7064MiB (39% inode=90%); [00:03:54] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [00:05:20] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57447MiB (12% inode=99%); [00:07:09] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57887MiB (12% inode=99%); [00:07:37] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57152MiB (12% inode=99%); [03:11:25] miraheze/CreateWiki - Universal-Omega the build passed. [03:25:04] miraheze/RequestSSL - Universal-Omega the build passed. [04:51:49] miraheze/puppet - AgentIsai the build has errored. [04:54:24] miraheze/puppet - AgentIsai the build has errored. [04:57:55] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [04:58:07] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [04:58:51] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [04:58:53] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [04:59:02] miraheze/puppet - AgentIsai the build has errored. [04:59:18] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [04:59:18] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [04:59:32] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:00:32] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:00:36] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:00:57] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:00:58] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:01:10] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:01:18] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:01:27] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:01:52] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:05:17] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:06:12] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:07:00] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:11:51] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:13:41] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:13:41] miraheze/mw-config - AgentIsai the build passed. [05:20:53] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:24:56] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:25:20] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 404 [05:33:50] miraheze/ErrorPages - AgentIsai the build passed. [05:36:01] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 410 [05:36:42] miraheze/ErrorPages - AgentIsai the build passed. [05:36:54] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [05:37:05] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 410 [05:38:02] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 410 [05:38:13] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 410 [05:44:30] miraheze/puppet - AgentIsai the build has errored. [05:46:53] !log [agent@mwtask181] starting deploy of {'config': True} to mw181 [05:46:55] !log [agent@mwtask181] finished deploy of {'config': True} to mw181 - SUCCESS in 1s [05:46:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:40] miraheze/RequestSSL - AgentIsai the build passed. [06:03:00] miraheze/puppet - AgentIsai the build has errored. [06:05:03] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.054 second response time [06:05:05] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.058 second response time [06:05:13] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.069 second response time [06:05:14] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:05:33] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:05:36] 410???? [06:05:43] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:05:51] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.070 second response time [06:05:55] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.059 second response time [06:06:06] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:06:09] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.320 second response time [06:06:12] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.066 second response time [06:06:13] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.067 second response time [06:06:19] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.058 second response time [06:06:27] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.269 second response time [06:06:33] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.257 second response time [06:06:34] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.059 second response time [06:06:35] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.061 second response time [06:06:50] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.060 second response time [06:06:50] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4116 bytes in 0.262 second response time [06:25:20] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.43, 17.47, 12.73 [06:29:20] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.95, 18.03, 14.15 [06:35:06] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.074 second response time [06:35:15] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.058 second response time [06:36:48] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4112 bytes in 0.071 second response time [06:49:20] miraheze/puppet - AgentIsai the build has errored. [06:49:56] miraheze/puppet - AgentIsai the build has errored. [06:55:17] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.49, 19.32, 17.07 [06:57:17] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.46, 17.99, 16.86 [07:06:00] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:06:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:26] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [07:06:27] miraheze/mw-config - Universal-Omega the build passed. [07:06:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:38] !log [blankeclair@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'TemplateStylesExtender'} to all [07:06:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:38] miraheze/puppet - AgentIsai the build has errored. [07:15:04] !log [blankeclair@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'TemplateStylesExtender'} to all - SUCCESS in 505s [07:15:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:15:27] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:27:52] !log [agent@mwtask181] starting deploy of {'config': True} to all [07:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:17] !log [agent@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 24s [07:28:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:59] !log [agent@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44'} to all [07:34:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:13] !log [agent@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [07:34:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:22] !log [agent@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'RequestSSL'} to all [07:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:46] !log [agent@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 24s [07:34:49] !log [agent@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44'} to all [07:34:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:58] !log [blankeclair@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'DataDump'} to all [07:43:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:06] !log [agent@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44'} to all - SUCCESS in 497s [07:43:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:13] miraheze/DataDump - BlankEclair the build passed. [07:43:18] lol [07:43:23] !log [blankeclair@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'DataDump'} to all - SUCCESS in 24s [07:43:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:28] !log [agent@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43'} to all [07:43:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:43:40] sir, i already ran l10n [07:47:05] miraheze/DataDump - BlankEclair the build passed. [07:51:22] !log [agent@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43'} to all - SUCCESS in 475s [07:51:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:16:48] Agent was running it again after you did because RequestSSL was just upgraded to add automation(!), so it needed to run i18n again to run for new messages as well. [09:16:58] ah fair lol [10:34:50] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49839MiB (10% inode=99%); [11:01:28] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49876MiB (10% inode=99%); [11:29:54] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49940MiB (10% inode=99%); [12:25:53] miraheze/ManageWiki - translatewiki the build passed. [13:06:50] !log [skye@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44'} to all [13:06:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:08:18] !log [skye@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44'} to all - SUCCESS in 88s [13:08:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:08:46] 88s? [13:08:54] it usually takes like 10 minutes [13:11:25] what the shit [13:31:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754224280000&orgId=1&to=1754227913357 [13:51:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754225360000&orgId=1&to=1754228960000 [14:06:04] !log [skye@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': '1.44'} to all [14:06:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:16:02] !log [skye@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': '1.44'} to all - SUCCESS in 598s [14:16:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:00] !log [skye@mwtask181] starting deploy of {'extension_list': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all [14:25:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:47] !log [skye@mwtask181] finished deploy of {'extension_list': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 47s [14:25:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:40] i've thrown all the spaghetti i have at the wall [15:56:01] it's buggered [16:06:53] i figured but if i have the spaghetti, might as well throw it [16:06:53] it should be split on version https://github.com/miraheze/mw-config/blob/5b7749904f38952a1bb303d146889b6b315c7e62/LocalSettings.php#L7174 [16:06:53] [GitHub] [miraheze/mw-config] LocalSettings.php @ 5b7749904f38952a1bb303d146889b6b315c7e62 | L7174: $wgLocalisationCacheConf['storeDirectory'] = '/srv/mediawiki/cache/' . $wi->version . '/l10n'; [16:07:04] i have a thought [16:07:25] @pskyechology name a wiki on nexttide [16:07:39] [1/2] eutwiki [16:07:40] [2/2] lhmnwiki [16:07:52] actually the first one is [16:07:59] everythingupgradetreewiki [16:08:34] oh yeah because it used testwiki and its on 1.43 [16:08:41] more spaghetti! [16:08:49] Error 1049: Unknown database 'eutwiki' [16:09:05] . i forgot [16:09:12] ok [16:09:28] I am running the 1.44 version of rebuildL10n from everythingupgradetreewiki [16:09:50] actually i should mesagefiles first [16:11:44] done [16:12:04] it's an amazingly long time since I had to actually think about how to call a maint script without mwscript @pskyechology [16:12:15] I just did it for the OATHAuth one per tech@ [16:12:26] there's a reason we have mwscript and mwdeploy [16:13:37] what's the exact command so i can reference it a few months later [16:13:40] it's hard actually thinking about how to do my job [16:13:55] @pskyechology sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildLocalisationCache.php --wiki=everythingupgradetreewiki --force [16:14:09] there's a reason i wrote automation [16:15:18] @pskyechology i'm going for a walk whether this works or not but it's a guess [16:15:33] I should add support to mwscript for varying the wiki for --version [16:15:51] someone make a task so i can add it to the endless backlog [16:17:56] !log [rhinos@mwtask181] starting deploy of {'folders': '/srv/mediawiki/cache/1.44/l10n'} to all [16:17:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:15] !log [rhinos@mwtask181] starting deploy of {'folders': '../.../srv/mediawiki/cache/1.44/l10n'} to all [16:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:43] !log [rhinos@mwtask181] starting deploy of {'folders': '../../srv/mediawiki/cache/1.44/l10n'} to all [16:18:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:09] @pskyechology is it bad that ^ works [16:21:59] !log [rhinos@mwtask181] finished deploy of {'folders': '../../srv/mediawiki/cache/1.44/l10n'} to all - SUCCESS in 196s [16:22:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:12] @pskyechology my hammer beating is done [16:22:23] hopefully i at least generated different errors [18:20:38] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03task 13https://github.com/miraheze/CreateWiki/commit/9a6569ce332bd40dd5a92a2b1dff63e790bcccb7 [18:20:38] 02CreateWiki/03task 07CosmicAlpha 039a6569c Merge branch 'main' into task [18:25:08] miraheze/CreateWiki - Universal-Omega the build has errored. [18:37:31] miraheze/CreateWiki - Universal-Omega the build passed. [18:46:12] [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03task 13https://github.com/miraheze/CreateWiki/commit/49b044cb87066e950dd66cc96ac6cc598db2447a [18:46:13] 02CreateWiki/03task 07CosmicAlpha 0349b044c Add throws [18:59:58] !log MariaDB [metawiki]> ALTER TABLE cw_requests MODIFY COLUMN cw_extra JSON NULL; [19:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:10] !log MariaDB [mhglobal]> ALTER TABLE cw_wikis MODIFY COLUMN wiki_extra JSON NULL; [19:01:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:14] [02puppet] 07Universal-Omega merged pull request #4459: db182: upgrade to MariaDB 11.8 (03main...03db182-mariadb-11.8) 13https://github.com/miraheze/puppet/pull/4459 [19:04:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b69cfc0879f49b3c0450a9b5c8c15f14fbf76856 [19:04:14] 02puppet/03main 07CosmicAlpha 03b69cfc0 db182: upgrade to MariaDB 11.8 [19:04:16] [02puppet] 07Universal-Omega 04deleted 03db182-mariadb-11.8 at 030f1f09b 13https://api.github.com/repos/miraheze/puppet/commit/0f1f09b [19:06:07] miraheze/CreateWiki - Universal-Omega the build passed. [19:10:26] PROBLEM - db182 MariaDB on db182 is CRITICAL: Can't connect to server on 'db182.fsslc.wtnet' (115) [19:10:59] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-2 [19:11:01] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-1 [19:12:26] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 121 Threads: 12 Questions: 277064 Slow queries: 0 Opens: 187 Open tables: 180 Queries per second avg: 2289.785 [19:12:59] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-2 [19:13:01] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-1 [19:13:24] [02puppet] 07Universal-Omega created 03mon181-mariadb-11.8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/b7983f5aa2ff [19:13:24] 02puppet/03mon181-mariadb-11.8 07CosmicAlpha 03b7983f5 mon181: upgrade to MariaDB 11.8 [19:13:31] [02puppet] 07Universal-Omega opened pull request #4461: mon181: upgrade to MariaDB 11.8 (03main...03mon181-mariadb-11.8) 13https://github.com/miraheze/puppet/pull/4461 [19:13:39] [02puppet] 07coderabbitai[bot] commented on pull request #4461:

[…] 13https://github.com/miraheze/puppet/pull/4461#issuecomment-3148642632 [19:16:45] !log upgrade db182 to mariadb 11.8 [19:16:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:55] !log upgrade kernel and all packages on db182 [19:16:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:07] !log reboot db182 [19:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:11] [02puppet] 07Universal-Omega merged pull request #4461: mon181: upgrade to MariaDB 11.8 (03main...03mon181-mariadb-11.8) 13https://github.com/miraheze/puppet/pull/4461 [19:18:11] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e7f097f8a31bad503b9731f06c9ddc659d376068 [19:18:11] 02puppet/03main 07CosmicAlpha 03e7f097f mon181: upgrade to MariaDB 11.8 [19:18:12] [02puppet] 07Universal-Omega 04deleted 03mon181-mariadb-11.8 at 03b7983f5 13https://api.github.com/repos/miraheze/puppet/commit/b7983f5 [19:24:38] !log upgraded mariadb-client on mon181 to 11.8 [19:24:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:27] !log upgraded grafana, icinga/icingacli/icingaweb2, and all other packages on mon181 [19:27:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:43] !log reboot mon181 [19:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/a05c49e5cdafdc51edaac537b68a8f219f92f0eb [19:29:07] 02puppet/03main 07CosmicAlpha 03a05c49e monitoring: default mariadb::version to 11.8 [19:31:07] CUSTOM - mon181 NTP time on mon181 is OK: NTP OK: Offset -1.353025436e-05 secs universalomega test [19:38:38] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/483874fadc07ea3f5e7e91bff1ab6218ca2caac6 [19:38:38] 02ManageWiki/03main 07CosmicAlpha 03483874f Return Message for getDescription… [19:44:01] miraheze/ManageWiki - Universal-Omega the build passed. [19:46:16] [02ManageWiki] 07Universal-Omega 04force-pushed 03RhinosF1-patch-1 from 038080780 to 035c672ea 13https://api.github.com/repos/miraheze/ManageWiki/commits/RhinosF1-patch-1 [19:46:16] 02ManageWiki/03RhinosF1-patch-1 07RhinosF1 035c672ea ToggleExtension: better help message when ->fatalError for failed to disable [19:47:01] [02ManageWiki] 07Universal-Omega merged 07RhinosF1's pull request #719: ToggleExtension: better help message when ->fatalError for failed to disable (03main...03RhinosF1-patch-1) 13https://github.com/miraheze/ManageWiki/pull/719 [19:47:01] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/59d2e2fedef990b16d5743941117caed96c40c1e [19:47:03] 02ManageWiki/03main 07RhinosF1 0359d2e2f ToggleExtension: better help message when ->fatalError for failed to disable (#719) [19:47:05] [02ManageWiki] 07Universal-Omega 04deleted 03RhinosF1-patch-1 at 035c672ea 13https://api.github.com/repos/miraheze/ManageWiki/commit/5c672ea [19:51:43] miraheze/ManageWiki - Universal-Omega the build passed. [19:52:36] miraheze/ManageWiki - Universal-Omega the build passed. [19:55:12] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=tintinpediawiki (END - exit=0) [19:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:06] [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3c2cc85f3262e8ef54d6818b83659bdd78e09a7a [20:04:06] 02ssl/03main 07Skye 033c2cc85 Update tintinpediawiki_redirect [20:06:25] miraheze/CreateWiki - Universal-Omega the build passed. [20:08:58] miraheze/CreateWiki - Universal-Omega the build has errored. [20:13:24] @pixldev now failures are actually related to the PR [20:17:53] [1/6] ``` [20:17:53] [2/6] maintenance/CheckLastWikiActivity.php:24 PhanPossiblyNonClassMethodCall Call to method getActorId on type ?\MediaWiki\User\User that could be a non-object [20:17:53] [3/6] maintenance/ManageInactiveWikis.php:56 PhanUnusedVariable Unused definition of variable $inactiveDays [20:17:54] [4/6] maintenance/ManageInactiveWikis.php:58 PhanUnusedVariable Unused definition of variable $databaseUtils [20:17:54] [5/6] maintenance/ManageInactiveWikis.php:59 PhanUnusedVariable Unused definition of variable $remoteWikiFactory [20:17:54] [6/6] ``` [20:35:35] Huh [20:35:55] I’ll look at this later, may need to ask some people on this stuff lol [20:35:55] CW's phan is much stricter then it used to be lol [20:38:55] [1/10] for the phan issues here at least just removing those unused variables and adding a check if defaultActor is null. Change to: [20:38:56] [2/10] ```php [20:38:56] [3/10] $defaultActor = $this->getServiceContainer()->getUserFactory() [20:38:56] [4/10] ->newFromName( 'MediaWiki default' ) [20:38:57] [5/10] ?->getActorId(); [20:38:57] [6/10] if ( $defaultActor === null ) { [20:38:57] [7/10] $this->fatalError( 'Default actor returned null' ); [20:38:57] [8/10] } [20:38:58] [9/10] ``` [20:38:58] [10/10] which by principle PHP will work but not sure if phan will like it lol [21:19:30] [02TSPortal] 07Universal-Omega created 03fix-migration (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/70b7a37dbf18 [21:19:30] 02TSPortal/03fix-migration 07CosmicAlpha 0370b7a37 Fix migration for updating DPAs underage [21:19:36] [02TSPortal] 07Universal-Omega opened pull request #28: Fix migration for updating DPAs underage (03main...03fix-migration) 13https://github.com/miraheze/TSPortal/pull/28 [21:19:41] [02TSPortal] 07coderabbitai[bot] commented on pull request #28:

[…] 13https://github.com/miraheze/TSPortal/pull/28#issuecomment-3148704236 [21:23:10] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03fix-migration 13https://github.com/miraheze/TSPortal/commit/47a376668efa22028ef1962e97cb8431178005b1 [21:23:10] 02TSPortal/03fix-migration 07CosmicAlpha 0347a3766 Include doctrine/dbal [21:23:54] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03fix-migration 13https://github.com/miraheze/TSPortal/commit/4c39098e2c4ed95a91ed99677b69d340b204726c [21:23:54] 02TSPortal/03fix-migration 07CosmicAlpha 034c39098 Bump version [21:27:03] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03fix-migration 13https://github.com/miraheze/TSPortal/commit/68c4b5f324834262b1a95b768371065bdfd94619 [21:27:04] 02TSPortal/03fix-migration 07CosmicAlpha 0368c4b5f Update [21:27:28] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03fix-migration 13https://github.com/miraheze/TSPortal/commit/993672f8e3586daad9b67dcf38d073897876b149 [21:27:28] 02TSPortal/03fix-migration 07CosmicAlpha 03993672f Update [21:28:50] [02TSPortal] 07Universal-Omega merged pull request #28: Fix migration for updating DPAs underage (03main...03fix-migration) 13https://github.com/miraheze/TSPortal/pull/28 [21:28:50] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/7419cd0ce9e936cac1f2102c57aaa3900a0c436b [21:28:51] 02TSPortal/03main 07CosmicAlpha 037419cd0 Fix migration for updating DPAs underage [21:28:53] [02TSPortal] 07Universal-Omega 04deleted 03fix-migration at 03993672f 13https://api.github.com/repos/miraheze/TSPortal/commit/993672f [21:30:02] [02TSPortal] 07Universal-Omega released Version 21 13https://github.com/miraheze/TSPortal/releases/tag/v21 [21:30:02] [02TSPortal] 07Universal-Omega tagged 03v21 at 03main 13https://github.com/miraheze/TSPortal/compare/v21 [21:30:52] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/92c8a79860de044ef46a68bac2270348e27d71db [21:30:53] 02puppet/03main 07CosmicAlpha 0392c8a79 Update TSPortal to v21 [21:48:46] [02TSPortal] 07Universal-Omega created 03fix-dbal (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/da73687564c8 [21:48:46] 02TSPortal/03fix-dbal 07CosmicAlpha 03da73687 Fix version of doctrine/dbal we use [21:48:52] [02TSPortal] 07Universal-Omega opened pull request #29: Fix version of doctrine/dbal we use (03main...03fix-dbal) 13https://github.com/miraheze/TSPortal/pull/29 [21:48:57] [02TSPortal] 07coderabbitai[bot] commented on pull request #29:

[…] 13https://github.com/miraheze/TSPortal/pull/29#issuecomment-3148720344 [21:52:50] PROBLEM - reports171 HTTPS on reports171 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 133 bytes in 0.010 second response time [21:53:07] PROBLEM - reports171 Check unit status of task-scheduler on reports171 is CRITICAL: CRITICAL: Status of the systemd unit task-scheduler [21:56:44] RECOVERY - reports171 HTTPS on reports171 is OK: HTTP OK: HTTP/2 200 - 1139 bytes in 0.029 second response time [21:57:00] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03fix-dbal 13https://github.com/miraheze/TSPortal/commit/784eb2e386224fc8fb6f558b4b37117ebc1a933f [21:57:01] 02TSPortal/03fix-dbal 07CosmicAlpha 03784eb2e Bump [21:57:07] RECOVERY - reports171 Check unit status of task-scheduler on reports171 is OK: OK: Status of the systemd unit task-scheduler [21:58:21] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03fix-dbal 13https://github.com/miraheze/TSPortal/commit/7f71c8fec595b86f419d0f7f72fad4b2987e96aa [21:58:21] 02TSPortal/03fix-dbal 07CosmicAlpha 037f71c8f Update [21:58:51] [02TSPortal] 07Universal-Omega merged pull request #29: Fix version of doctrine/dbal we use (03main...03fix-dbal) 13https://github.com/miraheze/TSPortal/pull/29 [21:58:51] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/a4aab9af22c83752a480591e7fb8e5331a36568c [21:58:52] 02TSPortal/03main 07CosmicAlpha 03a4aab9a Fix version of doctrine/dbal we use [21:58:52] [02TSPortal] 07Universal-Omega 04deleted 03fix-dbal at 037f71c8f 13https://api.github.com/repos/miraheze/TSPortal/commit/7f71c8f [21:59:22] [02TSPortal] 07Universal-Omega tagged 03v22 at 03main 13https://github.com/miraheze/TSPortal/compare/v22 [21:59:23] [02TSPortal] 07Universal-Omega released Version 22 13https://github.com/miraheze/TSPortal/releases/tag/v22 [21:59:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/33d9e056febd4829dc953dfad739e59f25fe11b2 [21:59:45] 02puppet/03main 07CosmicAlpha 0333d9e05 Update TSPortal to v22 [22:18:14] [02TSPortal] 07Universal-Omega created 03longtext (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/4c620e1dc211 [22:18:14] 02TSPortal/03longtext 07CosmicAlpha 034c620e1 Add migration to convert underage field to LONGTEXT [22:18:19] [02TSPortal] 07Universal-Omega opened pull request #30: Add migration to convert underage field to LONGTEXT (03main...03longtext) 13https://github.com/miraheze/TSPortal/pull/30 [22:18:29] [02TSPortal] 07coderabbitai[bot] commented on pull request #30:

[…] 13https://github.com/miraheze/TSPortal/pull/30#issuecomment-3148735217 [22:18:48] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03longtext 13https://github.com/miraheze/TSPortal/commit/b5a6099547cec57c70b379d2bb09d3a72eafe7f3 [22:18:48] 02TSPortal/03longtext 07CosmicAlpha 03b5a6099 Bump [22:20:20] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03longtext 13https://github.com/miraheze/TSPortal/commit/59cafe0274410d663c265d7ba0e02d83ebd76d26 [22:20:20] 02TSPortal/03longtext 07CosmicAlpha 0359cafe0 Update [22:21:18] [02TSPortal] 07Universal-Omega merged pull request #30: Add migration to convert underage field to LONGTEXT (03main...03longtext) 13https://github.com/miraheze/TSPortal/pull/30 [22:21:18] [02TSPortal] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/TSPortal/commit/d7b991164cb210d86f07cf8b0bdd4c59f375fbf4 [22:21:18] 02TSPortal/03main 07CosmicAlpha 03d7b9911 Add migration to convert underage field to LONGTEXT [22:21:19] [02TSPortal] 07Universal-Omega 04deleted 03longtext at 0359cafe0 13https://api.github.com/repos/miraheze/TSPortal/commit/59cafe0 [22:22:01] [02TSPortal] 07Universal-Omega released Version 23 13https://github.com/miraheze/TSPortal/releases/tag/v23 [22:22:02] [02TSPortal] 07Universal-Omega tagged 03v23 at 03main 13https://github.com/miraheze/TSPortal/compare/v23 [22:22:21] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/624ca80ffbbc41e9e10d100972ee295bee41b142 [22:22:21] 02puppet/03main 07CosmicAlpha 03624ca80 Update TSPortal to v23 [22:26:54] !log sudo -u www-data php artisan migrate on reports171 [22:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:28:19] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1967MiB (10% inode=90%); [22:34:26] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.43/extensions/Tilesheets,1.44/extensions/Tilesheets'} to test151 [22:34:27] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.43/extensions/Tilesheets,1.44/extensions/Tilesheets'} to test151 - SUCCESS in 0s [22:34:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:42] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.43/extensions/Tilesheets,1.44/extensions/Tilesheets'} to all [22:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:38:29] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.43/extensions/Tilesheets,1.44/extensions/Tilesheets'} to all - SUCCESS in 47s [22:38:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:19] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6573MiB (36% inode=89%); [23:04:05] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:05:02] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:05:39] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:06:15] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:06:58]