[00:03:21] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 26853MiB (5% inode=99%);
[00:06:03] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26613MiB (5% inode=99%);
[00:09:17] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54786MiB (12% inode=99%);
[00:10:02] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55168MiB (12% inode=99%);
[04:46:29] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49872MiB (10% inode=99%);
[05:12:00] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49936MiB (10% inode=99%);
[05:57:52] <MirahezeLSBot>	 !log [blankeclair@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_skins': 'Citizen'} to all
[05:57:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:58:16] <MirahezeLSBot>	 !log [blankeclair@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_skins': 'Citizen'} to all - SUCCESS in 24s
[05:58:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:36:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749803780000&orgId=1&to=1749807413348
[09:41:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749803960000&orgId=1&to=1749807560000
[09:47:38] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/updateCollation.php --wiki=historikawiki (END - exit=0)
[09:47:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:50:43] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/updateCollation.php --wiki=rapanuidictionaryprojectwiki (END - exit=0)
[09:50:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:30:02] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_skins': 'Metrolook'} to all
[10:30:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:30:25] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_skins': 'Metrolook'} to all - SUCCESS in 22s
[10:30:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:11] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'StructuredNavigation'} to test151
[11:02:13] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'StructuredNavigation'} to test151 - SUCCESS in 2s
[11:02:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:02:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:06:21] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'StructuredNavigation'} to all
[11:06:22] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost
[11:06:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:06:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:07:02] <MirahezeRelay>	 <abaddriverlol> uhh
[11:07:31] <MirahezeRelay>	 <abaddriverlol> do I just use --force?
[11:30:03] <BlankEclair>	 abaddriverlol: why did it fail?
[11:32:47] <MirahezeRelay>	 <abaddriverlol, replying to BlankEclair> `Canary check failed for publictestwiki.com@localhost. Aborting... - use --force to proceed`
[11:34:31] <BlankEclair>	 abaddriverlol: can you do this lil experiment for me because i'm lazy?
[11:35:00] <MirahezeRelay>	 <abaddriverlol> sure
[11:35:29] <BlankEclair>	 `for i in wikitide.net fsslc.wtnet; do curl --no-progress-meter -I https://publictestwiki.com/wiki/$RANDOM -H "X-Wikitide-Debug: mwtask171.${i}" | grep -i x-served-by; done`
[11:35:42] <MirahezeRelay>	 <aeywoo, replying to abaddriverlol> Wait, do you already have prod access?
[11:35:47] <BlankEclair>	 mwdeploy seems to set x-wikitide-debug: <meow>.wikitide.net
[11:35:50] <BlankEclair>	 do we still use that?
[11:36:02] <BlankEclair>	 the reason why i haven't ssh'd in is part laziness, and part anxiety being overwhelming atm :p
[11:36:38] <MirahezeRelay>	 <abaddriverlol, replying to BlankEclair> [1/2] x-served-by: mw181.fsslc.wtnet
[11:36:38] <MirahezeRelay>	 <abaddriverlol, replying to BlankEclair> [2/2] x-served-by: mw171.fsslc.wtnet
[11:36:43] <MirahezeRelay>	 <abaddriverlol, replying to aeywoo> yep since today
[11:36:48] <BlankEclair>	 huh
[11:36:52] <BlankEclair>	 aaaand i have laundry
[11:36:58] <BlankEclair>	 ugh, i can't have a break irl, can't i >_>
[11:37:09] <MirahezeRelay>	 <aeywoo, replying to abaddriverlol> How?
[11:38:55] <MirahezeRelay>	 <abaddriverlol, replying to aeywoo> I filled out the mentoring thing and was approved <https://issue-tracker.miraheze.org/T13845>
[12:08:40] <BlankEclair>	 abaddriver: sorry, i'm too stressed to be useful now
[12:12:51] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> It should spam a load of debug below
[12:24:59] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> [1/2] it didn't
[12:24:59] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1383059573157662831/image.png?ex=684d6a1a&is=684c189a&hm=20aa1eb2e1135b98b9a3b18c5437b6faa5468fdacac4b5546ab15e3f2a36ff60&
[12:25:01] <MirahezeRelay>	 <abaddriverlol> should I just try again?
[12:35:50] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> It did
[12:35:52] <MirahezeRelay>	 <rhinosf1> Status: 500
[12:36:04] <MirahezeRelay>	 <abaddriverlol> oh that
[12:36:10] <MirahezeRelay>	 <rhinosf1> It failed cause it got a 500
[12:36:28] <MirahezeRelay>	 <rhinosf1> So ye, try again
[12:36:35] <MirahezeRelay>	 <rhinosf1> Assuming the 500 was intermittent
[12:36:41] <MirahezeRelay>	 <rhinosf1> And not a real error
[12:37:00] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'StructuredNavigation'} to all
[12:37:01] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'StructuredNavigation'} to all - SUCCESS in 1s
[12:37:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:10] <MirahezeRelay>	 <abaddriverlol> 1s?
[12:40:52] <MirahezeRelay>	 <abaddriverlol> that's weird, worked though
[12:43:37] <MirahezeRelay>	 <abaddriverlol> nvm it didn't work
[12:44:08] <MirahezeRelay>	 <abaddriverlol> @rhinosf1 it gave me `StructuredNavigation already up to date. Skipping...` when upgrading
[12:44:18] <MirahezeRelay>	 <abaddriverlol> even though it is not up to date
[12:44:59] <MirahezeRelay>	 <rhinosf1> Use --folders
[12:45:07] <MirahezeRelay>	 <rhinosf1> I think that's a bug in the upgrading logic
[12:51:00] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.43/extensions/StructuredNavigation'} to all
[12:51:01] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151
[12:51:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:51:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:51:13] <MirahezeRelay>	 <abaddriverlol> ah cool, the same error again
[12:51:28] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> http 500 again
[12:52:27] <MirahezeRelay>	 <rhinosf1> @abaddriverlol means your upgrade is possibly causing a 500
[12:52:35] <MirahezeRelay>	 <rhinosf1> Or the script is broken somewhere
[12:52:41] <MirahezeRelay>	 <rhinosf1> That's @localhost though
[12:52:54] <MirahezeRelay>	 <rhinosf1> But it implies that a local curl to testwiki would 500
[12:53:10] <MirahezeRelay>	 <abaddriverlol, replying to rhinosf1> it shouldn't
[13:25:49] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> Weird
[15:05:00] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=ecopediawiki --r [[phab:T13701|Requested]] /home/somerandomdeveloper/BatchDeletionT13701.txt (START)
[15:05:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:05:06] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=ecopediawiki --r [[phab:T13701|Requested]] /home/somerandomdeveloper/BatchDeletionT13701.txt (END - exit=2)
[15:05:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:06:54] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=ecopediawiki --r '[[phab:T13701|Requested]]' /home/somerandomdeveloper/BatchDeletionT13701.txt (START)
[15:06:56] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/deleteBatch.php --wiki=ecopediawiki --r '[[phab:T13701|Requested]]' /home/somerandomdeveloper/BatchDeletionT13701.txt (END - exit=256)
[15:06:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:06:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:07:23] <MirahezeRelay>	 <abaddriverlol, replying to MirahezeLSBot> the example at <https://meta.miraheze.org/wiki/Tech:MediaWiki_appserver> is wrong, gg
[17:35:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749832520000&orgId=1&to=1749836153347
[17:45:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749833060000&orgId=1&to=1749836660000
[17:56:40] <MirahezeRelay>	 <rhinosf1, replying to abaddriverlol> How so?
[17:56:48] <MirahezeRelay>	 <abaddriverlol> it was missing quotes
[17:56:59] <MirahezeRelay>	 <abaddriverlol, replying to MirahezeLSBot> so the command ^ contained `--r [[phab:T13701|Requested]]`
[17:57:17] <MirahezeRelay>	 <abaddriverlol> i fixed it on the page already
[17:57:26] <MirahezeRelay>	 <rhinosf1> Ah good
[18:48:27] <MirahezeRelay>	 <songngu.xyz> just for notices, but it seems that StructuredNavigation hasn't been upgraded, [[phab:T13560]] are currently blocked
[18:48:29] <MirahezeRelay>	 <Wiki-Bot#2998> <https://issue-tracker.miraheze.org/T13560>
[18:51:54] <MirahezeRelay>	 <songngu.xyz> and so is [[mw:extension:DummyFandoomMainpageTags]]
[18:52:05] <MirahezeRelay>	 <Wiki-Bot#2998> <https://www.mediawiki.org/wiki/extension:DummyFandoomMainpageTags>
[19:01:05] <MirahezeRelay>	 <abaddriverlol, replying to songngu.xyz> yes, I tried to do that earlier today but ran into an issue, but as soon as that's fixed, I'll upgrade the extension
[19:01:14] <MirahezeRelay>	 <abaddriverlol, replying to songngu.xyz> which task?
[20:43:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749843800000&orgId=1&to=1749847433352
[20:48:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749844100000&orgId=1&to=1749847700000
[21:48:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749847700000&orgId=1&to=1749851333348
[21:58:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749848300000&orgId=1&to=1749851900000
[22:35:45] <MirahezeRelay>	 <songngu.xyz, replying to abaddriverlol> not specific, but rather there was some changes in the repo itself that I know of