[00:00:01] <MirahezeLSBot>	 !log [www-data@test151] Began automatic backing up
[00:00:02] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[00:00:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:00:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:01:17] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/fc58a2ff11c5393cce2cfff320dddb061eefe74d
[00:01:19] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03fc58a2f Bot: Auto-update Tech namespace pages 2025-08-01 00:01:15
[00:06:36] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 56739MiB (12% inode=99%);
[00:07:37] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56200MiB (12% inode=99%);
[00:07:39] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56268MiB (12% inode=99%);
[00:08:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754003300000&orgId=1&to=1754006933393
[00:23:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754004080000&orgId=1&to=1754007680000
[00:35:57] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/5dd6ae1e3e4c6143ee31229bcd3ac792a7f57992
[00:35:57] <MacFanBot>	 02ssl/03main 07Skye 035dd6ae1 Add blueben.xyz
[00:49:35] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/722ee2948c9c97ae28462bd5772eaee7b453d22b
[00:49:35] <MacFanBot>	 02ssl/03main 07Skye 03722ee29 Revert "Add blueben.xyz"
[00:49:40] <MirahezeRelay>	 <pskyechology> whoopsie :3
[00:52:40] <MirahezeRelay>	 <pskyechology> im guessing that was the curse of the yellow cloud
[02:11:59] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.049 second response time
[02:23:58] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4040 bytes in 0.019 second response time
[02:25:59] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.342 second response time
[02:35:58] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.049 second response time
[02:37:59] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.055 second response time
[02:55:25] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03modern 13https://github.com/miraheze/mw-config/commit/edc47187b4425f2977af2d99b07c8086817ca2b7
[02:55:26] <MacFanBot>	 02mw-config/03modern 07CosmicAlpha 03edc4718 Fix
[02:56:16] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[04:32:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754019140000&orgId=1&to=1754022773355
[04:37:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754019320000&orgId=1&to=1754022920000
[05:17:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754021840000&orgId=1&to=1754025473357
[05:47:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1754021840000&orgId=1&to=1754027240000
[08:36:29] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49880MiB (10% inode=99%);
[08:37:53] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49844MiB (10% inode=99%);
[09:13:16] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49930MiB (10% inode=99%);
[09:34:01] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.49, 19.78, 17.44
[09:36:01] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.13, 19.77, 17.72
[09:48:01] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.52, 20.27, 18.93
[09:50:01] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.76, 19.86, 18.96
[10:04:28] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.07, 20.44, 19.72
[10:06:23] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.21, 20.08, 19.68
[10:13:04] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.89, 20.71, 20.05
[10:18:48] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.45, 18.90, 19.74
[10:30:27] <MirahezeRelay>	 <pskyechology, replying to pskyechology> (no, i was just stupid)
[10:31:57] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/288f6939b56a9c008ce936f74cbf0106e2ef44a8
[10:31:57] <MacFanBot>	 02ssl/03main 07Skye 03288f693 Add binrayarchives.com
[11:34:06] <icinga-miraheze>	 PROBLEM - swiftobject181 Disk Space on swiftobject181 is WARNING: DISK WARNING - free space: / 146895MiB (10% inode=86%);
[12:10:05] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/4401401fc446a43e850279cace62774546a7f849
[12:10:05] <MacFanBot>	 02ssl/03main 07Skye 034401401 Add wiki.tmyt105.com and blueben.xyz
[12:54:33] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/19f9102fe7fdb493b3c3fe499205292bca9e003e
[12:54:33] <MacFanBot>	 02ssl/03main 07Skye 0319f9102 Remove wikitmyt105leyhpcom
[13:42:48] <icinga-miraheze>	 PROBLEM - swiftobject161 Disk Space on swiftobject161 is WARNING: DISK WARNING - free space: / 146810MiB (10% inode=86%);
[14:44:16] <MacFanBot>	 [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-12.3.0 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/346852b6ec59
[14:44:16] <MacFanBot>	 02mw-config/03dependabot/composer/phpunit/phpunit-12.3.0 07dependabot[bot] 03346852b Update phpunit/phpunit requirement from 12.2.9 to 12.3.0…
[14:44:18] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6046 (Update phpunit/phpunit requirement from 12.2.9 to 12.3.0) 13https://github.com/miraheze/mw-config/pull/6046
[14:44:20] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6046 (Update phpunit/phpunit requirement from 12.2.9 to 12.3.0) 13https://github.com/miraheze/mw-config/pull/6046
[14:44:22] <MacFanBot>	 [02mw-config] 07dependabot[bot] opened pull request #6046: Update phpunit/phpunit requirement from 12.2.9 to 12.3.0 (03main...03dependabot/composer/phpunit/phpunit-12.3.0) 13https://github.com/miraheze/mw-config/pull/6046
[14:44:24] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6046 (Update phpunit/phpunit requirement from 12.2.9 to 12.3.0) 13https://github.com/miraheze/mw-config/pull/6046
[14:44:26] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6046 (Update phpunit/phpunit requirement from 12.2.9 to 12.3.0) 13https://github.com/miraheze/mw-config/pull/6046
[14:45:06] <miraheze-github>	 miraheze/mw-config - dependabot[bot] the build passed.
[17:05:07] <MacFanBot>	 [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6046: Update phpunit/phpunit requirement from 12.2.9 to 12.3.0 (03main...03dependabot/composer/phpunit/phpunit-12.3.0) 13https://github.com/miraheze/mw-config/pull/6046
[17:05:08] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/6ab1f376347852d11bebc32449af3446dd7cd812
[17:05:08] <MacFanBot>	 02mw-config/03main 07dependabot[bot] 036ab1f37 Update phpunit/phpunit requirement from 12.2.9 to 12.3.0 (#6046)…
[17:05:08] <MacFanBot>	 [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/phpunit/phpunit-12.3.0 at 03346852b 13https://api.github.com/repos/miraheze/mw-config/commit/346852b
[17:06:08] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[18:26:34] <MacFanBot>	 [02DataDump] 07Universal-Omega created 03job-status (+1 new commit) 13https://github.com/miraheze/DataDump/commit/c9cb70d8a937
[18:26:34] <MacFanBot>	 02DataDump/03job-status 07CosmicAlpha 03c9cb70d Don't rerun job if status is already completed
[18:26:39] <MacFanBot>	 [02DataDump] 07Universal-Omega opened pull request #107: Don't rerun job if status is already completed (03main...03job-status) 13https://github.com/miraheze/DataDump/pull/107
[18:26:46] <MacFanBot>	 [02DataDump] 07coderabbitai[bot] commented on pull request #107: <details> […] 13https://github.com/miraheze/DataDump/pull/107#issuecomment-3145410000
[18:31:08] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[18:34:55] <MacFanBot>	 [02DataDump] 07Universal-Omega merged pull request #107: Don't rerun job if status is already completed (03main...03job-status) 13https://github.com/miraheze/DataDump/pull/107
[18:34:55] <MacFanBot>	 [02DataDump] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/17ddaeade8ff1b902e2b825a8513865ad093bbbb
[18:34:55] <MacFanBot>	 02DataDump/03main 07CosmicAlpha 0317ddaea Don't rerun job if status is already completed
[18:34:56] <MacFanBot>	 [02DataDump] 07Universal-Omega 04deleted 03job-status at 03c9cb70d 13https://api.github.com/repos/miraheze/DataDump/commit/c9cb70d
[18:35:14] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'DataDump'} to all
[18:35:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:36:16] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'DataDump'} to all - SUCCESS in 62s
[18:36:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:39:35] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[19:31:10] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/02442aa1d7c2aa86768f1f69bcc59d11b8b690ab
[19:31:10] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 0302442aa Well... let's not chunk dumps in parts of 2MB... whoops
[19:31:39] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[19:31:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:32:05] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s
[19:32:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:32:10] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[19:40:11] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev created 03moved-repos (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/e67d8e5c53f8
[19:40:11] <MacFanBot>	 02mediawiki-repos/03moved-repos 07SomeRandomDeveloper 03e67d8e5 Update URLs and branches for repos that were moved to Gerrit…
[19:40:47] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev opened pull request #83: Update URLs and branches for repos that were moved to Gerrit (03main...03moved-repos) 13https://github.com/miraheze/mediawiki-repos/pull/83
[19:40:54] <MacFanBot>	 [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #83: <details> […] 13https://github.com/miraheze/mediawiki-repos/pull/83#issuecomment-3145589582
[19:46:13] <MacFanBot>	 [02mediawiki-repos] 07Universal-Omega approved pull request #83 13https://github.com/miraheze/mediawiki-repos/pull/83#pullrequestreview-3080388771
[19:49:53] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/e721bd5a79771e12fe0278267fd1ca145937d379
[19:49:55] <MacFanBot>	 02mediawiki-repos/03main 07SomeRandomDeveloper 03e721bd5 Update URLs and branches for repos that were moved to Gerrit (#83)…
[19:49:57] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev merged pull request #83: Update URLs and branches for repos that were moved to Gerrit (03main...03moved-repos) 13https://github.com/miraheze/mediawiki-repos/pull/83
[19:49:59] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev 04deleted 03moved-repos at 03e67d8e5 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/e67d8e5
[20:16:00] <MirahezeRelay>	 <abaddriverlol> !log ran "sudo -u www-data rm -rf /srv/mediawiki-staging/{1.43,1.44}/{extensions/AutoCreatePage,extensions/EmbedSpotify,extensions/JavascriptSlideshow,extensions/MachineTranslation,skins/Monaco}" on test151 and ran puppet afterwards
[20:16:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:18:36] <MirahezeRelay>	 <abaddriverlol> !log ran "sudo -u www-data rm -rf /srv/mediawiki-staging/{1.43,1.44}/{extensions/AutoCreatePage,extensions/EmbedSpotify,extensions/JavascriptSlideshow,extensions/MachineTranslation,skins/Monaco}" on mwtask181 and ran puppet afterwards
[20:18:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:19:59] <MirahezeRelay>	 <abaddriverlol, replying to abaddriverlol> ^ @cosmicalpha you might want to consider archiving the old gh repos so people don't create PRs/issues anymore (also it would be useful to have projects on phab for repos like Monaco which currently have open issues)
[20:20:30] <MirahezeRelay>	 <cosmicalpha> good idea to both
[20:21:09] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'EmbedSpotify'} to test151
[20:21:10] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'EmbedSpotify'} to test151 - SUCCESS in 0s
[20:21:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:21:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:22:30] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedSpotify'} to test151
[20:22:31] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedSpotify'} to test151 - SUCCESS in 0s
[20:22:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:22:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:22:51] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedSpotify'} to all
[20:22:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:23:16] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedSpotify'} to all - SUCCESS in 25s
[20:23:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:56:26] <MacFanBot>	 [02puppet] 07pixDeVl opened pull request #4458: CSP additions (07miraheze:03main...07pixDeVl:03patch-1) 13https://github.com/miraheze/puppet/pull/4458
[20:56:35] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4458: <details> […] 13https://github.com/miraheze/puppet/pull/4458#issuecomment-3145752779
[21:40:05] <MirahezeRelay>	 <cosmicalpha> !log backup first then run update.php for arsenalrobloxwikiwiki
[21:40:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:47:39] <MirahezeLSBot>	 !log [universalomega@test151] starting deploy of {'extension_list': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to test151
[21:47:40] <MirahezeLSBot>	 !log [universalomega@test151] finished deploy of {'extension_list': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to test151 - SUCCESS in 1s
[21:47:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:47:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:49:04] <MirahezeLSBot>	 !log [universalomega@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ResetWikiCaches (START)
[21:49:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:49:12] <MirahezeLSBot>	 !log [universalomega@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ResetWikiCaches (END - exit=0)
[21:49:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:36:15] <icinga-miraheze>	 PROBLEM - swiftobject151 Disk Space on swiftobject151 is WARNING: DISK WARNING - free space: / 146891MiB (10% inode=86%);
[22:41:32] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.57, 9.52, 5.85
[22:43:32] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.57, 7.98, 5.75
[22:51:32] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 5.39, 10.86, 8.23
[22:53:32] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.28, 7.77, 7.41
[23:33:26] <miraheze-github>	 miraheze/python-functions - dependabot[bot] the build passed.