[00:00:04] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[00:00:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:01:02] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/77c0b2d1fb48546b6a648be23266f9decc71a4ef
[00:01:02] <MacFanBot>	 02statichelp/03main 07WikiTideBot 0377c0b2d Bot: Auto-update Tech namespace pages 2026-04-01 00:01:00
[00:03:16] <MirahezeRelay>	 <thewwrnerdguy, replying to MirahezeLSBot> is this a fake version of me
[00:04:33] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Scribunto'} to test151
[00:04:34] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Scribunto'} to test151 - SUCCESS in 1s
[00:04:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:04:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:04:37] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Scribunto'} to all
[00:04:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:05:02] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Scribunto'} to all - SUCCESS in 24s
[00:05:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:11:34] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'AbuseFilter'} to test151
[01:11:35] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'AbuseFilter'} to test151 - SUCCESS in 1s
[01:11:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:11:38] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'AbuseFilter'} to all
[01:11:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:11:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:12:06] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'AbuseFilter'} to all - SUCCESS in 27s
[01:12:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:12:08] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'l10n': True, 'versions': '1.45'} to test151
[01:12:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:12:20] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'l10n': True, 'versions': '1.45'} to all
[01:12:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:17:55] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'l10n': True, 'versions': '1.45'} to test151 - SUCCESS in 346s
[01:17:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:21:11] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'l10n': True, 'versions': '1.45'} to all - SUCCESS in 531s
[01:21:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:35:46] <MacFanBot>	 [02mw-config] 07lihaohong6 opened pull request #6355: T15166: add new echo permission (07miraheze:03main...07lihaohong6:03echo) 13https://github.com/miraheze/mw-config/pull/6355
[02:36:34] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'config': True} to test151
[02:36:35] <MirahezeLSBot>	 !log [petramagna@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[02:36:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:36:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:36:47] <miraheze-github>	 miraheze/mw-config - lihaohong6 the build passed.
[02:37:57] <MirahezeLSBot>	 !log [petramagna@test151] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:ResetWikiCaches --wiki=exttestwikibeta (END - exit=0)
[02:38:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:38:20] <MacFanBot>	 [02mw-config] 07lihaohong6 merged pull request #6355: T15166: add new echo permission (07miraheze:03main...07lihaohong6:03echo) 13https://github.com/miraheze/mw-config/pull/6355
[02:38:20] <MacFanBot>	 [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b163cbdfac1b6a729985ca3228d1dc6cc8c85567
[02:38:21] <MacFanBot>	 02mw-config/03main 07Peter Li 03b163cbd T15166: add new echo permission (#6355)
[02:38:52] <MirahezeLSBot>	 !log [petramagna@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[02:38:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:39:18] <MirahezeLSBot>	 !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s
[02:39:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:41:07] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:ResetWikiCaches --wiki=strinovawiki (END - exit=0)
[02:41:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:53:55] <MirahezeLSBot>	 !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:ResetWikiCaches --wiki=metawiki --all-wikis (END - exit=0)
[02:53:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:57:24] <MirahezeRelay>	 <pixldev> @posix_memalign did CA make a new script to run the all wiki scripts faster across multiple things or am I Tripping
[02:59:07] <MirahezeRelay>	 <posix_memalign, replying to pixldev> I'm not sure. I know that Random wrote this https://github.com/SomeMWDev/mirarust/blob/main/src/foreachwikiindblist-parallel.rs but it's not on puppet. So you'd have to compile and `scp` it yourself.
[02:59:07] <MacFanBot>	 [GitHub] [SomeMWDev/mirarust] src/foreachwikiindblist-parallel.rs @ main
[04:35:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.62, 15.04, 8.26
[04:37:45] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 8.99, 12.55, 8.18
[04:40:14] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 5 backends are down. mw152 mw161 mw162 mw163 mw201
[04:40:24] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw152 mw161 mw162 mw181 mw153 mw163 mw173 mw183 mw203
[04:44:24] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[04:46:14] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[07:49:41] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 19.99, 10.31
[07:51:38] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 7.44, 14.77, 9.56
[08:15:28] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'world', 'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to test151
[08:15:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:17:58] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[08:18:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:21:35] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'world', 'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to test151
[08:21:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:22:34] <MirahezeRelay>	 <abaddriverlol, replying to pixldev> iirc --all-wikis is faster than foreachwikiindblist because it iterates through the DBs within the PHP script instead of spawning one script for each DB
[08:27:49] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'pull': 'world', 'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to test151 - SUCCESS in 373s
[08:27:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:30:29] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/87e329f5820b986da1f301aba60fb20a6d6c4181
[12:30:29] <MacFanBot>	 02ssl/03main 07WikiTideBot 0387e329f Bot: Auto-update domain lists
[14:40:27] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'world', 'world': True, 'l10n': True, 'versions': '1.45'} to test151
[14:40:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:46:52] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'pull': 'world', 'world': True, 'l10n': True, 'versions': '1.45'} to test151 - SUCCESS in 384s
[14:46:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:12:15] <MacFanBot>	 [02mw-config] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/91076a39eb6a
[15:12:15] <MacFanBot>	 02mw-config/03paladox-patch-6 07paladox 0391076a3 Adjust wgThumbnailBuckets and wgUploadThumbnailRenderMap…
[15:12:20] <MacFanBot>	 [02mw-config] 07paladox opened pull request #6356: Adjust wgThumbnailBuckets and wgUploadThumbnailRenderMap (03main...03paladox-patch-6) 13https://github.com/miraheze/mw-config/pull/6356
[15:13:33] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[15:13:45] <MacFanBot>	 [02mw-config] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/01d5b440d75fa811b8bf7f4c2ef712341afc80aa
[15:13:45] <MacFanBot>	 02mw-config/03main 07paladox 0301d5b44 Adjust wgThumbnailBuckets and wgUploadThumbnailRenderMap (#6356)…
[15:13:46] <MacFanBot>	 [02mw-config] 07paladox merged pull request #6356: Adjust wgThumbnailBuckets and wgUploadThumbnailRenderMap (03main...03paladox-patch-6) 13https://github.com/miraheze/mw-config/pull/6356
[15:13:47] <MacFanBot>	 [02mw-config] 07paladox 04deleted 03paladox-patch-6 at 0391076a3 13https://api.github.com/repos/miraheze/mw-config/commit/91076a3
[15:14:15] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[15:14:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:14:28] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[15:14:29] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s
[15:14:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:14:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:14:41] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s
[15:14:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:14:58] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[16:51:42] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[16:55:42] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 52%
[16:57:42] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 10%
[17:09:17] <MacFanBot>	 [02TSPortal] 07renovate[bot] created 03renovate/prod-deps (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/e25515379dc2
[17:09:17] <MacFanBot>	 02TSPortal/03renovate/prod-deps 07renovate[bot] 03e255153 Update prod dependencies
[17:09:21] <MacFanBot>	 [02TSPortal] 07renovate[bot] added the label 'composer' to pull request #119 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/119
[17:09:21] <MacFanBot>	 [02TSPortal] 07renovate[bot] added the label 'composer' to pull request #119 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/119
[17:09:23] <MacFanBot>	 [02TSPortal] 07renovate[bot] added the label 'dependencies' to pull request #119 (Update prod dependencies) 13https://github.com/miraheze/TSPortal/pull/119
[17:09:25] <MacFanBot>	 [02TSPortal] 07renovate[bot] opened pull request #119: Update prod dependencies (03main...03renovate/prod-deps) 13https://github.com/miraheze/TSPortal/pull/119
[17:55:42] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 46%
[17:59:42] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 12%
[18:50:02] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to all
[18:50:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:51:40] <MirahezeRelay>	 <abaddriverlol> my laptop crashed, gg
[18:53:15] <MirahezeRelay>	 <abaddriverlol> had to kill mwdeploy since I didn't have access to the shell anymore after restarting
[18:53:22] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'upgrade_vendor': True, 'world': True, 'l10n': True, 'versions': '1.45'} to all
[18:53:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:58:52] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw161
[18:58:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:00:40] <MirahezeRelay>	 <abaddriverlol> [1/13] ```
[19:00:41] <MirahezeRelay>	 <abaddriverlol> [2/13] Apr 01 18:59:00 mw161 php[2261251]: PHP Fatal error:  Uncaught Error: Failed opening required '/srv/mediawiki/1.45/vendor/composer/../../extensions/SemanticMediaWiki/includes/GlobalFu>
[19:00:41] <MirahezeRelay>	 <abaddriverlol> [3/13] Apr 01 18:59:00 mw161 php[2261251]: Stack trace:
[19:00:41] <MirahezeRelay>	 <abaddriverlol> [4/13] Apr 01 18:59:00 mw161 php[2261251]: #0 /srv/mediawiki/1.45/vendor/composer/autoload_real.php(49): {closure:ComposerAutoloaderInit_mediawiki_vendor::getLoader():41}('b132fa83088803c...>
[19:00:42] <MirahezeRelay>	 <abaddriverlol> [5/13] Apr 01 18:59:00 mw161 php[2261251]: #1 /srv/mediawiki/1.45/vendor/autoload.php(22): ComposerAutoloaderInit_mediawiki_vendor::getLoader()
[19:00:42] <MirahezeRelay>	 <abaddriverlol> [6/13] Apr 01 18:59:00 mw161 php[2261251]: #2 /srv/mediawiki/1.45/includes/AutoLoader.php(348): require_once('/srv/mediawiki/...')
[19:00:42] <MirahezeRelay>	 <abaddriverlol> [7/13] Apr 01 18:59:00 mw161 php[2261251]: #3 /srv/mediawiki/1.45/includes/Setup.php(111): require_once('/srv/mediawiki/...')
[19:00:43] <MirahezeRelay>	 <abaddriverlol> [8/13] Apr 01 18:59:00 mw161 php[2261251]: #4 /srv/mediawiki/1.45/includes/WebStart.php(72): require_once('/srv/mediawiki/...')
[19:00:43] <MirahezeRelay>	 <abaddriverlol> [9/13] Apr 01 18:59:00 mw161 php[2261251]: #5 /srv/mediawiki/1.45/load.php(22): require('/srv/mediawiki/...')
[19:00:43] <MirahezeRelay>	 <abaddriverlol> [10/13] Apr 01 18:59:00 mw161 php[2261251]: #6 /srv/mediawiki/config/initialise/entrypoints/load.php(3): require('/srv/mediawiki/...')
[19:00:44] <MirahezeRelay>	 <abaddriverlol> [11/13] Apr 01 18:59:00 mw161 php[2261251]: #7 {main}
[19:00:44] <MirahezeRelay>	 <abaddriverlol> [12/13] Apr 01 18:59:00 mw161 php[2261251]:   thrown in /srv/mediawiki/1.45/vendor/composer/autoload_real.php on line 4
[19:00:45] <MirahezeRelay>	 <abaddriverlol> [13/13] ```
[19:03:32] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'upgrade_vendor': True, 'world': True, 'versions': '1.45'} to mwtask151
[19:03:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:03:39] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'upgrade_vendor': True, 'world': True, 'versions': '1.45'} to mwtask151 - SUCCESS in 7s
[19:03:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:04:35] <MirahezeRelay>	 <abaddriverlol> seems fine, might've been opcache, retrying
[19:04:44] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'upgrade_vendor': True, 'world': True, 'versions': '1.45'} to all
[19:04:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:06:03] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw163
[19:06:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:06:38] <MirahezeRelay>	 <abaddriverlol> uggggh
[19:07:47] <MirahezeRelay>	 <abaddriverlol> same error but everything looks fine afterwards again
[19:08:45] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'upgrade_vendor': True, 'world': True, 'versions': '1.45'} to all
[19:08:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:09:19] <MirahezeRelay>	 <abaddriverlol> gonna try one more time, otherwise I'll skip the canary check since it seems to succeed on the second try
[19:11:04] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw201
[19:11:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:13:20] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'world': True, 'force': True, 'versions': '1.45'} to all
[19:13:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:14:10] <MirahezeRelay>	 <abaddriverlol> @rhinosf1 I wonder if we can somehow solve this in mwdeploy
[19:14:20] <MirahezeRelay>	 <abaddriverlol> e.g. retry the canary check after x seconds if it fails
[19:15:02] <MirahezeRelay>	 <rhinosf1> Could do
[19:15:17] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'world': True, 'force': True, 'versions': '1.45'} to all - SUCCESS in 117s
[19:15:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:15:25] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'l10n': True, 'versions': '1.45'} to all
[19:15:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:18:53] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'l10n': True, 'versions': '1.45'} to all - SUCCESS in 207s
[19:18:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:21:09] <MirahezeRelay>	 <abaddriverlol> There seems to be a high rate of PHP warnings which I'll look into / file a task for, but otherwise looks fine (and nobody has complained in #general yet)
[19:25:00] <SomeRandomDev>	 !log sudo salt-ssh -E 'mw.*' cmd.run 'puppet agent -tv'
[19:25:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:48:09] <icinga-miraheze>	 PROBLEM - os162 PowerDNS Recursor on os162 is CRITICAL: No response from DNS 127.0.0.1
[21:49:44] <icinga-miraheze>	 PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: No response from DNS 127.0.0.1
[21:50:08] <icinga-miraheze>	 RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.028 seconds response time. os162.fsslc.wtnet returns 10.0.16.141
[21:50:32] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png]
[21:51:42] <icinga-miraheze>	 RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.178 seconds response time. db151.fsslc.wtnet returns 10.0.15.110
[22:00:28] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/60fc4775351ad29041b93e022cb2699d575fc619
[22:00:28] <MacFanBot>	 02ssl/03main 07WikiTideBot 0360fc477 Bot: Auto-update domain lists
[22:13:15] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'ProofreadPage'} to test151
[22:13:16] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'ProofreadPage'} to test151 - SUCCESS in 1s
[22:13:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:18] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'ProofreadPage'} to all
[22:13:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:13:44] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'ProofreadPage'} to all - SUCCESS in 26s
[22:13:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:18:32] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:47:42] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 51%
[22:51:42] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 39%
[23:31:24] <MacFanBot>	 [02ssl] 07danlessa opened pull request #907: Messed up DNS / Custom Domain for pedalhidrograficowiki when adding a second domain (07miraheze:03main...07danlessa:03main) 13https://github.com/miraheze/ssl/pull/907
[23:33:30] <MacFanBot>	 [02ssl] 07coderabbitai[bot] commented on pull request #907: No actionable comments were generated in the recent review. 🎉 […] 13https://github.com/miraheze/ssl/pull/907#issuecomment-4173578344