[00:05:46] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26781MiB (5% inode=99%);
[00:06:30] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 26567MiB (5% inode=99%);
[00:08:30] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 55075MiB (12% inode=99%);
[00:09:46] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55287MiB (12% inode=99%);
[00:30:09] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.04, 18.17, 12.05
[00:34:09] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.40, 19.70, 14.08
[00:36:52] <icinga-miraheze>	 PROBLEM - cp201 APT on cp201 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (14 critical updates).
[00:36:54] <MirahezeLSBot>	 !log [universalomega@cp201] test
[00:36:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:38:11] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0af8b5345c246c390c43acffed4f6a54169903de
[00:38:11] <MacFanBot>	 02puppet/03main 07CosmicAlpha 030af8b53 Add cp201 to cache role
[00:38:52] <icinga-miraheze>	 RECOVERY - cp201 APT on cp201 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[00:49:25] <MirahezeRelay>	 <cosmicalpha> !log created cp201
[00:49:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:52:30] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e2c7992f9255f10073432ed367499c6732f7f21b
[00:52:30] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03e2c7992 varnish: add cp201 to local_tls_terminator
[00:53:50] <MacFanBot>	 [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/c8d84bb2f57ad587f3303c59302f871f20be54b6
[00:53:50] <MacFanBot>	 02dns/03main 07CosmicAlpha 03c8d84bb Add cp201 PTR
[00:54:49] <miraheze-github>	 miraheze/dns - Universal-Omega the build passed.
[00:55:36] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/7ffc72ae643eef8149a19c678e256eaf17d60fa3
[00:55:36] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 037ffc72a Add cp201 to `$wgCdnServers`
[00:56:30] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:57:12] <MacFanBot>	 [02puppet] 07Universal-Omega created 03cp201 (+1 new commit) 13https://github.com/miraheze/puppet/commit/8ebedcd091e0
[00:57:12] <MacFanBot>	 02puppet/03cp201 07CosmicAlpha 038ebedcd Pool in cp201
[00:57:16] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4409: Pool in cp201 (03main...03cp201) 13https://github.com/miraheze/puppet/pull/4409
[00:57:21] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4409: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4409#issuecomment-2986173854
[00:58:33] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[00:58:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:58:59] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s
[00:59:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:59:18] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4409: Pool in cp201 (03main...03cp201) 13https://github.com/miraheze/puppet/pull/4409
[00:59:19] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/19ed0302bc8d54ae966743d38c92719b31045d66
[00:59:19] <MacFanBot>	 02puppet/03main 07CosmicAlpha 0319ed030 Pool in cp201
[00:59:19] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03cp201 at 038ebedcd 13https://api.github.com/repos/miraheze/puppet/commit/8ebedcd
[01:08:23] <MirahezeRelay>	 <cosmicalpha> !log added cp201 to load balancer pool on cloudflare
[01:08:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:29:10] <MacFanBot>	 [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/eeb280e02e02b8ca22e14e68ff09963d3a9d402a
[01:29:10] <MacFanBot>	 02ssl/03main 07MacFan4000 03eeb280e add nimby.connie.moe and docs.ben3coder.dev
[01:32:50] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7c5dceed479e00e3873f4a7bc869a20b4aa3c4c9
[01:32:50] <MacFanBot>	 02puppet/03main 07CosmicAlpha 037c5dcee Run update-static-tech-docs daily (for now)
[02:04:49] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/6e20bcc48eaaada0e254a4f0daacfe73efa377c2
[02:04:49] <MacFanBot>	 02statichelp/03main 07WikiTideBot 036e20bcc Auto-update Tech namespace pages 2025-06-19 02:04:47.251635
[06:01:22] <MirahezeRelay>	 <songngu.xyz> `[db692be75c45df88c7971c21] 2025-06-19 06:01:06: Lỗi chí tử loại “LogicException”`
[06:01:56] <MirahezeRelay>	 <songngu.xyz> Something old from sematic mw, giving me error when try to delete the property
[06:02:09] <MirahezeRelay>	 <songngu.xyz> `https://lophocmatngu.wiki/Property:Gi%E1%BB%9Bi_t%C3%ADnh_nh%C3%A2n_v%E1%BA%ADt?action=delete`
[06:55:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 17.72, 12.16
[07:05:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.96, 23.97, 18.36
[07:07:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 24.31, 19.17
[07:10:43] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 21.05, 18.13, 11.62
[07:16:43] <icinga-miraheze>	 RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 19.50, 19.84, 14.58
[07:29:06] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49878MiB (10% inode=99%);
[07:29:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 12.47, 20.04, 21.90
[07:33:53] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.97, 17.32, 20.23
[07:46:29] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49948MiB (10% inode=99%);
[08:11:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.48, 18.65, 13.58
[08:13:20] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.68, 17.48, 11.50
[08:15:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 21.75, 16.03
[08:17:20] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 18.99, 18.67, 13.38
[08:19:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.98, 23.22, 17.99
[08:21:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 23.40, 18.68
[08:25:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.30, 23.67, 19.89
[08:37:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.09, 23.84, 21.78
[09:56:44] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=vocaloidlyricswiki --import-prefix=fandom:vocaloidlyrics --from=Aceredshirt13 --no-run (END - exit=0)
[09:56:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:57:27] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=vocaloidlyricswiki --import-prefix=fandom:vocaloidlyrics --from=Aceredshirt13 (END - exit=0)
[09:57:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:08:17] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/attachLatest.php --wiki=lhmnwiki (END - exit=0)
[10:08:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:08:43] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/attachLatest.php --wiki=lhmnwiki --fix (END - exit=0)
[10:08:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:09:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.26, 21.46, 23.69
[10:13:24] <MirahezeRelay>	 <abaddriverlol, replying to songngu.xyz> opened <https://issue-tracker.miraheze.org/T13870> to track this
[10:25:53] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.08, 19.27, 20.40
[10:45:02] <MacFanBot>	 [02mw-config] 07SomeMWDev approved pull request #5993 13https://github.com/miraheze/mw-config/pull/5993#pullrequestreview-2942486053
[10:45:15] <MacFanBot>	 [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/368dd595429479a1f2a9192653e52768a97400b1
[10:45:15] <MacFanBot>	 02mw-config/03main 07Asko 03368dd59 T13868: Change Semantic MediaWiki footer icon for universalunionwiki (#5993)…
[10:45:15] <MacFanBot>	 [02mw-config] 07SomeMWDev merged 07asko1's pull request #5993: T13868: Change Semantic MediaWiki footer icon for universalunionwiki (07miraheze:03main...07asko1:03T13868) 13https://github.com/miraheze/mw-config/pull/5993
[10:45:36] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[10:45:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:46:00] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s
[10:46:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:46:13] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[10:53:19] <MirahezeRelay>	 <abaddriverlol> !log ran "DELETE FROM page WHERE page_id = 810;" on lhmnwiki (T13870)
[10:53:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:03:17] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev created 03themetoggle-branch (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/9e0029dd8bef
[11:03:17] <MacFanBot>	 02mediawiki-repos/03themetoggle-branch 07SomeRandomDeveloper 039e0029d Use REL1_43 as the branch for ThemeToggle…
[11:03:46] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev opened pull request #76: Use REL1_43 as the branch for ThemeToggle (03main...03themetoggle-branch) 13https://github.com/miraheze/mediawiki-repos/pull/76
[11:03:51] <MacFanBot>	 [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #76: <details open="true"> […] 13https://github.com/miraheze/mediawiki-repos/pull/76#issuecomment-2987670196
[11:06:26] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev merged pull request #76: Use REL1_43 as the branch for ThemeToggle (03main...03themetoggle-branch) 13https://github.com/miraheze/mediawiki-repos/pull/76
[11:06:26] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/5f57c64789b1488dc8c85b74062cbd664c05aeaf
[11:06:26] <MacFanBot>	 02mediawiki-repos/03main 07SomeRandomDeveloper 035f57c64 Use REL1_43 as the branch for ThemeToggle (#76)…
[11:06:27] <MacFanBot>	 [02mediawiki-repos] 07SomeMWDev 04deleted 03themetoggle-branch at 039e0029d 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/9e0029d
[11:07:16] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'ThemeToggle'} to test151
[11:07:18] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'ThemeToggle'} to test151 - SUCCESS in 2s
[11:07:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:07:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:14:46] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'ThemeToggle'} to all
[11:14:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:17:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 35.80, 23.79, 18.66
[11:19:02] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:19:10] <icinga-miraheze>	 PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:19:28] <MirahezeRelay>	 <abaddriverlol> mwtask181 is not liking l10n...
[11:19:57] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[11:20:02] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[11:20:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:20:59] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.326 second response time
[11:21:06] <icinga-miraheze>	 RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u6 (protocol 2.0)
[11:21:57] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.039 second response time
[11:28:53] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23016MiB (10% inode=95%);
[11:34:53] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 34144MiB (15% inode=95%);
[11:35:11] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_43 ThemeToggle]
[11:35:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 11.58, 18.50, 22.39
[11:37:30] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_43 ThemeToggle],Exec[git_checkout_MediaWiki-REL1_44 ThemeToggle]
[11:41:53] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.55, 15.02, 19.56
[11:44:02] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-11 (+1 new commit) 13https://github.com/miraheze/puppet/commit/188d044d9341
[11:44:02] <MacFanBot>	 02puppet/03paladox-patch-11 07paladox 03188d044 changeprop201: Decrease notificationGetStartedJob reenqueue_delay…
[11:44:06] <MacFanBot>	 [02puppet] 07paladox opened pull request #4410: changeprop201: Decrease notificationGetStartedJob reenqueue_delay (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4410
[11:44:11] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4410: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4410#issuecomment-2987776454
[11:44:17] <MacFanBot>	 [02puppet] 07paladox merged pull request #4410: changeprop201: Decrease notificationGetStartedJob reenqueue_delay (03main...03paladox-patch-11) 13https://github.com/miraheze/puppet/pull/4410
[11:44:17] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5b50e0b8e5ab2baea32179f40701ba7bb6f2cdd3
[11:44:17] <MacFanBot>	 02puppet/03main 07paladox 035b50e0b changeprop201: Decrease notificationGetStartedJob reenqueue_delay (#4410)…
[11:44:18] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-11 at 03188d044 13https://api.github.com/repos/miraheze/puppet/commit/188d044
[11:45:28] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:53:16] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[11:56:12] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'ThemeToggle'} to all
[11:56:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:57:02] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:57:06] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'ThemeToggle'} to all - SUCCESS in 54s
[11:57:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:57:23] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b8bd3e63273e8b4512b6d8ecf403ec59da43c63d
[11:57:23] <MacFanBot>	 02puppet/03main 07paladox 03b8bd3e6 mediawiki_task: lower mediawiki::php::fpm::fpm_workers_multiplier to 1.5
[11:58:56] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 4252 bytes in 0.204 second response time
[12:00:49] <icinga-miraheze>	 PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[12:00:53] <icinga-miraheze>	 PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:00:54] <icinga-miraheze>	 PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[12:00:56] <icinga-miraheze>	 PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:01:01] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[12:01:04] <icinga-miraheze>	 PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[12:01:09] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:01:10] <icinga-miraheze>	 PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[12:01:12] <icinga-miraheze>	 PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[12:01:13] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received
[12:01:19] <icinga-miraheze>	 PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[12:02:47] <icinga-miraheze>	 RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time
[12:02:48] <icinga-miraheze>	 RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time
[12:02:51] <icinga-miraheze>	 RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 4252 bytes in 0.185 second response time
[12:02:52] <icinga-miraheze>	 RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time
[12:02:59] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time
[12:03:00] <icinga-miraheze>	 RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 275 bytes in 0.015 second response time
[12:03:06] <icinga-miraheze>	 RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time
[12:03:07] <icinga-miraheze>	 RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time
[12:03:08] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 4252 bytes in 0.187 second response time
[12:03:09] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time
[12:03:18] <icinga-miraheze>	 RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.007 second response time
[12:11:11] <MacFanBot>	 [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/5389a9537d1da058b4879ac4c9d66132aed66e88
[12:11:11] <MacFanBot>	 02DiscordNotifications/03main 07translatewiki.net 035389a95 Localisation updates from https://translatewiki.net.
[12:11:13] <MacFanBot>	 [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/f72651596931297682dd19fd1a6f38dfe7264726
[12:11:14] <MacFanBot>	 02ManageWiki/03main 07translatewiki.net 03f726515 Localisation updates from https://translatewiki.net.
[12:11:16] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/9d0f15e9bdd197ac9dfd9174963ae92b3a26ebaf
[12:11:18] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 039d0f15e Localisation updates from https://translatewiki.net.
[12:17:01] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:24:05] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:32:08] <miraheze-github>	 miraheze/DiscordNotifications - translatewiki the build passed.
[13:36:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750336580000&orgId=1&to=1750340213352
[13:51:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750337480000&orgId=1&to=1750341080000
[14:07:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750338440000&orgId=1&to=1750342073354
[14:17:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750338800000&orgId=1&to=1750342400000
[15:16:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750342580000&orgId=1&to=1750346213356
[15:21:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750342880000&orgId=1&to=1750346480000
[15:37:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750343840000&orgId=1&to=1750347473353
[15:57:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750344860000&orgId=1&to=1750348460000
[16:13:51] <MacFanBot>	 [02mw-config] 07songnguxyz commented on pull request #5529: I wonder where will this be getting us to... 13https://github.com/miraheze/mw-config/pull/5529#issuecomment-2988615016
[16:14:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750346060000&orgId=1&to=1750349693352
[16:24:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750346480000&orgId=1&to=1750350080000
[16:36:21] <MacFanBot>	 [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03periodic_jobs 13https://github.com/miraheze/pywikibot-config/commit/b316504c02e9159c743de59e991d831c65186099
[16:36:21] <MacFanBot>	 02pywikibot-config/03periodic_jobs 07CosmicAlpha 03b316504 Update
[16:38:49] <MacFanBot>	 [02pywikibot-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/pywikibot-config/commit/1ba7d23d6ee4c9b52eabedf6a58a93edec1c8fcf
[16:38:50] <MacFanBot>	 02pywikibot-config/03main 07CosmicAlpha 031ba7d23 Rename timers to periodic_jobs…
[16:38:52] <MacFanBot>	 [02pywikibot-config] 07Universal-Omega merged pull request #6: Rename timers to periodic_jobs (03main...03periodic_jobs) 13https://github.com/miraheze/pywikibot-config/pull/6
[16:38:54] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/05452167db69d9acf43476efaa4fc61e6c9f638a
[16:38:55] <MacFanBot>	 02puppet/03main 07CosmicAlpha 030545216 Use periodic_jobs
[16:38:57] <MacFanBot>	 [02pywikibot-config] 07Universal-Omega 04deleted 03periodic_jobs at 03b316504 13https://api.github.com/repos/miraheze/pywikibot-config/commit/b316504
[16:43:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750347800000&orgId=1&to=1750351433355
[16:53:42] <MacFanBot>	 [02puppet] 07Universal-Omega created 03systemd-monitor-critical (+1 new commit) 13https://github.com/miraheze/puppet/commit/350e0b8c0e3e
[16:53:42] <MacFanBot>	 02puppet/03systemd-monitor-critical 07CosmicAlpha 03350e0b8 systemd::monitor: add support for setting critical
[16:53:46] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4411: systemd::monitor: add support for setting critical (03main...03systemd-monitor-critical) 13https://github.com/miraheze/puppet/pull/4411
[16:53:51] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4411: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4411#issuecomment-2988693010
[16:53:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750348400000&orgId=1&to=1750352000000
[16:54:48] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4411: systemd::monitor: add support for setting critical (03main...03systemd-monitor-critical) 13https://github.com/miraheze/puppet/pull/4411
[16:54:48] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/61f18c358fa27ad5ecc00e501870c654d6edca7a
[16:54:48] <MacFanBot>	 02puppet/03main 07CosmicAlpha 0361f18c3 systemd::monitor: add support for setting critical
[16:54:49] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03systemd-monitor-critical at 03350e0b8 13https://api.github.com/repos/miraheze/puppet/commit/350e0b8
[16:55:37] <MacFanBot>	 [02puppet] 07Universal-Omega created 03systemd-monitor-services (+1 new commit) 13https://github.com/miraheze/puppet/commit/45bd402479e5
[16:55:37] <MacFanBot>	 02puppet/03systemd-monitor-services 07CosmicAlpha 0345bd402 systemd: add support for monitoring services
[16:55:42] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4412: systemd: add support for monitoring services (03main...03systemd-monitor-services) 13https://github.com/miraheze/puppet/pull/4412
[16:55:47] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4412: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4412#issuecomment-2988696226
[16:56:23] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor-services 13https://github.com/miraheze/puppet/commit/461747d781ea075f9fd2a366f56d0d5ae9c2e2aa
[16:56:23] <MacFanBot>	 02puppet/03systemd-monitor-services 07CosmicAlpha 03461747d Fix
[16:57:44] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor-services 13https://github.com/miraheze/puppet/commit/95596531b532993d61d3bc56e724e10387a61302
[16:57:44] <MacFanBot>	 02puppet/03systemd-monitor-services 07CosmicAlpha 039559653 Use https
[16:58:07] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03systemd-monitor-services 13https://github.com/miraheze/puppet/commit/de2ceb7abd58fd1a3b8c289e71158c49b1f35849
[16:58:07] <MacFanBot>	 02puppet/03systemd-monitor-services 07CosmicAlpha 03de2ceb7 Alignment
[17:09:28] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4412: systemd: add support for monitoring services (03main...03systemd-monitor-services) 13https://github.com/miraheze/puppet/pull/4412
[17:09:29] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/04704a331525bc86d6a84c341c934e27c4b3dcf1
[17:09:30] <MacFanBot>	 02puppet/03main 07CosmicAlpha 0304704a3 systemd: add support for monitoring services
[17:09:32] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03systemd-monitor-services at 03de2ceb7 13https://api.github.com/repos/miraheze/puppet/commit/de2ceb7
[19:00:59] <MacFanBot>	 [02puppet] 07Universal-Omega created 03backups-cleanup from 03main (+0 new commit) 13https://github.com/miraheze/puppet/compare/backups-cleanup
[19:02:30] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03backups-cleanup 13https://github.com/miraheze/puppet/commit/a8d31f8a3ffe4fd82476610b7f2e16e4dda6f6b0
[19:02:30] <MacFanBot>	 02puppet/03backups-cleanup 07CosmicAlpha 03a8d31f8 Cleanup and normalize backups
[19:02:46] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4413: Cleanup and normalize backups (03main...03backups-cleanup) 13https://github.com/miraheze/puppet/pull/4413
[19:02:53] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4413: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4413#issuecomment-2988930280
[19:05:12] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03backups-cleanup 13https://github.com/miraheze/puppet/commit/848aa3227722292a63f70aeed8a5db46f2c20541
[19:05:12] <MacFanBot>	 02puppet/03backups-cleanup 07CosmicAlpha 03848aa32 Fix
[19:05:42] <MacFanBot>	 [02puppet] 07github-actions[bot] pushed 1 new commit to 03backups-cleanup 13https://github.com/miraheze/puppet/commit/632cd41d24248803d3fd8c1055178f3a18b403bd
[19:05:44] <MacFanBot>	 02puppet/03backups-cleanup 07github-actions 03632cd41 CI: lint puppet code to standards…
[19:06:46] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03backups-cleanup 13https://github.com/miraheze/puppet/commit/96d1f1b0b98307231099a788aee47144fe34283e
[19:06:46] <MacFanBot>	 02puppet/03backups-cleanup 07CosmicAlpha 0396d1f1b Fix
[19:27:52] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49846MiB (10% inode=99%);
[20:56:53] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23119MiB (10% inode=95%);
[21:00:53] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 27532MiB (12% inode=95%);
[21:42:08] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03backups-cleanup 13https://github.com/miraheze/puppet/commit/69b821413efb29b8fa112e2f83a2aa1325e42234
[21:42:08] <MacFanBot>	 02puppet/03backups-cleanup 07CosmicAlpha 0369b8214 Rename
[21:42:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750365740000&orgId=1&to=1750369373355
[21:47:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750365980000&orgId=1&to=1750369580000
[22:25:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750368320000&orgId=1&to=1750371953352
[22:30:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750368500000&orgId=1&to=1750372100000
[22:52:30] <MacFanBot>	 [02puppet] 07Universal-Omega created 03techdocs-commit (+1 new commit) 13https://github.com/miraheze/puppet/commit/125681ae0159
[22:52:30] <MacFanBot>	 02puppet/03techdocs-commit 07CosmicAlpha 03125681a techdocs: add Bot prefix to commit message and update time format
[22:52:34] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4414: techdocs: add Bot prefix to commit message and update time format (03main...03techdocs-commit) 13https://github.com/miraheze/puppet/pull/4414
[22:52:40] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4414: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4414#issuecomment-2989293852
[22:53:59] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[22:54:19] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4414: techdocs: add Bot prefix to commit message and update time format (03main...03techdocs-commit) 13https://github.com/miraheze/puppet/pull/4414
[22:54:20] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2e3556d6a699d6e8b50e6868f2960bd4b57b4a2f
[22:54:20] <MacFanBot>	 02puppet/03main 07CosmicAlpha 032e3556d techdocs: add Bot prefix to commit message and update time format
[22:54:20] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03techdocs-commit at 03125681a 13https://api.github.com/repos/miraheze/puppet/commit/125681a
[22:55:30] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[23:09:04] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/7d9178c7fe53cdf51d7bcde80a488d3db1e72348
[23:09:04] <MacFanBot>	 02puppet/03main 07CosmicAlpha 037d9178c techdocs: fix a formatting issue
[23:10:14] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[23:10:47] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/28f94654121a2b0f342e8d56b52746249972e3cc
[23:10:47] <MacFanBot>	 02puppet/03main 07CosmicAlpha 0328f9465 Remove TODO
[23:12:30] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[23:15:54] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b327a6502e7db6f1d8387aa1520e912095a1ada3
[23:15:54] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03b327a65 techdocs: don't have empty * lines
[23:16:57] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[23:18:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750371500000&orgId=1&to=1750375133353
[23:38:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750372520000&orgId=1&to=1750376120000
[23:55:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1750373720000&orgId=1&to=1750377353354