[00:01:19] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/9b2a68be4f13f620c6c94d742ee014ed81cc2109
[00:01:19] <MacFanBot>	 02statichelp/03main 07WikiTideBot 039b2a68b Bot: Auto-update Tech namespace pages 2025-08-31 00:01:17
[00:07:27] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57132MiB (12% inode=99%);
[00:07:33] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57379MiB (12% inode=99%);
[00:08:13] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56652MiB (12% inode=99%);
[01:31:18] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=awikilgjhwiki --new=coldlightwiki --user=Skye (END - exit=0)
[01:34:33] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=coldlightwiki (END - exit=0)
[01:34:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:45:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756604756851[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756586660000&orgId=1&to=1756604756851
[01:47:31] <MacFanBot>	 [02DiscordNotifications] 07pskyechology opened pull request #110: Fix displaying incorrect author on uploads when using Moderation (07miraheze:03main...07pskyechology:03uploadauthor) 13https://github.com/miraheze/DiscordNotifications/pull/110
[01:47:37] <MacFanBot>	 [02DiscordNotifications] 07coderabbitai[bot] commented on pull request #110: <details> […] 13https://github.com/miraheze/DiscordNotifications/pull/110#issuecomment-3239666746
[01:50:50] <miraheze-github>	 miraheze/DiscordNotifications - pskyechology the build has errored.
[02:03:48] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw183
[02:03:54] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw183
[02:03:58] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw183
[02:04:04] <icinga-miraheze>	 PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:05:48] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[02:05:52] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[02:05:56] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[02:06:03] <icinga-miraheze>	 RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 4371 bytes in 0.211 second response time
[02:23:43] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw172 mw191 mw193
[02:23:48] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw172 mw191 mw193
[02:23:52] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw172 mw191 mw193
[02:24:03] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw172 mw193
[02:24:38] <MirahezeRelay>	 <zppix> Hmmm
[02:24:48] <MirahezeRelay>	 <zppix> Thats probably not good
[02:25:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.16, 17.49, 9.22
[02:25:41] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[02:25:48] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[02:25:52] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[02:26:03] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[02:29:20] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.45, 8.54, 7.39
[02:39:19] <miraheze-github>	 miraheze/DiscordNotifications - pskyechology the build has errored.
[02:56:30] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 52.92, 25.47, 11.53
[02:57:18] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw192
[02:57:48] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw192
[02:57:52] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw192
[02:58:03] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw192
[02:59:16] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[02:59:48] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[02:59:52] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[03:00:03] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[03:02:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.90, 11.55, 10.03
[03:04:03] <icinga-miraheze>	 PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 2059MiB (6% inode=81%);
[03:04:27] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.68, 8.17, 8.97
[03:08:03] <icinga-miraheze>	 RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4158MiB (13% inode=81%);
[03:08:34] <MirahezeRelay>	 <pskyechology> minor amounts of eepy
[03:11:52] <miraheze-github>	 miraheze/DiscordNotifications - pskyechology the build passed.
[03:12:07] <MacFanBot>	 [02DiscordNotifications] 07pskyechology merged pull request #110: Fix displaying incorrect author on uploads when using Moderation (07miraheze:03main...07pskyechology:03uploadauthor) 13https://github.com/miraheze/DiscordNotifications/pull/110
[03:12:07] <MacFanBot>	 [02DiscordNotifications] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/da2bece2dfb7b8222adac580080673c9bda3d96f
[03:12:09] <MacFanBot>	 02DiscordNotifications/03main 07Skye 03da2bece Fix displaying incorrect author on uploads when using Moderation (#110)…
[03:13:32] <MirahezeLSBot>	 !log [skye@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'DiscordNotifications'} to all
[03:13:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:14:54] <MirahezeRelay>	 <pskyechology> temporary fail in name resolution i wonder why :3
[03:15:04] <MirahezeRelay>	 <pskyechology> totally not trying to upgrade exts while they're taking a nap
[03:19:59] <MirahezeRelay>	 <pskyechology> damn even icinga is not resolvable
[03:21:15] <MirahezeRelay>	 <pskyechology> nvm i am have stupid
[03:21:37] <MirahezeRelay>	 <pskyechology> ok round 2 lets ball
[03:21:41] <MirahezeLSBot>	 !log [skye@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_extensions': 'DiscordNotifications'} to all
[03:21:46] <MirahezeLSBot>	 !log [skye@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_extensions': 'DiscordNotifications'} to all - SUCCESS in 4s
[03:21:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:21:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:22:00] <MirahezeRelay>	 <pskyechology> EXCUSE ME
[03:23:52] <MirahezeRelay>	 <cosmicalpha, replying to pskyechology> use --force-upgrade
[03:24:07] <MirahezeRelay>	 <pskyechology> thanks i was just looking for the right flavor for force
[03:24:29] <MirahezeLSBot>	 !log [skye@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'DiscordNotifications'} to all
[03:24:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:25:11] <MirahezeLSBot>	 !log [skye@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'DiscordNotifications'} to all - SUCCESS in 42s
[03:25:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:35:43] <miraheze-github>	 miraheze/DiscordNotifications - pskyechology the build passed.
[05:45:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756619156882[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756586660000&orgId=1&to=1756619156882
[06:21:39] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.80, 20.84, 20.17
[06:23:38] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.86, 20.24, 20.05
[06:31:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.39, 20.76, 20.35
[06:35:32] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.76, 20.39, 20.32
[06:43:27] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.81, 20.64, 20.45
[06:45:26] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.23, 20.34, 20.38
[06:49:23] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.31, 20.63, 20.48
[06:51:22] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.67, 20.31, 20.39
[06:55:20] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.31, 20.56, 20.49
[07:15:10] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.83, 20.12, 20.40
[07:23:05] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.51, 20.38, 20.43
[08:02:13] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49843MiB (10% inode=99%);
[08:45:27] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49865MiB (10% inode=99%);
[08:53:33] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49929MiB (10% inode=99%);
[09:44:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.60, 20.10, 20.39
[09:45:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756633556918[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756586660000&orgId=1&to=1756633556918
[09:48:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.47, 20.58, 20.51
[09:50:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.22, 20.24, 20.39
[10:06:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.17, 20.66, 20.24
[10:10:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.17, 20.19, 20.17
[11:34:43] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.20, 20.64, 19.84
[11:38:41] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.51, 20.19, 19.86
[12:02:27] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.65, 20.38, 19.86
[12:04:26] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.79, 19.98, 19.78
[12:32:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.48, 19.99, 19.68
[12:34:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.01, 20.00, 19.72
[13:45:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756647956947[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756586660000&orgId=1&to=1756647956947
[15:45:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756655156965[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756651520000&orgId=1&to=1756655120000
[15:54:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.95, 11.10, 5.65
[15:55:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756655756967[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652120000&orgId=1&to=1756655756967
[15:56:44] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.99, 7.89, 5.11
[16:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756656656969[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756656656969[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https
[16:10:57] <icinga-miraheze>	 .wikitide.net/d/GtxbP1Xnk?from=1756652120000&orgId=1&to=1756656620000
[16:40:34] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.34, 10.24, 6.22
[16:42:31] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.61, 7.28, 5.62
[17:19:17] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.60, 10.52, 5.77
[17:21:14] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.52, 8.77, 5.64
[18:06:45] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.51, 10.71, 7.36
[18:08:42] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 11.82, 13.12, 8.76
[18:10:38] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.42, 9.18, 7.83
[18:20:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 23.54, 15.00, 9.88
[18:24:27] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.80, 9.72, 8.90
[18:30:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 8.97, 14.34, 11.32
[18:32:27] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.84, 9.84, 10.04
[18:42:45] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 63%
[18:44:45] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42%
[18:46:45] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 8%
[18:50:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 9.31, 12.65, 9.47
[18:52:27] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.32, 8.84, 8.45
[18:56:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 6.87, 12.98, 10.65
[19:08:27] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.99, 7.84, 11.51
[19:12:27] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.00, 4.02, 9.09
[19:45:25] <icinga-miraheze>	 PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1965MiB (10% inode=90%);
[20:02:59] <icinga-miraheze>	 RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6743MiB (37% inode=90%);
[20:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756671057004[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756671057004
[22:20:22] <MirahezeRelay>	 <pskyechology> i just realized we had 12 million jobs at one point
[22:37:17] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Testing pdns issues --Void, last run 3 minutes ago with 0 failures
[22:48:03] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php purgeParserCache --wiki=falcomwiki --age=3600 (END - exit=0)
[22:48:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:55:18] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[23:48:49] <MirahezeRelay>	 <abaddriverlol, replying to pskyechology> we're probably the biggest employer of the entire US