[00:00:02] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[00:00:03] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[00:00:08] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[00:00:53] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:01:02] <icinga-miraheze>	 RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:01:14] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:01:15] <icinga-miraheze>	 RECOVERY - swiftobject201 Puppet on swiftobject201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:01:22] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[00:09:31] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 53780MiB (11% inode=99%);
[00:11:25] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 54073MiB (11% inode=99%);
[01:21:37] <MirahezeLSBot>	 !log [blankeclair@mwtask181] Starting import for luntikfanonwiki (XML: None; Images: LuntikFanonWikiImagesImport) (START)
[01:21:38] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=luntikfanonwiki '--comment=Importing images from https://luntikfanon.fandom.com/ru/ ([[phorge:T13824|T13824]])' -- LuntikFanonWikiImagesImport (START)
[01:21:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:21:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:38:32] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=luntikfanonwiki '--comment=Importing images from https://luntikfanon.fandom.com/ru/ ([[phorge:T13824|T13824]])' -- LuntikFanonWikiImagesImport (END - exit=0)
[01:38:33] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=luntikfanonwiki --update (START)
[01:38:34] <MirahezeLSBot>	 !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=luntikfanonwiki --update (END - exit=0)
[01:38:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:38:35] <MirahezeLSBot>	 !log [blankeclair@mwtask181] Finished import for luntikfanonwiki (XML: None; Images: LuntikFanonWikiImagesImport) (END - exit=0)
[01:38:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:38:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:38:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:30:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749605420000&orgId=1&to=1749609053347
[02:35:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749605600000&orgId=1&to=1749609200000
[03:41:31] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49861MiB (10% inode=99%);
[03:55:25] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49918MiB (10% inode=99%);
[05:39:58] <MacFanBot>	 [02ssl] 07BlankEclair opened pull request #860: T13791: Redirect www.wikimoma.art to wikimoma.art (07miraheze:03main...07BlankEclair:03T13791) 13https://github.com/miraheze/ssl/pull/860
[05:40:26] <MacFanBot>	 [02ssl] 07BlankEclair merged pull request #860: T13791: Redirect www.wikimoma.art to wikimoma.art (07miraheze:03main...07BlankEclair:03T13791) 13https://github.com/miraheze/ssl/pull/860
[05:40:28] <MacFanBot>	 [02ssl] 07BlankEclair pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f13c2607a4747f0ed4522a953aa5bfb41daae0ee
[05:40:29] <MacFanBot>	 02ssl/03main 07Claire Elaina 03f13c260 T13791: Redirect www.wikimoma.art to wikimoma.art (#860)…
[05:43:13] <MacFanBot>	 [02ssl] 07coderabbitai[bot] commented on pull request #860: A new redirect entry named `wwwwikimomaart` was added to the `redirects.yaml` configuration file. This entry redirects traffic from `www.wikimoma.art` to `wikimoma.art` and specifies SSL and certificate authority details. No other modifications were […] 13https://github.com/miraheze/ssl/pull/860#issuecomment-2961326944
[07:57:14] <icinga-miraheze>	 PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[07:57:49] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[07:58:03] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[07:58:49] <icinga-miraheze>	 PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[07:59:15] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[07:59:21] <RhinosF1>	 Eh
[07:59:29] <RhinosF1>	 BlankEclair: what's the error
[08:00:06] <MirahezeRelay>	 <blankeclair> oh fuck my life
[08:00:20] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:01:03] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:02:27] <MirahezeRelay>	 <blankeclair> aaaaaand i don't know where puppet is
[08:02:55] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_Pywikibot-stable],Exec[git_pull_IRC-Discord-Relay-relaybot]
[08:02:57] <MirahezeRelay>	 <blankeclair> whatevs, the identifier is puppet-agent
[08:02:59] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:04:55] <BlankEclair>	 RhinosF1: Jun 11 07:59:06 mw192 puppet-agent[813010]: (/Stage[main]/Mediawiki::Multiversion/Git::Clone[femiwiki-deploy-1.43]/Exec[git_pull_femiwiki-deploy-1.43]/returns) fatal: unable to access 'https://github.com/miraheze/femiwiki-deploy/': CONNECT tunnel failed, response 503
[08:05:48] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet],Exec[git_pull_ssl],Exec[git_pull_mediawiki-repos],Exec[git_pull_pywikibot-config]
[08:05:56] <icinga-miraheze>	 PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:07:44] <BlankEclair>	 squid on bastion is responding HTTP/1.1 503 Service Unavailable to `all_proxy=http://bastion.fsslc.wtnet:8080 curl -v https://github.com/miraheze/femiwiki-deploy/`
[08:08:08] <BlankEclair>	 < X-Squid-Error: ERR_CONNECT_FAIL 110
[08:08:46] <BlankEclair>	 did we get ip banned by github lmfao
[08:09:25] <BlankEclair>	 when doing a HTTP instead of HTTPS: <p id="sysmsg">The system returned: <i>(110) Connection timed out</i></p>
[08:09:37] <BlankEclair>	 (ping me next if you need me)
[08:09:54] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:10:24] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[08:10:26] <icinga-miraheze>	 PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:11:53] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:12:59] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:13:31] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:19:42] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[08:20:01] <icinga-miraheze>	 PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43]
[08:20:02] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phorge-extensions]
[08:20:21] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.43]
[08:21:43] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid]
[08:23:15] <icinga-miraheze>	 RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:23:15] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:23:52] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:24:03] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[08:25:02] <icinga-miraheze>	 RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:26:20] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[08:27:03] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[08:28:55] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[08:29:05] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[08:31:46] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:31:56] <icinga-miraheze>	 RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:35:56] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[08:36:23] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:36:26] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[08:37:12] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:37:53] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[08:39:28] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:45:44] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:45:54] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[08:46:25] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[08:47:44] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[08:47:58] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:19:59] <MacFanBot>	 [02DiscordNotifications] 07asko1 opened pull request #101: T13820: Add support for notifying page undeletions (07miraheze:03main...07asko1:03PageUndeleteComplete) 13https://github.com/miraheze/DiscordNotifications/pull/101
[09:20:06] <MacFanBot>	 [02DiscordNotifications] 07coderabbitai[bot] commented on pull request #101: <details open="true"> […] 13https://github.com/miraheze/DiscordNotifications/pull/101#issuecomment-2961881058
[09:21:26] <MirahezeRelay>	 <cosmicalpha> @asko1 why does that hook need to pass so many variables lol. That's a lot...
[09:24:48] <MirahezeRelay>	 <asko1, replying to cosmicalpha> [1/2] just the way it came, I don't know what to say lol
[09:24:48] <MirahezeRelay>	 <asko1, replying to cosmicalpha> [2/2] gonna blame it on being introduced in 1.40 (rather recently = more bells and whistles)
[09:25:56] <MirahezeRelay>	 <cosmicalpha> lol yeah I dont like hooks with so many parameters lol. Anyway I'll review and merge your PR if at least the 1.43 CI passes if 1.44 is still unhappy for unrelated reasons.
[09:26:12] <MirahezeRelay>	 <cosmicalpha> probably not now though — it's 3:30AM
[09:27:22] <MirahezeRelay>	 <asko1> [1/2] yeah that's fair
[09:27:22] <MirahezeRelay>	 <asko1> [2/2] btw you might wanna consider upping the version at some point, been stuck at 3.0.0 for like 2 years :P
[09:41:25] <miraheze-github>	 miraheze/DiscordNotifications - asko1 the build passed.
[09:41:30] <MirahezeRelay>	 <asko1> wow? all of them passed?
[10:05:34] <MacFanBot>	 [02DiscordNotifications] 07Universal-Omega merged 07asko1's pull request #101: T13820: Add support for notifying page undeletions (07miraheze:03main...07asko1:03PageUndeleteComplete) 13https://github.com/miraheze/DiscordNotifications/pull/101
[10:05:34] <MacFanBot>	 [02DiscordNotifications] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/186c8c402e0f55d1cc59d75ced004519eca5fb3a
[10:05:36] <MacFanBot>	 02DiscordNotifications/03main 07Asko 03186c8c4 T13820: Add support for notifying page undeletions (#101)…
[10:06:01] <MirahezeRelay>	 <cosmicalpha> @aeywoo ^
[10:06:18] <MirahezeRelay>	 <cosmicalpha> @asko1 ^
[10:06:24] <MirahezeRelay>	 <cosmicalpha> lol whoops sorry for wrong ping
[10:06:37] <MirahezeRelay>	 <cosmicalpha> clicked the wrong one lol
[10:07:42] <MirahezeRelay>	 <cosmicalpha> now it's 4AM so I should sleep
[10:09:59] <MirahezeRelay>	 <aeywoo, replying to cosmicalpha> Hmmmm.
[10:10:50] <MirahezeRelay>	 <cosmicalpha> sorry about that accidental ping but left it so it didn't turn into a ghost ping
[10:10:50] <MirahezeRelay>	 <asko1> begins with A, close enough
[10:27:03] <miraheze-github>	 miraheze/DiscordNotifications - Universal-Omega the build passed.
[11:06:53] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.07, 18.09, 11.99
[11:08:49] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 20.09, 13.47
[11:14:40] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.78, 17.87, 15.17
[11:32:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749637940000&orgId=1&to=1749641573345
[11:37:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749638240000&orgId=1&to=1749641840000
[15:37:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749652640000&orgId=1&to=1749656273344
[15:42:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749652820000&orgId=1&to=1749656420000
[16:03:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749654200000&orgId=1&to=1749657833347
[16:08:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749654500000&orgId=1&to=1749658100000
[16:13:08] <MacFanBot>	 [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/d4b47dc32b27e676a0e9d4ff55d188308565e81b
[16:13:08] <MacFanBot>	 02ssl/03main 07MacFan4000 03d4b47dc add www.englishbaronage.qzz.io
[17:18:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749658700000&orgId=1&to=1749662333345
[17:23:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749658940000&orgId=1&to=1749662540000
[17:28:51] <MacFanBot>	 [02dns] 07Universal-Omega created 03wtnet-services (+1 new commit) 13https://github.com/miraheze/dns/commit/d391efff225e
[17:28:51] <MacFanBot>	 02dns/03wtnet-services 07CosmicAlpha 03d391eff Add services to wtnet
[17:28:57] <MacFanBot>	 [02dns] 07Universal-Omega opened pull request #691: Add services to wtnet (03main...03wtnet-services) 13https://github.com/miraheze/dns/pull/691
[17:29:52] <miraheze-github>	 miraheze/dns - Universal-Omega the build passed.
[18:04:43] <MacFanBot>	 [02dns] 07Universal-Omega merged pull request #691: Add services to wtnet (03main...03wtnet-services) 13https://github.com/miraheze/dns/pull/691
[18:04:44] <MacFanBot>	 [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/23b5b8b18620e09b3145ce17f0e50613c66dd27d
[18:04:44] <MacFanBot>	 02dns/03main 07CosmicAlpha 0323b5b8b Add services to wtnet
[18:04:45] <MacFanBot>	 [02dns] 07Universal-Omega 04deleted 03wtnet-services at 03d391eff 13https://api.github.com/repos/miraheze/dns/commit/d391eff
[18:06:31] <miraheze-github>	 miraheze/dns - Universal-Omega the build passed.
[18:14:48] <MacFanBot>	 [02puppet] 07Universal-Omega created 03jobrunner-internal (+1 new commit) 13https://github.com/miraheze/puppet/commit/f21a3fea6acc
[18:14:48] <MacFanBot>	 02puppet/03jobrunner-internal 07CosmicAlpha 03f21a3fe Update jobrunner to internal network
[18:14:53] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4386: Update jobrunner to internal network (03main...03jobrunner-internal) 13https://github.com/miraheze/puppet/pull/4386
[18:14:59] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4386: <details open="true"> […] 13https://github.com/miraheze/puppet/pull/4386#issuecomment-2963754412
[18:20:06] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03jobrunner-internal 13https://github.com/miraheze/puppet/commit/920e5eaf94d0f8b1aeebe6ef329d8ed318a09be3
[18:20:06] <MacFanBot>	 02puppet/03jobrunner-internal 07CosmicAlpha 03920e5ea Update
[18:23:11] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03jobrunner-internal 13https://github.com/miraheze/puppet/commit/1eeb2ec7b421e7d0f1c76759493df6720ec12f49
[18:23:11] <MacFanBot>	 02puppet/03jobrunner-internal 07CosmicAlpha 031eeb2ec Update
[18:25:03] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03jobrunner-internal 13https://github.com/miraheze/puppet/commit/22fba5d2eee9504c6e606d92dad3a6dafb4aba83
[18:25:03] <MacFanBot>	 02puppet/03jobrunner-internal 07CosmicAlpha 0322fba5d Update
[18:31:30] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03jobrunner-internal 13https://github.com/miraheze/puppet/commit/da8b37873637db2eb11db3cbc20f55644bded6ea
[18:31:30] <MacFanBot>	 02puppet/03jobrunner-internal 07CosmicAlpha 03da8b378 Remove
[18:32:24] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 39.75, 20.68, 9.71
[18:33:17] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw173 mw202
[18:33:19] <icinga-miraheze>	 PROBLEM - cloud20 IPMI Sensors on cloud20 is CRITICAL: IPMI Status: Critical [168 system event log (SEL) entries present]
[18:34:43] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw202
[18:34:58] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[18:35:20] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[18:35:56] <icinga-miraheze>	 PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:36:23] <icinga-miraheze>	 RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.514 second response time
[18:40:18] <icinga-miraheze>	 PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:40:50] <icinga-miraheze>	 RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.602 second response time
[18:45:48] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw151 mw152
[18:46:49] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[18:56:15] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [6 system event log (SEL) entries present]
[19:00:13] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03jobrunner-internal 13https://github.com/miraheze/puppet/commit/52192885cbb283dc43e5db6b9c222d0d3849b648
[19:00:13] <MacFanBot>	 02puppet/03jobrunner-internal 07CosmicAlpha 035219288 Remove ssl
[19:05:11] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.47, 2.94, 11.53
[19:09:05] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.30, 2.15, 9.28
[19:12:45] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4386: Update jobrunner to internal network (03main...03jobrunner-internal) 13https://github.com/miraheze/puppet/pull/4386
[19:12:45] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/23cdcacf293c80b012d0be6fd1716418d916c7f4
[19:12:47] <MacFanBot>	 02puppet/03main 07CosmicAlpha 0323cdcac Update jobrunner to internal network
[19:12:49] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03jobrunner-internal at 035219288 13https://api.github.com/repos/miraheze/puppet/commit/5219288
[19:39:36] <MacFanBot>	 [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/81f17a1b302bab16f45f6a8a34f3505852b8d99e
[19:39:36] <MacFanBot>	 02dns/03main 07CosmicAlpha 0381f17a1 wikitide.net: remove jobrunner service aliases…
[19:40:28] <miraheze-github>	 miraheze/dns - Universal-Omega the build passed.
[19:50:36] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5328aca8039ddfbe1b2e10b84c3682a240d4faa2
[19:50:36] <MacFanBot>	 02puppet/03main 07CosmicAlpha 035328aca Remove default_value for puppetserver_hostname
[20:18:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749669500000&orgId=1&to=1749673133347
[20:28:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749670040000&orgId=1&to=1749673640000
[20:43:11] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Package[langcodes],Exec[create python venv],Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members]
[20:44:42] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.98, 16.44, 9.70
[20:48:04] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 25.08, 17.83, 9.93
[20:50:04] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 17.59, 17.85, 10.93
[20:51:13] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:58:08] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 22.39, 15.11
[21:04:04] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 21.43, 16.69
[21:06:04] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.17, 20.03, 16.73
[21:10:04] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 29.28, 24.02, 19.03
[21:14:42] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.31, 23.96, 22.53
[21:17:41] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 16.83, 21.25, 20.98
[21:18:42] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 24.52, 23.05
[21:19:39] <icinga-miraheze>	 RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 14.37, 18.87, 20.15
[21:24:31] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 18.40, 22.75, 22.81
[21:25:55] <MirahezeRelay>	 <blankeclair> u good?
[21:26:50] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/24f36f94522a4278eb68e8a91be519af4f570c66
[21:26:50] <MacFanBot>	 02puppet/03main 07CosmicAlpha 0324f36f9 Add fsslc.wtnet version of cloud
[21:28:50] <MirahezeRelay>	 <cosmicalpha, replying to blankeclair> lot of jobs (60k) + me running i18n at the same time probably didn't play well
[21:31:45] <MirahezeRelay>	 <blankeclair> i'm gonna take it as a no
[21:32:42] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.16, 23.11, 23.71
[21:32:45] <MirahezeRelay>	 <blankeclair> [1/2] oooooof
[21:32:46] <MirahezeRelay>	 <blankeclair> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1382472651242475751/Screenshot_20250612-073224.png?ex=684b477d&is=6849f5fd&hm=575c45808e14beb60857fb6b195b3833e163564fad120d55d0a7f995d3a2146f&
[21:34:17] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 10.49, 16.21, 19.89
[21:38:42] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.57, 13.52, 19.30
[23:31:08] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27229MiB (5% inode=99%);
[23:56:49] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27200MiB (5% inode=99%);