[00:00:59] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/679eef0b9b17ce0604ddb78c3f83cb28464bb41c
[00:00:59] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03679eef0 Bot: Auto-update Tech namespace pages 2026-07-08 00:00:55
[00:52:26] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 44541MiB (9.8% inode=100%);
[00:54:26] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 87089MiB (19.2% inode=100%);
[00:56:25] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44509MiB (9.8% inode=100%);
[00:58:25] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 87161MiB (19.2% inode=100%);
[01:02:37] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 21.56, 20.32, 11.44
[01:06:37] <icinga-miraheze>	 RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 2.03, 9.96, 9.18
[01:59:59] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 64.37, 29.55, 14.77
[02:01:34] <MirahezeRelay>	 <posix_memalign> Whole farm is down. Depooling.
[02:01:50] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 5 backends are down. mw161 mw163 mw191 mw193 mw201
[02:02:10] <MirahezeLSBot>	 !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all
[02:02:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:02:31] <MirahezeLSBot>	 !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 21s
[02:02:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:03:28] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[02:03:50] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[02:07:35] <MirahezeRelay>	 <posix_memalign> Load is going down. Repooling.
[02:07:53] <MirahezeLSBot>	 !log [petramagna@mwtask181] starting deploy of {'config': True, 'force': True} to all
[02:07:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:08:11] <MirahezeLSBot>	 !log [petramagna@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 18s
[02:08:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:17:06] <MirahezeRelay>	 <posix_memalign> Load is normal about repooling. Would still be nice to have an infra look into the proclist though. osdev.wiki was under heavy scraping but it has been going on for quite a while.
[02:23:59] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 2.96, 3.88, 11.38
[02:27:59] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.01, 3.82, 9.59
[02:31:28] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[02:42:32] <icinga-miraheze>	 PROBLEM - cloud21 IPMI Sensors on cloud21 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present]
[02:50:43] <MirahezeRelay>	 <orduin, replying to posix_memalign> anything for japolandballwiki?
[03:11:25] <MirahezeRelay>	 <orduin, replying to icinga-miraheze> inlet temperature, should be fine
[03:22:48] <icinga-miraheze>	 PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present]
[03:25:17] <MirahezeRelay>	 <orduin> hmm, that's two servers now with high inlet temperatures
[03:25:45] <MirahezeRelay>	 <orduin> if this keeps up I'm gonna have to dig up the ticket submission form for our current hosting
[03:33:07] <MirahezeRelay>	 <posix_memalign, replying to orduin> Seems to be due to inefficiently coded js with with lots of api.php requests. Most traffic comes from Japan so it's likely not bots. Could also be caused by something that disables parser cache as well.
[03:33:38] <icinga-miraheze>	 PROBLEM - cloud19 IPMI Sensors on cloud19 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present]
[03:34:05] <MirahezeRelay>	 <orduin> Might be more of a db optimization thing, there was some searches that were running for over two minutes
[03:35:06] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.37, 14.17, 6.78
[03:35:21] <MirahezeRelay>	 <orduin> ok, that's three servers hitting high inlet temps
[03:37:06] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.09, 9.95, 6.13
[03:38:32] <icinga-miraheze>	 PROBLEM - cloud21 IPMI Sensors on cloud21 is CRITICAL: IPMI Status: Critical [2 system event log (SEL) entries present]
[05:42:34] <icinga-miraheze>	 PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui]
[06:12:30] <icinga-miraheze>	 RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:33:22] <MacFanBot>	 [02mw-config] 07SomeMWDev created 03T15625 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/120120648372
[12:33:22] <MacFanBot>	 02mw-config/03T15625 07SomeRandomDeveloper 031201206 T15625: Globally disable WikiPoints
[12:33:32] <MacFanBot>	 [02mw-config] 07SomeMWDev opened pull request #6456: T15625: Globally disable WikiPoints (03main...03T15625) 13https://github.com/miraheze/mw-config/pull/6456
[12:34:47] <MacFanBot>	 [02mw-config] 07github-actions[bot] pushed 1 new commit to 03T15625 13https://github.com/miraheze/mw-config/commit/9ef2457100f7c2f8d27594dcde8e9d721f0a1e05
[12:34:48] <MacFanBot>	 02mw-config/03T15625 07github-actions 039ef2457 CI: lint code to MediaWiki standards…
[12:35:19] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[12:36:09] <miraheze-github>	 miraheze/mw-config - github-actions[bot] the build passed.
[12:36:12] <MacFanBot>	 [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/ff5e83bc6a2fe8237a1408fe5099b4c9d1f39e90
[12:36:12] <MacFanBot>	 02mw-config/03main 07SomeRandomDeveloper 03ff5e83b T15625: Globally disable WikiPoints (#6456)…
[12:36:12] <MacFanBot>	 [02mw-config] 07SomeMWDev merged pull request #6456: T15625: Globally disable WikiPoints (03main...03T15625) 13https://github.com/miraheze/mw-config/pull/6456
[12:36:13] <MacFanBot>	 [02mw-config] 07SomeMWDev 04deleted 03T15625 at 039ef2457 13https://api.github.com/repos/miraheze/mw-config/commit/9ef2457
[12:36:18] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[12:36:19] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s
[12:36:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:36:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:36:24] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[12:36:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:36:50] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s
[12:36:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:13] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php ManageWiki:ToggleExtension --wiki=loginwikibeta --name=wikipoints --disable --all-wikis --execute (END - exit=0)
[12:37:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:23] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[12:37:45] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php ManageWiki:ToggleExtension --wiki=loginwiki --name=wikipoints --disable --all-wikis --execute (END - exit=0)
[12:37:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:39:46] <MacFanBot>	 [02mw-config] 07SomeMWDev created 03T15625-remove (+1 new commit) 13https://github.com/miraheze/mw-config/commit/8e1503afe1fd
[12:39:46] <MacFanBot>	 02mw-config/03T15625-remove 07SomeRandomDeveloper 038e1503a T15625: Undeploy WikiPoints
[12:39:56] <MacFanBot>	 [02mw-config] 07SomeMWDev opened pull request #6457: T15625: Undeploy WikiPoints (03main...03T15625-remove) 13https://github.com/miraheze/mw-config/pull/6457
[12:40:47] <MacFanBot>	 [02mw-config] 07SomeMWDev merged pull request #6457: T15625: Undeploy WikiPoints (03main...03T15625-remove) 13https://github.com/miraheze/mw-config/pull/6457
[12:40:49] <MacFanBot>	 [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/726d62b53c90b18fc0cd7808ba0e892f18a65441
[12:40:50] <MacFanBot>	 02mw-config/03main 07SomeRandomDeveloper 03726d62b T15625: Undeploy WikiPoints (#6457)
[12:40:52] <MacFanBot>	 [02mw-config] 07SomeMWDev 04deleted 03T15625-remove at 038e1503a 13https://api.github.com/repos/miraheze/mw-config/commit/8e1503a
[12:41:07] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[12:41:59] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[12:42:26] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[12:42:27] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s
[12:42:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:42:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:42:49] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[12:42:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:43:14] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s
[12:43:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:43:46] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] sudo -u www-data rm -rf /srv/mediawiki-staging/1.45/extensions/WikiPoints/
[12:43:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:43:50] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] sudo -u www-data rm -rf /srv/mediawiki-staging/1.46/extensions/WikiPoints/
[12:43:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:44:14] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] sudo -u www-data rm -rf /srv/mediawiki-staging/1.45/extensions/WikiPoints/
[12:44:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:45:02] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] sudo -u www-data rm -r /srv/mediawiki/1.45/extensions/WikiPoints/ /srv/mediawiki/1.46/extensions/WikiPoints/
[12:45:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:00] <MirahezeLSBot>	 !log [somerandomdeveloper@puppet181] sudo salt-ssh -E mw.* cmd.run sudo -u www-data rm -rf /srv/mediawiki/1.45/extensions/WikiPoints/
[12:46:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:29] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'world': True, 'l10n': True, 'extension_list': True, 'versions': ['1.45', '1.46']} to test151
[12:46:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:38] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'world': True, 'l10n': True, 'extension_list': True, 'versions': '1.45'} to all
[12:46:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:41] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[12:46:43] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[12:46:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:46] <MirahezeRelay>	 <abaddriverlol> whoops
[12:46:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:46:49] <MirahezeRelay>	 <abaddriverlol> wanted to deploy without --world
[12:46:57] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'l10n': True, 'extension_list': True, 'versions': ['1.45', '1.46']} to test151
[12:47:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:47:04] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'l10n': True, 'extension_list': True, 'versions': '1.45'} to all
[12:47:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:53:18] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'l10n': True, 'extension_list': True, 'versions': ['1.45', '1.46']} to test151 - SUCCESS in 380s
[12:53:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:56:16] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'l10n': True, 'extension_list': True, 'versions': '1.45'} to all - SUCCESS in 551s
[12:56:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:12:29] <MacFanBot>	 [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-13.2.4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0590ca5e1710
[14:12:29] <MacFanBot>	 02mw-config/03dependabot/composer/phpunit/phpunit-13.2.4 07dependabot[bot] 030590ca5 Update phpunit/phpunit requirement from 13.2.2 to 13.2.4…
[14:12:30] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6458 (Update phpunit/phpunit requirement from 13.2.2 to 13.2.4) 13https://github.com/miraheze/mw-config/pull/6458
[14:12:30] <MacFanBot>	 [02mw-config] 07dependabot[bot] opened pull request #6458: Update phpunit/phpunit requirement from 13.2.2 to 13.2.4 (03main...03dependabot/composer/phpunit/phpunit-13.2.4) 13https://github.com/miraheze/mw-config/pull/6458
[14:12:32] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6458 (Update phpunit/phpunit requirement from 13.2.2 to 13.2.4) 13https://github.com/miraheze/mw-config/pull/6458
[14:12:35] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6458 (Update phpunit/phpunit requirement from 13.2.2 to 13.2.4) 13https://github.com/miraheze/mw-config/pull/6458
[14:12:36] <MacFanBot>	 [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6458 (Update phpunit/phpunit requirement from 13.2.2 to 13.2.4) 13https://github.com/miraheze/mw-config/pull/6458
[14:12:38] <MacFanBot>	 [02mw-config] 07dependabot[bot] commented on pull request #6454: Superseded by #6458. 13https://github.com/miraheze/mw-config/pull/6454#issuecomment-4915704236
[14:12:40] <MacFanBot>	 [02mw-config] 07dependabot[bot] closed pull request #6454: Update phpunit/phpunit requirement from 13.2.2 to 13.2.3 (03main...03dependabot/composer/phpunit/phpunit-13.2.3) 13https://github.com/miraheze/mw-config/pull/6454
[14:12:43] <MacFanBot>	 [02mw-config] 07dependabot[bot] 04deleted 03dependabot/composer/phpunit/phpunit-13.2.3 at 036d8395b 13https://api.github.com/repos/miraheze/mw-config/commit/6d8395b
[14:13:37] <miraheze-github>	 miraheze/mw-config - dependabot[bot] the build passed.
[18:30:32] <icinga-miraheze>	 RECOVERY - cloud21 IPMI Sensors on cloud21 is OK: IPMI Status: OK
[18:30:42] <MirahezeLSBot>	 !log [void@cloud21] clear ipmi sel
[18:31:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:32:03] <MirahezeLSBot>	 !log [void@cloud20] clear ipmi sel
[18:32:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:32:46] <MirahezeLSBot>	 !log [void@cloud19] clear ipmi sel
[18:32:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:33:09] <icinga-miraheze>	 RECOVERY - cloud20 IPMI Sensors on cloud20 is OK: IPMI Status: OK
[18:33:38] <icinga-miraheze>	 RECOVERY - cloud19 IPMI Sensors on cloud19 is OK: IPMI Status: OK
[20:39:16] <icinga-miraheze>	 PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089
[20:39:35] <icinga-miraheze>	 PROBLEM - ns2 SSH on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:40:30] <icinga-miraheze>	 PROBLEM - Host ns2 is DOWN: PING CRITICAL - Packet loss = 100%
[20:45:03] <icinga-miraheze>	 RECOVERY - Host ns2 is UP: PING OK - Packet loss = 0%, RTA = 125.54 ms
[20:45:18] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.280 seconds response time. ns2.wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66
[20:45:36] <icinga-miraheze>	 RECOVERY - ns2 SSH on ns2 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0)
[21:51:47] <MirahezeLSBot>	 !log [universalomega@phorge171] Destroyed a user
[21:51:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:57:41] <MirahezeRelay>	 <abaddriverlol> my favourite type of log entry
[22:05:31] <MirahezeRelay>	 <thewwrnerdguy> watch as CA DESTROYS a user with FACTS and LOGIC
[22:36:06] <MirahezeRelay>	 <randomizer.jpeg, replying to MirahezeLSBot> :stare: