[00:05:35] <MirahezeLSBot>	 !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php namespaceDupes --wiki=wikigeniuswiki --add-suffix=_(old) (END - exit=512)
[00:05:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:05:58] <MirahezeLSBot>	 !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php namespaceDupes --wiki=wikigeniuswiki --add-suffix='_(old)' (END - exit=0)
[00:06:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:06:17] <MirahezeLSBot>	 !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php namespaceDupes --wiki=wikigeniuswiki --add-suffix='_(old)' --fix (END - exit=0)
[00:06:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:07:00] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57149MiB (12% inode=99%);
[00:07:04] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56936MiB (12% inode=99%);
[00:08:23] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57612MiB (12% inode=99%);
[00:36:29] <MirahezeLSBot>	 !log [void@db181] [wikigeniuswiki]> UPDATE page SET page_content_model="wikitext" WHERE page_content_model LIKE "wikibase%"; UPDATE content SET content_model=1 WHERE content_model=6; UPDATE content SET content_model=1 WHERE content_model=8; -- Deleted content models
[00:36:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:53:44] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=dappervolkwiki dump.xml --no-updates (START)
[01:53:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:02:35] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3
[03:20:48] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=dappervolkwiki dump.xml --no-updates (END - exit=0)
[03:20:49] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=dappervolkwiki (START)
[03:20:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:20:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:04:49] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=dappervolkwiki (END - exit=0)
[04:04:51] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=dappervolkwiki --update (END - exit=0)
[04:04:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:04:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:48:53] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[05:27:46] <icinga-miraheze>	 PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 43947MiB (9% inode=99%);
[05:29:46] <icinga-miraheze>	 RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 56151MiB (12% inode=99%);
[06:03:59] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.17, 12.92, 6.47
[06:07:54] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.73, 11.73, 7.42
[06:09:52] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 17.74, 12.19, 7.99
[06:11:46] <icinga-miraheze>	 PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 44066MiB (9% inode=99%);
[06:13:49] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.45, 11.19, 8.56
[06:15:46] <icinga-miraheze>	 RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 183546MiB (40% inode=99%);
[06:15:47] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.84, 7.97, 7.70
[06:21:40] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.34, 12.49, 9.33
[06:23:38] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.91, 10.67, 8.97
[06:25:36] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.52, 7.52, 8.02
[06:29:30] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 18.76, 11.87, 9.47
[06:31:28] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.45, 8.34, 8.46
[07:50:56] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_pywikibot-config]
[08:03:23] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3
[08:06:26] <MacFanBot>	 [02puppet] 07RhinosF1 created 03RhinosF1-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/c2498a9de952
[08:06:26] <MacFanBot>	 02puppet/03RhinosF1-patch-3 07RhinosF1 03c2498a9 [users] rhinos: pause my ssh keys…
[08:06:31] <MacFanBot>	 [02puppet] 07RhinosF1 opened pull request #4453: [users] rhinos: pause my ssh keys (03main...03RhinosF1-patch-3) 13https://github.com/miraheze/puppet/pull/4453
[08:06:37] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4453: <details> […] 13https://github.com/miraheze/puppet/pull/4453#issuecomment-3095661224
[08:09:32] <RhinosF1>	 !log destory W342 (spam)
[08:09:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:11:20] <RhinosF1>	 !log destorying a lot of spam on phorge
[08:11:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:12:58] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[09:24:44] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49840MiB (10% inode=99%);
[09:37:46] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49861MiB (10% inode=99%);
[10:04:07] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49912MiB (10% inode=99%);
[10:09:20] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 8 backends are down. mw151 mw161 mw171 mw181 mw183 mw191 mw192 mw203
[10:10:02] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 6 backends are down. mw171 mw182 mw163 mw191 mw202 mw203
[10:10:21] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw202 mw203
[10:11:20] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[10:12:21] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[10:14:02] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[12:11:26] <MacFanBot>	 [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/0ecd567b4b9f350d8a4e079c352437707fbf5434
[12:11:26] <MacFanBot>	 02DiscordNotifications/03main 07translatewiki.net 030ecd567 Localisation updates from https://translatewiki.net.
[12:11:27] <MacFanBot>	 [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/4676bc1e79b7d98910ab45791fc38037bd58ba49
[12:11:27] <MacFanBot>	 02ImportDump/03main 07translatewiki.net 034676bc1 Localisation updates from https://translatewiki.net.
[12:11:29] <MacFanBot>	 [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/497d3426802b7c4e1fba00dc702281e0277c0da7
[12:11:31] <MacFanBot>	 [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/985d2252b036b4db99ff35699d248b1e7c828af9
[12:11:33] <MacFanBot>	 [02DataDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/2570b340e22c6a8a46f27fd94b5d26c4a562a9d6
[12:11:35] <MacFanBot>	 02landing/03main 07translatewiki.net 03497d342 Localisation updates from https://translatewiki.net.
[12:11:37] <MacFanBot>	 [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/f3cf1ff193fa11d3ddc73defcf535b35ce4a4883
[12:11:39] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/59956d83a79014ec8e421fdb90545d55fabbc184
[12:11:40] <MacFanBot>	 02MatomoAnalytics/03main 07translatewiki.net 03985d225 Localisation updates from https://translatewiki.net.
[12:11:42] <MacFanBot>	 02DataDump/03main 07translatewiki.net 032570b34 Localisation updates from https://translatewiki.net.
[12:11:43] <MacFanBot>	 02ManageWiki/03main 07translatewiki.net 03f3cf1ff Localisation updates from https://translatewiki.net.
[12:11:45] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 0359956d8 Localisation updates from https://translatewiki.net.
[12:12:15] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:15:41] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:15:49] <miraheze-github_>	 miraheze/ManageWiki - translatewiki the build has errored.
[12:16:01] <miraheze-github>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[12:16:24] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:17:01] <miraheze-github>	 miraheze/ImportDump - translatewiki the build has errored.
[12:35:04] <miraheze-github>	 miraheze/DiscordNotifications - translatewiki the build passed.
[13:02:49] <MirahezeLSBot>	 !log [rhinos@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_pack': 'mleb'} to all
[13:02:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:04:42] <MirahezeLSBot>	 !log [rhinos@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_pack': 'mleb'} to all - SUCCESS in 113s
[13:04:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:04:55] <MirahezeLSBot>	 !log [rhinos@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43'} to all
[13:04:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:05:28] <MacFanBot>	 [02mw-config] 07RhinosF1 merged 07lihaohong6's pull request #6028: Add missing sql table for Extension:Translate (07miraheze:03main...07lihaohong6:03master) 13https://github.com/miraheze/mw-config/pull/6028
[13:05:28] <MacFanBot>	 [02mw-config] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/3238ebb37ebbea9a701567b955d127dc601fd1a7
[13:05:29] <MacFanBot>	 02mw-config/03main 07Peter Li 033238ebb Add missing sql table for Extension:Translate (#6028)…
[13:06:19] <MirahezeLSBot>	 !log [rhinos@test151] starting deploy of {'config': True} to test151
[13:06:20] <MirahezeLSBot>	 !log [rhinos@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[13:06:20] <miraheze-github>	 miraheze/mw-config - RhinosF1 the build passed.
[13:06:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:06:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:13:10] <MirahezeLSBot>	 !log [rhinos@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43'} to all - SUCCESS in 494s
[13:13:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:13:20] <MirahezeLSBot>	 !log [rhinos@mwtask181] starting deploy of {'config': True} to all
[13:13:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:13:49] <MirahezeLSBot>	 !log [rhinos@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 29s
[13:13:50] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=danmachiwiki dump.xml --no-updates (START)
[13:13:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:13:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:44:23] <MacFanBot>	 [02puppet] 07RhinosF1 merged pull request #4453: [users] rhinos: pause my ssh keys (03main...03RhinosF1-patch-3) 13https://github.com/miraheze/puppet/pull/4453
[13:44:25] <MacFanBot>	 [02puppet] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9a636838b9dff4a84f796fa8653aadb4f283b9e4
[13:44:27] <MacFanBot>	 [02puppet] 07RhinosF1 04deleted 03RhinosF1-patch-3 at 03c2498a9 13https://api.github.com/repos/miraheze/puppet/commit/c2498a9
[13:44:28] <MacFanBot>	 02puppet/03main 07RhinosF1 039a63683 [users] rhinos: pause my ssh keys (#4453)…
[14:15:03] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=danmachiwiki dump.xml --no-updates (END - exit=0)
[14:15:04] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=danmachiwiki (START)
[14:15:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:15:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:35:17] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=danmachiwiki (END - exit=0)
[14:35:18] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/initSiteStats.php --wiki=danmachiwiki --update (END - exit=0)
[14:35:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:35:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:30:39] <miraheze-github>	 miraheze/DiscordNotifications - pskyechology the build has errored.
[17:30:02] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 74.41, 42.15, 18.65
[17:39:53] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.22, 8.16, 11.84
[17:43:49] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.80, 4.12, 9.35
[19:26:26] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[19:26:27] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s
[19:26:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:26:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:31:32] <MirahezeRelay>	 <posix_memalign> [1/2] @pskyechology I think translate works now
[19:31:32] <MirahezeRelay>	 <posix_memalign> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1396937658076237824/image.png?ex=687fe713&is=687e9593&hm=ae72e34ba8e2f7433a429f856ae7696da6204def5ad172b854c2e5deaed8e0ab&
[19:32:17] <MirahezeRelay>	 <pskyechology, replying to posix_memalign> yeah it seems like rhinos deployed without pulling the new config
[19:32:23] <MirahezeRelay>	 <pskyechology> small oopsie :3
[20:17:20] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1753125410000&orgId=1&to=1753129040013
[20:27:20] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?from=1753125410000&orgId=1&to=1753129610000
[20:27:30] <MirahezeRelay>	 <rhinosf1, replying to pskyechology> Someone is a bit stupid aren't they
[20:33:18] <MirahezeRelay>	 <pskyechology, replying to rhinosf1> nah, remembering everything is bothersome anyway
[23:41:55] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DataMaps'} to test151
[23:41:56] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'DataMaps'} to test151 - SUCCESS in 1s
[23:41:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:42:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:57:24] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'SimpleBatchUpload'} to test151
[23:57:25] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'SimpleBatchUpload'} to test151 - SUCCESS in 1s
[23:57:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:57:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log