[00:01:27] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/f0debad2593c0cc5ae56f8a8de131e7f10f9f4db
[00:01:27] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03f0debad Bot: Auto-update Tech namespace pages 2025-09-30 00:01:25
[00:10:21] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 54635MiB (12% inode=99%);
[00:10:32] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54122MiB (11% inode=99%);
[00:10:43] <icinga-miraheze>	 RECOVERY - llm191 ollama process on llm191 is OK: PROCS OK: 1 process with args '/usr/bin/ollama'
[00:10:57] <icinga-miraheze>	 RECOVERY - llm191 ollama port 11434 on llm191 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11434
[00:12:22] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 53629MiB (11% inode=99%);
[00:25:26] <icinga-miraheze>	 PROBLEM - swift-lb.wikitide.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 14 day(s) (Wed 15 Oct 2025 12:16:28 AM GMT +0000).
[00:32:43] <icinga-miraheze>	 PROBLEM - llm191 ollama process on llm191 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/bin/ollama'
[00:40:43] <icinga-miraheze>	 RECOVERY - llm191 ollama process on llm191 is OK: PROCS OK: 1 process with args '/usr/bin/ollama'
[01:50:26] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 70.60, 24.39, 10.07
[02:07:36] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 1.15, 5.77, 11.97
[02:11:36] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.77, 3.00, 9.41
[02:38:59] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.70, 11.11, 7.72
[02:40:53] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.86, 7.73, 6.88
[03:15:53] <icinga-miraheze>	 [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759198520000&orgId=1&to=1759202153353
[03:18:22] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49805MiB (10% inode=99%);
[03:25:53] <icinga-miraheze>	 [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1759198520000&orgId=1&to=1759202720000
[03:38:32] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49866MiB (10% inode=99%);
[04:02:21] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49907MiB (10% inode=99%);
[05:16:10] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[08:00:10] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1
[10:50:13] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_skins': 'Citizen'} to test151
[10:50:15] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_skins': 'Citizen'} to test151 - SUCCESS in 2s
[10:50:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:50:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:05:33] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/4dc6b7c3d9f07440a7258e66281c0de4a0e67a03
[14:05:33] <MacFanBot>	 02ssl/03main 07WikiTideBot 034dc6b7c Bot: Auto-update domain lists
[14:07:36] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.18, 11.54, 7.19
[14:09:36] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.61, 7.87, 6.37
[14:51:06] <MacFanBot>	 [02MirahezeMagic] 07AgentIsai pushed 1 new commit to 03requeueforeval 13https://github.com/miraheze/MirahezeMagic/commit/89cfc6701bea6e3256193b41358476faa8c3260c
[14:51:06] <MacFanBot>	 02MirahezeMagic/03requeueforeval 07Agent Isai 0389cfc67 Fix
[14:51:19] <MacFanBot>	 [02MirahezeMagic] 07AgentIsai commented on pull request #607: @coderabbitai review 13https://github.com/miraheze/MirahezeMagic/pull/607#issuecomment-3352583864
[14:51:27] <MacFanBot>	 [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #607: <details> […] 13https://github.com/miraheze/MirahezeMagic/pull/607#issuecomment-3352584401
[14:54:39] <MacFanBot>	 [02MirahezeMagic] 07coderabbitai[bot] left a review on pull request #607: **Actionable comments posted: 1** […] 13https://github.com/miraheze/MirahezeMagic/pull/607#pullrequestreview-3285426172
[14:54:39] <MacFanBot>	 [02MirahezeMagic] 07coderabbitai[bot] left a file comment in pull request #607 0389cfc67: _⚠️ Potential issue_ | _🟠 Major_ […] 13https://github.com/miraheze/MirahezeMagic/pull/607#discussion_r2391908550
[15:03:12] <miraheze-github>	 miraheze/MirahezeMagic - AgentIsai the build passed.
[15:05:29] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/709e5e2a8009899ef35aa30da08c98ea6e45959e
[15:05:29] <MacFanBot>	 02ssl/03main 07WikiTideBot 03709e5e2 Bot: Auto-update domain lists
[16:35:41] <MacFanBot>	 [02dns] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/4448fc6a1039067c8c2852e2fdc300e8f63d3d69
[16:35:41] <MacFanBot>	 02dns/03main 07MacFan4000 034448fc6 add ACME challenges for wikitide.net
[16:37:13] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/81c561ea950b3244bac589f55ba2757cbb8eefa3
[16:37:13] <MacFanBot>	 02ssl/03main 07WikiTideBot 0381c561e Bot: Update SSL cert for wikitide.net
[16:38:23] <miraheze-github>	 miraheze/dns - MacFan4000 the build passed.
[16:42:07] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_skins': 'Citizen'} to all
[16:42:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:42:31] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_skins': 'Citizen'} to all - SUCCESS in 24s
[16:42:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:49:59] <MacFanBot>	 [02mw-config] 07SomeMWDev created 03add-wgCitizenHeaderPosition (+1 new commit) 13https://github.com/miraheze/mw-config/commit/c826ac41fb66
[16:49:59] <MacFanBot>	 02mw-config/03add-wgCitizenHeaderPosition 07SomeRandomDeveloper 03c826ac4 Add wgCitizenHeaderPosition
[16:50:33] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151
[16:50:34] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[16:50:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:50:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:57:31] <icinga-miraheze>	 RECOVERY - graylog161 graylog tls port 12210 ssl cert check on graylog161 is OK: OK - Certificate 'wikitide.net' will expire on Mon Dec 29 15:38:37 2025 +0000.TCP OK - 0.052 second response time on localhost port 12210
[16:58:58] <MacFanBot>	 [02mw-config] 07SomeMWDev 04force-pushed 03add-wgCitizenHeaderPosition from 03c826ac4 to 0399abeae 13https://api.github.com/repos/miraheze/mw-config/commits/add-wgCitizenHeaderPosition
[16:58:58] <MacFanBot>	 02mw-config/03add-wgCitizenHeaderPosition 07SomeRandomDeveloper 0399abeae Add wgCitizenHeaderPosition
[16:59:16] <MacFanBot>	 [02mw-config] 07SomeMWDev opened pull request #6133: Add wgCitizenHeaderPosition (03main...03add-wgCitizenHeaderPosition) 13https://github.com/miraheze/mw-config/pull/6133
[16:59:36] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.17, 10.38, 5.65
[17:00:10] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[17:00:47] <MacFanBot>	 [02mw-config] 07SomeMWDev merged pull request #6133: Add wgCitizenHeaderPosition (03main...03add-wgCitizenHeaderPosition) 13https://github.com/miraheze/mw-config/pull/6133
[17:00:50] <MacFanBot>	 [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/eed4fee1737570af6796d12296426d6fac4c0ff5
[17:00:52] <MacFanBot>	 [02mw-config] 07SomeMWDev 04deleted 03add-wgCitizenHeaderPosition at 0399abeae 13https://api.github.com/repos/miraheze/mw-config/commit/99abeae
[17:00:53] <MacFanBot>	 02mw-config/03main 07SomeRandomDeveloper 03eed4fee Add wgCitizenHeaderPosition (#6133)
[17:00:57] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[17:01:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:01:15] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151
[17:01:16] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[17:01:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:01:20] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s
[17:01:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:01:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:01:36] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.28, 7.19, 5.05
[17:01:44] <miraheze-github>	 miraheze/mw-config - SomeMWDev the build passed.
[17:16:23] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[phd]
[17:26:23] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:31:44] <MacFanBot>	 [02puppet] 07AgentIsai pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3f7ce8785ed7d7f03e7fb531f2ea10356eec106a
[17:31:44] <MacFanBot>	 02puppet/03main 07Agent Isai 033f7ce87 Explicitly require slapd to restart on ssl update
[17:41:36] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 11.81, 13.70, 7.55
[17:43:36] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.45, 9.50, 6.75
[17:53:26] <icinga-miraheze>	 RECOVERY - swift-lb.wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Mon 29 Dec 2025 03:38:37 PM GMT +0000.
[18:05:36] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.58, 13.81, 7.66
[18:09:36] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.38, 10.26, 7.90
[18:11:36] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.89, 7.09, 7.02
[18:23:27] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/cafc4db6ca9405b4b414f8137d3cf84e5a681820
[18:23:27] <MacFanBot>	 02ssl/03main 07Skye 03cafc4db Redirect chalaco.miraheze.org to wiki.chalaco.org
[18:24:09] <miraheze-github>	 miraheze/ssl - pskyechology the build has errored.
[18:27:13] <MacFanBot>	 [02ssl] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/cdd49840ba0a7d7fb310ca18a91abb9a1d2a4183
[18:27:14] <MacFanBot>	 02ssl/03main 07Skye 03cdd4984 Fix lack of quotation marks
[18:27:15] <MirahezeRelay>	 <pskyechology> whoopsie
[18:27:22] <MirahezeRelay>	 <pskyechology> watch miraheze explode
[18:27:50] <MirahezeRelay>	 <hihello_what_are_you_doing_here> :kaboom:
[18:28:20] <MirahezeRelay>	 <pskyechology> so glad that one accounts for my complacency
[18:38:59] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[18:44:00] <MirahezeRelay>	 <paladox> !log restart nginx on swiftproxy and ldap
[18:44:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:10:12] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27187MiB (5% inode=99%);
[21:49:09] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27212MiB (5% inode=99%);
[21:55:42] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27242MiB (5% inode=99%);
[22:12:25] <icinga-miraheze>	 PROBLEM - bast161 Current Load on bast161 is CRITICAL: LOAD CRITICAL - total load average: 3.95, 3.10, 1.36
[22:14:03] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[22:14:26] <icinga-miraheze>	 PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[22:14:57] <icinga-miraheze>	 PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[22:16:22] <icinga-miraheze>	 RECOVERY - bast161 Current Load on bast161 is OK: LOAD OK - total load average: 0.20, 1.43, 1.06
[22:40:31] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[22:41:12] <icinga-miraheze>	 RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[22:42:03] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures