[00:01:28] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/7b04bc59dcf20d77ef8ff97b9a00c1a5d5857b4d
[00:01:29] <MacFanBot>	 02statichelp/03main 07WikiTideBot 037b04bc5 Bot: Auto-update Tech namespace pages 2025-10-12 00:01:26
[00:11:48] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 53947MiB (11% inode=99%);
[00:13:18] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 54450MiB (11% inode=99%);
[00:13:49] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 53331MiB (11% inode=99%);
[02:01:37] <MacFanBot>	 [02CreateWiki] 07Universal-Omega created 03revert-781-ollama (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/8a8a52a54d5f
[02:01:37] <MacFanBot>	 02CreateWiki/03revert-781-ollama 07CosmicAlpha 038a8a52a Revert "Refactor to use Ollama instead of OpenAI (#781)"…
[02:02:24] <MacFanBot>	 [02CreateWiki] 07Universal-Omega opened pull request #785: Revert "Refactor to use Ollama instead of OpenAI" (03main...03revert-781-ollama) 13https://github.com/miraheze/CreateWiki/pull/785
[02:02:46] <MacFanBot>	 [02CreateWiki] 07coderabbitai[bot] commented on pull request #785: <details> […] 13https://github.com/miraheze/CreateWiki/pull/785#issuecomment-3393832054
[02:04:45] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:CheckWikiDatabases --wiki=metawiki --delete (END - exit=0)
[02:04:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:07:09] <MacFanBot>	 [02CreateWiki] 07Universal-Omega merged pull request #785: Revert "Refactor to use Ollama instead of OpenAI" (03main...03revert-781-ollama) 13https://github.com/miraheze/CreateWiki/pull/785
[02:07:09] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/eee53fa2eb984b85c7bd1fd5e23f1be860a0a3e5
[02:07:11] <MacFanBot>	 02CreateWiki/03main 07CosmicAlpha 03eee53fa Revert "Refactor to use Ollama instead of OpenAI" (#785)…
[02:07:13] <MacFanBot>	 [02CreateWiki] 07Universal-Omega 04deleted 03revert-781-ollama at 038a8a52a 13https://api.github.com/repos/miraheze/CreateWiki/commit/8a8a52a
[02:07:41] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:CheckWikiDatabases --wiki=metawiki --tables --delete (END - exit=0)
[02:07:42] <MacFanBot>	 [02CreateWiki] 07codecov[bot] commented on pull request #785: :x: Patch coverage is `1.76991%` with `111 lines` in your changes missing coverage. Please review. […] 13https://github.com/miraheze/CreateWiki/pull/785#issuecomment-3393836522
[02:07:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:07:56] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all
[02:08:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:08:20] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 24s
[02:08:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:17:49] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49768MiB (10% inode=99%);
[02:19:35] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/ce4f30f5742abb347935a7bcf47cf5ccf6f0bacb
[02:19:35] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 03ce4f30f Temp turn off job queue for creations
[02:20:27] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[02:20:55] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c9ddc76fa75560daac32763cd40e63c2b73d1cd6
[02:20:56] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03c9ddc76 Temp AI dryrun
[02:24:36] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[02:24:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:24:56] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[02:24:59] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s
[02:25:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:26:38] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:CheckWikiDatabases --wiki=metawiki --delete (END - exit=0)
[02:26:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:29:03] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[02:39:48] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49836MiB (10% inode=99%);
[02:55:18] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49882MiB (10% inode=99%);
[03:04:26] <icinga-miraheze>	 PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 2603MiB (8% inode=80%);
[03:06:26] <icinga-miraheze>	 RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4804MiB (15% inode=80%);
[10:20:30] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[10:51:49] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/ae012cdd2e29aa59be7ca2f4b7a98e8ba9c01c47
[10:51:49] <MacFanBot>	 02puppet/03main 07paladox 03ae012cd matomo: upgrade to 5.5.0
[11:11:55] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/a14001541521
[11:11:55] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 03a140015 cache: lower file size to 370g
[11:12:01] <MacFanBot>	 [02puppet] 07paladox opened pull request #4563: cache: lower file size to 370g (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4563
[11:12:11] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4563: <details> […] 13https://github.com/miraheze/puppet/pull/4563#issuecomment-3394178769
[11:15:33] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56312MiB (12% inode=99%);
[11:15:41] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 56936MiB (12% inode=99%);
[11:15:46] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 55485MiB (12% inode=99%);
[11:16:06] <MacFanBot>	 [02puppet] 07paladox retitled PR #4563: "cache: lower file size to 370g" ➜ "cache: lower file size to 360g" 13https://github.com/miraheze/puppet/pull/4563
[11:16:14] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-3 13https://github.com/miraheze/puppet/commit/b2584c8f4f250e1d4c497721d94b75c78125931a
[11:16:15] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 03b2584c8 Update cache.yaml
[11:16:20] <MacFanBot>	 [02puppet] 07paladox merged pull request #4563: cache: lower file size to 360g (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4563
[11:16:21] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1e5b557edc7a69adcbf60191e3d9ba94e9c77219
[11:16:21] <MacFanBot>	 02puppet/03main 07paladox 031e5b557 cache: lower file size to 360g (#4563)
[11:16:22] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-3 at 03b2584c8 13https://api.github.com/repos/miraheze/puppet/commit/b2584c8
[11:18:07] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[11:26:07] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[11:28:06] <icinga-miraheze>	 RECOVERY - cloud19 NTP time on cloud19 is OK: NTP OK: Offset -2.339482307e-05 secs
[11:28:23] <MirahezeRelay>	 <paladox> !log restart ntpd service on cloud19
[11:28:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:16:24] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.65, 8.94, 7.42
[13:18:24] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.89, 8.13, 7.31
[13:32:19] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.73, 9.91, 8.61
[13:36:18] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.54, 9.02, 8.60
[14:13:49] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49744MiB (10% inode=99%);
[14:32:02] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.13, 12.89, 9.25
[14:34:02] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 6.76, 10.28, 8.74
[14:36:01] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.01, 9.40, 8.59
[14:41:48] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49829MiB (10% inode=99%);
[14:57:18] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49855MiB (10% inode=99%);
[15:00:00] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.93, 10.50, 8.84
[15:02:00] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.89, 9.34, 8.62
[15:10:00] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.93, 11.68, 9.97
[15:12:00] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.47, 10.05, 9.59
[16:00:30] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1
[16:39:48] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.20, 11.52, 8.16
[16:43:47] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.58, 8.93, 7.87
[17:05:40] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.75, 10.31, 8.76
[17:07:40] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.88, 9.41, 8.61
[17:41:29] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.58, 15.59, 11.85
[17:45:28] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 5.94, 10.81, 10.81
[17:51:27] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.38, 8.71, 9.87
[18:30:16] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 23.52, 12.76, 9.71
[18:32:15] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.27, 10.69, 9.34
[18:34:15] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 21.95, 13.31, 10.34
[18:36:14] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.81, 11.45, 10.00
[18:38:14] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.20, 9.69, 9.52
[19:10:05] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.89, 10.50, 8.83
[19:12:04] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.87, 8.84, 8.43
[20:08:45] <icinga-miraheze>	 PROBLEM - bots171 IRC Log Bot on bots171 is CRITICAL: PROCS CRITICAL: 0 processes with args 'adminlogbot.py'
[20:24:45] <icinga-miraheze>	 RECOVERY - bots171 IRC Log Bot on bots171 is OK: PROCS OK: 1 process with args 'adminlogbot.py'
[20:44:34] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/928da2eff604
[20:44:34] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 03928da2e prometheus: increase --storage.tsdb.retention.time to 1000…
[20:44:38] <MacFanBot>	 [02puppet] 07paladox opened pull request #4564: prometheus: increase --storage.tsdb.retention.time to 1000 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4564
[20:45:07] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4564: <details> […] 13https://github.com/miraheze/puppet/pull/4564#issuecomment-3395336004
[20:46:03] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-3 13https://github.com/miraheze/puppet/commit/2c3bfbfdbd95ab30b688a715737211ba2776f1a1
[20:46:03] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 032c3bfbf Update prometheus.systemd.erb
[20:46:29] <MacFanBot>	 [02puppet] 07paladox merged pull request #4564: prometheus: increase --storage.tsdb.retention.time to 1000 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4564
[20:46:29] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5f3cfa6760c73cdd35a53f2a530e353d96dbae3a
[20:46:31] <MacFanBot>	 02puppet/03main 07paladox 035f3cfa6 prometheus: increase --storage.tsdb.retention.time to 1000 (#4564)…
[20:46:33] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-3 at 032c3bfbf 13https://api.github.com/repos/miraheze/puppet/commit/2c3bfbf
[20:47:04] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/35a94946ac8f
[20:47:04] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 0335a9494 prometheus: increase --storage.tsdb.retention.size to 120gb…
[20:47:09] <MacFanBot>	 [02puppet] 07paladox opened pull request #4565: prometheus: increase --storage.tsdb.retention.size to 120gb (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4565
[20:47:15] <MacFanBot>	 [02puppet] 07paladox merged pull request #4565: prometheus: increase --storage.tsdb.retention.size to 120gb (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4565
[20:47:17] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0c511a95f2bdf15cca629a8b76a179e04865a10a
[20:47:19] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-3 at 0335a9494 13https://api.github.com/repos/miraheze/puppet/commit/35a9494
[20:47:20] <MacFanBot>	 02puppet/03main 07paladox 030c511a9 prometheus: increase --storage.tsdb.retention.size to 120gb (#4565)…
[20:47:22] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4565: <details> […] 13https://github.com/miraheze/puppet/pull/4565#issuecomment-3395337598