[00:01:18] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/85af93ea4a66abf8c46859a17affe9c2f313313a [00:01:18] 02statichelp/03main 07WikiTideBot 0385af93e Bot: Auto-update Tech namespace pages 2025-11-16 00:01:16 [00:04:51] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26759MiB (5% inode=99%); [00:14:51] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 72608MiB (15% inode=99%); [00:15:11] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 65771MiB (14% inode=99%); [02:55:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f4e6912b00f848acf646a227b6f57a73f500e6ec [02:55:18] 02ssl/03main 07WikiTideBot 03f4e6912 Bot: Auto-update domain lists [03:03:23] PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 2122MiB (6% inode=79%); [03:07:23] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4406MiB (14% inode=79%); [04:05:20] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a506cfbf1fdb7b5e1990376667125d3d4453b635 [04:05:20] 02ssl/03main 07WikiTideBot 03a506cfb Bot: Auto-update domain lists [05:58:17] !log [blankeclair@mwtask181:~]$ curl -X DELETE https://opensearch-mw.wikitide.net/hivesocietywiki_{content,general} [05:58:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:49] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=hivesocietywiki --startOver (END - exit=0) [05:58:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:59] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=hivesocietywiki (END - exit=0) [05:59:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:04] !log Logging via /usr/local/bin/logsalmsg "sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=buildislandslarpwiki --new=durraniswiki "--user=BlankEclair (Miraheze)" (END - exit=0)" [06:04:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:39:11] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49729MiB (10% inode=99%); [08:59:21] !log [blankeclair@mwtask181] Starting import for robloxrivalswiki (XML: None; Images: all_images/) (START) [08:59:22] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=robloxrivalswiki --sleep=1 '--comment=Importing images from https://robloxrivals.fandom.com ([[phorge:T14534|T14534]])' -- all_images/ (START) [08:59:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:59:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:04:53] !log [blankeclair@mwtask181:~]$ curl -X DELETE https://opensearch-mw.wikitide.net/cpzerowiki_{content,general} [09:04:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:05:22] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=cpzerowiki --startOver (END - exit=0) [09:05:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:31] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=cpzerowiki (END - exit=0) [09:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:43] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=robloxrivalswiki --sleep=1 '--comment=Importing images from https://robloxrivals.fandom.com ([[phorge:T14534|T14534]])' -- all_images/ (END - exit=0) [09:27:44] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=robloxrivalswiki --update (START) [09:27:45] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=robloxrivalswiki --update (END - exit=0) [09:27:46] !log [blankeclair@mwtask181] Finished import for robloxrivalswiki (XML: None; Images: all_images/) (END - exit=0) [09:27:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:38:10] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [10:46:10] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [11:08:10] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [11:10:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49840MiB (10% inode=99%); [11:16:10] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [13:09:07] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49810MiB (10% inode=99%); [14:17:22] [02puppet] 07MacFan4000 opened pull request #4605: move somerandomdeveloper to ops (07miraheze:03main...07MacFan4000:03patch-70) 13https://github.com/miraheze/puppet/pull/4605 [14:17:27] [02puppet] 07coderabbitai[bot] commented on pull request #4605:
[…] 13https://github.com/miraheze/puppet/pull/4605#issuecomment-3538801019 [15:03:11] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27196MiB (5% inode=99%); [15:48:21] [02puppet] 07paladox merged 07MacFan4000's pull request #4605: move somerandomdeveloper to ops (07miraheze:03main...07MacFan4000:03patch-70) 13https://github.com/miraheze/puppet/pull/4605 [15:48:21] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/24fbebb34bddeee98a895a2959856bbb096a1332 [15:48:23] 02puppet/03main 07MacFan4000 0324fbebb move somerandomdeveloper to ops (#4605) [15:56:55] !log update SRD's team on GitHub (he was already owner by @paladox) [15:56:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:51] !log empower SRD on Phorge [15:57:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:12] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[openwebui_pip_install] [16:42:18] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:00:13] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/81ac209bca3a [18:00:13] 02puppet/03paladox-patch-6 07paladox 0381ac209 Remove varnish usage setting from phorge171.yaml [18:00:18] [02puppet] 07paladox opened pull request #4606: Remove varnish usage setting from phorge171.yaml (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4606 [18:00:24] [02puppet] 07coderabbitai[bot] commented on pull request #4606:
[…] 13https://github.com/miraheze/puppet/pull/4606#issuecomment-3539069397 [18:00:48] [02puppet] 07paladox retitled PR #4606: "Remove varnish usage setting from phorge171.yaml" ➜ "phorge171: unset nginx::use_varnish_directly" 13https://github.com/miraheze/puppet/pull/4606 [18:02:53] [02puppet] 07paladox merged pull request #4606: phorge171: unset nginx::use_varnish_directly (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4606 [18:02:53] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0571fcaefde36b5bfc8a1ce0daa6076d11d6c7ef [18:02:53] 02puppet/03main 07paladox 030571fca phorge171: unset nginx::use_varnish_directly (#4606)… [18:02:55] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 0381ac209 13https://api.github.com/repos/miraheze/puppet/commit/81ac209 [18:37:33] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 40% [18:39:33] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 16% [18:59:09] [02mw-config] 07SomeMWDev created 03excimer (+1 new commit) 13https://github.com/miraheze/mw-config/commit/fbccb75c2b4d [18:59:09] 02mw-config/03excimer 07SomeRandomDeveloper 03fbccb75 [POC] Generate flame graph using excimer [18:59:30] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [18:59:31] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:59:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:02] [02mw-config] 07SomeMWDev pushed 1 new commit to 03excimer 13https://github.com/miraheze/mw-config/commit/b81b0eb27810a1cf42c03d4c697c506ea485f55f [19:01:02] 02mw-config/03excimer 07SomeRandomDeveloper 03b81b0eb use if instead of elseif [19:01:11] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [19:01:12] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:01:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:44] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [19:25:45] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:25:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:03] [02mw-config] 07SomeMWDev 04deleted 03excimer at 03b81b0eb 13https://api.github.com/repos/miraheze/mw-config/commit/b81b0eb [19:54:52] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.18, 18.17, 10.97 [19:58:44] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.30, 17.83, 12.61 [20:09:09] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 26965MiB (5% inode=99%); [20:13:16] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 76169MiB (16% inode=99%); [20:15:09] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 71244MiB (15% inode=99%); [20:54:02] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.79, 18.94, 12.73 [20:56:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.23, 17.76, 13.12 [23:23:48] [02ErrorPages] 07SomeMWDev created 03sync-managewiki-cache (+1 new commit) 13https://github.com/miraheze/ErrorPages/commit/17a2389f2739 [23:23:48] 02ErrorPages/03sync-managewiki-cache 07SomeRandomDeveloper 0317a2389 Sync ManageWiki cache for deleted wikis… [23:25:47] [02ErrorPages] 07SomeMWDev opened pull request #54: Sync ManageWiki cache for deleted wikis (03main...03sync-managewiki-cache) 13https://github.com/miraheze/ErrorPages/pull/54 [23:25:53] [02ErrorPages] 07coderabbitai[bot] commented on pull request #54:
[…] 13https://github.com/miraheze/ErrorPages/pull/54#issuecomment-3539473446 [23:26:36] miraheze/ErrorPages - SomeMWDev the build passed. [23:29:32] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to [mwtask181, mw183] [23:29:33] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to [mwtask181, mw183] - SUCCESS in 0s [23:29:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:35] !log [somerandomdeveloper@mwtask181] starting deploy of {'errorpages': True} to mw183 [23:30:36] !log [somerandomdeveloper@mwtask181] finished deploy of {'errorpages': True} to mw183 - SUCCESS in 1s [23:30:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:52] [02ErrorPages] 07SomeMWDev pushed 1 new commit to 03sync-managewiki-cache 13https://github.com/miraheze/ErrorPages/commit/5bf8adadc5a4e097eeff2a0965d12cbea099f67f [23:59:52] 02ErrorPages/03sync-managewiki-cache 07SomeRandomDeveloper 035bf8ada temporarily duplicate caching settings