[00:01:11] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/d1dee8db95cbda640c4e859f7b7731a522c9eb2c [00:01:11] 02statichelp/03main 07WikiTideBot 03d1dee8d Bot: Auto-update Tech namespace pages 2025-08-24 00:01:09 [00:07:27] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56796MiB (12% inode=99%); [00:07:33] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57188MiB (12% inode=99%); [00:08:13] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56484MiB (12% inode=99%); [03:03:48] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [03:04:03] PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 2093MiB (6% inode=81%); [03:08:03] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4195MiB (13% inode=81%); [06:57:40] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [07:05:40] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:42:13] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49829MiB (10% inode=99%); [08:01:28] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49875MiB (10% inode=99%); [08:03:48] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [08:25:33] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49921MiB (10% inode=99%); [10:11:50] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 7.17, 6.29, 5.38 [10:13:50] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 6.63, 6.43, 5.54 [10:33:50] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 7.89, 6.59, 5.82 [10:35:50] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 4.41, 5.83, 5.65 [11:19:01] [02mw-config] 07SomeMWDev created 03T14182 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/87970d100ece [11:19:01] 02mw-config/03T14182 07SomeRandomDeveloper 0387970d1 T14182: Set $wgTranslateDisabledTargetLanguages for astralpartywiki [11:19:42] [02mw-config] 07SomeMWDev opened pull request #6070: T14182: Set $wgTranslateDisabledTargetLanguages for astralpartywiki (03main...03T14182) 13https://github.com/miraheze/mw-config/pull/6070 [11:20:35] miraheze/mw-config - SomeMWDev the build passed. [11:20:59] [02mw-config] 07SomeMWDev merged pull request #6070: T14182: Set $wgTranslateDisabledTargetLanguages for astralpartywiki (03main...03T14182) 13https://github.com/miraheze/mw-config/pull/6070 [11:21:01] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/552e5db167ecbc4d62130a4dfc6c577a51bfcc7d [11:21:03] [02mw-config] 07SomeMWDev 04deleted 03T14182 at 0387970d1 13https://api.github.com/repos/miraheze/mw-config/commit/87970d1 [11:21:05] 02mw-config/03main 07SomeRandomDeveloper 03552e5db T14182: Set $wgTranslateDisabledTargetLanguages for astralpartywiki (#6070)… [11:21:24] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [11:21:50] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [11:21:54] miraheze/mw-config - SomeMWDev the build passed. [11:22:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:30:47] [02mw-config] 07SomeMWDev merged pull request #6063: Remove duplicated settings and array keys (03main...03cleanup-duplicates) 13https://github.com/miraheze/mw-config/pull/6063 [11:30:47] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/1154a83d43ac01407b8d88bd5b3628e8e5dd3d02 [11:30:47] 02mw-config/03main 07SomeRandomDeveloper 031154a83 Remove duplicated settings and array keys (#6063)… [11:30:48] [02mw-config] 07SomeMWDev 04deleted 03cleanup-duplicates at 038f1d33c 13https://api.github.com/repos/miraheze/mw-config/commit/8f1d33c [11:31:43] miraheze/mw-config - SomeMWDev the build passed. [11:31:44] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [11:31:45] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [11:31:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:31:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:10] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [11:32:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:32:39] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 28s [11:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:35] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 11.39, 5.15 [13:40:02] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=adrenalinerblxwiki --new=adrenwiki --user=Skye (END - exit=0) [13:40:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:40:27] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.24, 9.73, 7.03 [13:41:16] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=adrenwiki (END - exit=0) [13:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:42:49] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1967MiB (10% inode=90%); [19:02:20] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6724MiB (37% inode=89%); [20:15:45] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=sheshenchudiwiki --new=mcacgnwiki --user=Skye (END - exit=0) [20:19:08] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=mcacgnwiki (END - exit=0) [20:19:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756071140000&orgId=1&to=1756074776680 [22:57:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756072610000&orgId=1&to=1756076276684[Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756071140000&orgId=1&to=1756076276684 [23:12:56] [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756072610000&orgId=1&to=1756077176686[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756071140000&orgId=1&to=1756077140000 [23:40:58] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 17.80, 15.66, 7.83 [23:42:54] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 3.33, 10.87, 6.99 [23:44:51] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.10, 7.67, 6.27