[00:13:25] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73655MiB (16% inode=99%); [00:13:35] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74068MiB (16% inode=99%); [00:14:15] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 73313MiB (16% inode=99%); [00:15:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.15, 23.95, 23.49 [00:31:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 23.55, 23.23 [00:39:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.31, 23.91, 23.62 [00:45:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 23.39, 23.37 [01:11:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.40, 22.44, 23.97 [01:15:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.70, 14.40, 20.38 [03:15:32] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/b6246f4bb82093704e14583eca9828228452de74 [03:15:32] 02ssl/03main 07WikiTideBot 03b6246f4 Bot: Auto-update domain lists [03:33:40] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.82, 18.20, 14.46 [03:35:40] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.81, 17.37, 14.60 [03:57:20] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [04:05:20] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [05:55:30] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/cf952ea5e85588c4588917d1a2f8aa828fe56fd2 [05:55:30] 02ssl/03main 07WikiTideBot 03cf952ea Bot: Auto-update domain lists [06:05:34] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/df8a661b5edfba24167101823a6c22056b634892 [06:05:34] 02ssl/03main 07WikiTideBot 03df8a661 Bot: Auto-update domain lists [06:31:06] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 61978MiB (10% inode=99%); [06:55:39] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 19.92, 12.00 [06:59:06] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 35449MiB (5% inode=99%); [07:23:06] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 166034MiB (28% inode=99%); [07:49:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 7.22, 17.36, 22.88 [07:53:39] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.89, 11.64, 19.25 [09:31:41] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.25, 19.91, 15.54 [09:33:40] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.71, 19.94, 16.10 [10:17:46] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.83, 14.14, 9.12 [10:19:46] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.79, 11.47, 8.76 [10:21:46] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.21, 9.14, 8.22 [11:19:25] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [11:21:25] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 17% [12:25:02] miraheze/GlobalNewFiles - translatewiki the build passed. [12:27:03] miraheze/IncidentReporting - translatewiki the build passed. [12:27:33] miraheze/MatomoAnalytics - translatewiki the build passed. [12:29:19] miraheze/DataDump - translatewiki the build passed. [12:29:40] miraheze/RemovePII - translatewiki the build passed. [12:33:53] miraheze/ManageWiki - translatewiki the build passed. [12:36:09] miraheze/MirahezeMagic - translatewiki the build passed. [12:40:06] miraheze/ImportDump - translatewiki the build passed. [12:41:48] miraheze/CreateWiki - translatewiki the build passed. [13:06:17] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.62, 13.07, 9.37 [13:08:15] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.05, 9.94, 8.67 [14:05:41] miraheze/mw-config - SomeMWDev the build passed. [14:06:01] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [14:06:02] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [14:06:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:09] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:06:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:32] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [14:06:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:54] miraheze/mw-config - SomeMWDev the build passed. [14:18:44] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [14:18:45] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:18:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:42] miraheze/mw-config - SomeMWDev the build passed. [14:31:26] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [14:31:27] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [14:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:45] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:31:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:08] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [14:32:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:15] miraheze/mw-config - SomeMWDev the build passed. [15:31:46] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 8.65, 12.15, 8.75 [15:33:46] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.77, 9.68, 8.26 [15:56:12] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_skins': 'Monaco'} to test151 [15:56:14] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_skins': 'Monaco'} to test151 - SUCCESS in 1s [15:56:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:19] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_skins': 'Monaco'} to all [15:56:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:44] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_skins': 'Monaco'} to all - SUCCESS in 25s [15:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:52] yay!!! monaco!!! :3 [16:19:44] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49764MiB (10% inode=99%); [16:27:43] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 11.85, 12.58, 8.91 [16:29:43] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.19, 9.41, 8.19 [16:31:13] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 17.35, 11.07, 7.26 [16:33:08] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 7.15, 9.40, 7.12 [16:44:56] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49891MiB (10% inode=99%); [16:46:01] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49812MiB (10% inode=99%); [16:55:28] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=aurorafestwiki --new=outscwiki --user=Skye (END - exit=0) [16:55:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:32] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=outscwiki (END - exit=0) [16:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:21:55] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ResetWiki --wiki=loginwiki --dbname=modernnationswiki --requester=Dutchy (END - exit=0) [17:22:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:45] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php Cargo:cargoRecreateData --wiki=bluearchivewiki --table unique_gear --replacement (END - exit=0) [18:01:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:35] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/ac4b57d11e2fce3b095b93727aae611025cc80c0 [18:15:35] 02ssl/03main 07WikiTideBot 03ac4b57d Bot: Auto-update domain lists [19:49:31] [02puppet] 07paladox merged 07MacFan4000's pull request #4592: bacckupwikis: specify IA collection as miraheze_stacks (07miraheze:03main...07MacFan4000:03patch-69) 13https://github.com/miraheze/puppet/pull/4592 [19:49:32] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/fa593d778d0e4a8dcfda3f96bb3d13e762381c58 [19:49:33] 02puppet/03main 07MacFan4000 03fa593d7 bacckupwikis: specify IA collection as miraheze_stacks (#4592)… [21:01:15] Thanks MacFan4000 [21:01:51] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php Cargo:cargoRecreateData --wiki=100bangaiwiki --table Characters --replacement (END - exit=256) [21:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:16] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 52% [21:09:15] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 31% [22:55:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/91a674209f442fa567307f7313c51b8a6002f34b [22:55:29] 02ssl/03main 07WikiTideBot 0391a6742 Bot: Auto-update domain lists [23:05:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/8770ac0556eb7a18ff9b9b2aa3821ebe66040134 [23:05:19] 02ssl/03main 07WikiTideBot 038770ac0 Bot: Auto-update domain lists