[00:00:01] !log [www-data@test151] Began automatic backing up [00:00:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:00:05] !log [www-data@mwtask181] Began automatic backing up [00:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:01:17] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/c270aed6c5b447883e12e129c837c0b21c478fb0 [00:01:17] 02statichelp/03main 07WikiTideBot 03c270aed Bot: Auto-update Tech namespace pages 2025-07-01 00:01:15 [00:03:13] woa, it log [00:07:40] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56943MiB (12% inode=99%); [00:07:43] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57354MiB (12% inode=99%); [00:07:49] !log CREATE DATABASE skyewikibeta [00:07:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:52] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56910MiB (12% inode=99%); [00:11:30] !log manually recreated skyewikibeta database [00:11:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:07] !log skyewikibeta: DROP TABLE echo_event; DROP TABLE echo_notification; DROP TABLE echo_email_batch; DROP TABLE echo_target_page; DROP TABLE echo_push_provider; DROP TABLE echo_push_subscription; DROP TABLE echo_push_topic; [00:16:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:34] !log [blankeclair@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php sql --wiki=skyewikibeta /srv/mediawiki/1.43/extensions/Echo/sql/mysql/tables-generated.sql (END - exit=0) [00:16:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:11] remind me to never downgrade mediawiki [02:32:08] !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [02:32:08] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 19.35, 13.88 [02:32:09] !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [02:32:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:06] !log [skye@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [02:39:06] !log [skye@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [02:39:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.03, 23.89, 20.48 [02:49:41] [02mw-config] 07asko1 opened pull request #6013: T13922: Override wgTweekiSknGridNone for factoriopluswiki (07miraheze:03main...07asko1:03T13922) 13https://github.com/miraheze/mw-config/pull/6013 [02:50:42] miraheze/mw-config - asko1 the build passed. [02:52:17] woe is me, really need to come up with a funny username [02:54:08] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.54, 19.89, 20.16 [03:04:56] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 23940MiB (10% inode=95%); [03:08:24] !log [void@puppet181] Upgraded packages on bots171: sudo [03:08:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:36] !log [void@puppet181] Upgraded packages on cp161: sudo [03:08:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:50] !log [void@puppet181] Upgraded packages on changeprop201: sudo [03:08:51] RECOVERY - bots171 APT on bots171 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [03:08:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:04] !log [void@puppet181] Upgraded packages on cloud15: sudo [03:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:07] RECOVERY - changeprop201 APT on changeprop201 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [03:09:18] !log [void@puppet181] Upgraded packages on cloud19: sudo [03:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:32] !log [void@puppet181] Upgraded packages on db171: sudo [03:09:34] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 129 packages available for upgrade (0 critical updates). [03:09:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:43] !log [void@puppet181] Upgraded packages on db172: sudo [03:09:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:56] RECOVERY - db151 APT on db151 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [03:09:56] !log [void@puppet181] Upgraded packages on db151: sudo [03:09:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:12] !log [void@puppet181] Upgraded packages on bast161: sudo [03:10:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:19] RECOVERY - cp161 APT on cp161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:10:26] !log [void@puppet181] Upgraded packages on cloud17: sudo [03:10:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:30] RECOVERY - bast161 APT on bast161 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [03:10:40] !log [void@puppet181] Upgraded packages on kafka181: sudo [03:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:48] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 130 packages available for upgrade (0 critical updates). [03:10:53] !log [void@puppet181] Upgraded packages on matomo151: sudo [03:10:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:03] RECOVERY - db172 APT on db172 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:11:07] !log [void@puppet181] Upgraded packages on cloud16: sudo [03:11:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:11] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [03:11:21] !log [void@puppet181] Upgraded packages on db181: sudo [03:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:28] RECOVERY - db171 APT on db171 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [03:11:35] !log [void@puppet181] Upgraded packages on cloud20: sudo [03:11:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:38] RECOVERY - db181 APT on db181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [03:11:47] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [03:11:49] !log [void@puppet181] Upgraded packages on cp171: sudo [03:11:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:11:52] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [03:12:02] !log [void@puppet181] Upgraded packages on cp201: sudo [03:12:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:05] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [03:12:10] RECOVERY - cp201 APT on cp201 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:12:16] !log [void@puppet181] Upgraded packages on bast181: sudo [03:12:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:30] !log [void@puppet181] Upgraded packages on db161: sudo [03:12:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:46] !log [void@puppet181] Upgraded packages on eventgate181: sudo [03:12:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:56] RECOVERY - cp171 APT on cp171 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [03:12:59] !log [void@puppet181] Upgraded packages on db182: sudo [03:13:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:04] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 130 packages available for upgrade (0 critical updates). [03:13:16] !log [void@puppet181] Upgraded packages on cloud18: sudo [03:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:29] !log [void@puppet181] Upgraded packages on cp191: sudo [03:13:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:34] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:13:43] !log [void@puppet181] Upgraded packages on graylog161: sudo [03:13:45] RECOVERY - bast181 APT on bast181 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [03:13:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:57] !log [void@puppet181] Upgraded packages on ldap171: sudo [03:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:04] RECOVERY - db161 APT on db161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [03:14:09] !log [void@puppet181] Upgraded packages on mem151: sudo [03:14:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:21] RECOVERY - db182 APT on db182 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [03:14:24] !log [void@puppet181] Upgraded packages on mem191: sudo [03:14:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:31] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [03:14:33] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [03:14:37] !log [void@puppet181] Upgraded packages on mem161: sudo [03:14:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:46] RECOVERY - mem191 APT on mem191 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [03:14:51] !log [void@puppet181] Upgraded packages on mem201: sudo [03:14:52] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [03:14:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:04] !log [void@puppet181] Upgraded packages on mon181: sudo [03:15:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:15] !log [void@puppet181] Upgraded packages on mattermost1: sudo [03:15:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:20] RECOVERY - cp191 APT on cp191 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [03:15:25] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [03:15:29] RECOVERY - mem151 APT on mem151 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [03:15:30] !log [void@puppet181] Upgraded packages on mw152: sudo [03:15:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:46] !log [void@puppet181] Upgraded packages on mw153: sudo [03:15:47] RECOVERY - mem201 APT on mem201 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [03:15:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:15:54] RECOVERY - mem161 APT on mem161 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [03:16:01] !log [void@puppet181] Upgraded packages on mw151: sudo [03:16:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:07] RECOVERY - mon181 APT on mon181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:16:18] RECOVERY - mw153 APT on mw153 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:16:18] !log [void@puppet181] Upgraded packages on mw162: sudo [03:16:19] RECOVERY - mw152 APT on mw152 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:16:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:35] !log [void@puppet181] Upgraded packages on mw172: sudo [03:16:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:51] !log [void@puppet181] Upgraded packages on mw161: sudo [03:16:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:07] !log [void@puppet181] Upgraded packages on mw193: sudo [03:17:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:14] RECOVERY - mw151 APT on mw151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:17:22] RECOVERY - mw161 APT on mw161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:17:24] !log [void@puppet181] Upgraded packages on mw171: sudo [03:17:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:28] RECOVERY - mw193 APT on mw193 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:17:40] !log [void@puppet181] Upgraded packages on mw201: sudo [03:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:56] RECOVERY - mw172 APT on mw172 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:17:56] !log [void@puppet181] Upgraded packages on mw163: sudo [03:17:57] RECOVERY - mw171 APT on mw171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:17:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:00] RECOVERY - mw201 APT on mw201 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:18:03] RECOVERY - mw162 APT on mw162 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:18:12] !log [void@puppet181] Upgraded packages on mw202: sudo [03:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:28] !log [void@puppet181] Upgraded packages on mw191: sudo [03:18:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:33] RECOVERY - mw191 APT on mw191 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:18:43] !log [void@puppet181] Upgraded packages on mwtask151: sudo [03:18:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:58] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:19:00] RECOVERY - mw202 APT on mw202 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:19:00] !log [void@puppet181] Upgraded packages on mw182: sudo [03:19:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:16] !log [void@puppet181] Upgraded packages on mw203: sudo [03:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:34] !log [void@puppet181] Upgraded packages on mw192: sudo [03:19:35] RECOVERY - mw163 APT on mw163 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:19:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:50] !log [void@puppet181] Upgraded packages on mw181: sudo [03:19:52] RECOVERY - mw181 APT on mw181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:19:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:08] !log [void@puppet181] Upgraded packages on mw173: sudo [03:20:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:11] RECOVERY - mw182 APT on mw182 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:20:25] !log [void@puppet181] Upgraded packages on mw183: sudo [03:20:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:42] !log [void@puppet181] Upgraded packages on mwtask161: sudo [03:20:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:53] RECOVERY - mw183 APT on mw183 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:20:55] !log [void@puppet181] Upgraded packages on os151: sudo [03:20:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:07] RECOVERY - mw173 APT on mw173 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:21:08] !log [void@puppet181] Upgraded packages on rdb151: sudo [03:21:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:11] RECOVERY - mw203 APT on mw203 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:21:22] !log [void@puppet181] Upgraded packages on prometheus151: sudo [03:21:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:32] RECOVERY - mw192 APT on mw192 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:21:38] !log [void@puppet181] Upgraded packages on mwtask171: sudo [03:21:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:51] !log [void@puppet181] Upgraded packages on os161: sudo [03:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:03] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:22:03] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 49 packages available for upgrade (0 critical updates). [03:22:05] !log [void@puppet181] Upgraded packages on phorge171: sudo [03:22:05] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:22:08] RECOVERY - os161 APT on os161 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [03:22:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:15] RECOVERY - os151 APT on os151 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [03:22:19] !log [void@puppet181] Upgraded packages on os191: sudo [03:22:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:29] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [03:22:37] !log [void@puppet181] Upgraded packages on mwtask181: sudo [03:22:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:40] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:22:52] !log [void@puppet181] Upgraded packages on os162: sudo [03:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:06] !log [void@puppet181] Upgraded packages on reports171: sudo [03:23:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:21] !log [void@puppet181] Upgraded packages on ns1: sudo [03:23:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:36] !log [void@puppet181] Upgraded packages on puppet181: sudo [03:23:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:41] RECOVERY - reports171 APT on reports171 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [03:23:46] RECOVERY - os191 APT on os191 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [03:23:47] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [03:23:49] !log [void@puppet181] Upgraded packages on os201: sudo [03:23:50] RECOVERY - ns1 APT on ns1 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [03:23:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:06] !log [void@puppet181] Upgraded packages on ns2: sudo [03:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:22] !log [void@puppet181] Upgraded packages on swiftproxy171: sudo [03:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:38] !log [void@puppet181] Upgraded packages on swiftobject151: sudo [03:24:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:45] RECOVERY - os162 APT on os162 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [03:24:46] RECOVERY - ns2 APT on ns2 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [03:24:53] !log [void@puppet181] Upgraded packages on test151: sudo [03:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:08] !log [void@puppet181] Upgraded packages on swiftproxy161: sudo [03:25:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:26] !log [void@puppet181] Upgraded packages on swiftobject201: sudo [03:25:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:29] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:25:31] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [03:25:42] !log [void@puppet181] Upgraded packages on swiftobject191: sudo [03:25:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:46] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:25:47] RECOVERY - os201 APT on os201 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [03:25:58] !log [void@puppet181] Upgraded packages on swiftac171: sudo [03:26:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:26:11] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [03:26:15] !log [void@puppet181] Upgraded packages on swiftobject161: sudo [03:26:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:26:31] !log [void@puppet181] Upgraded packages on swiftobject171: sudo [03:26:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:26:38] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 70 packages available for upgrade (0 critical updates). [03:26:39] RECOVERY - test151 APT on test151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [03:26:50] !log [void@puppet181] Upgraded packages on swiftobject181: sudo [03:26:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:27:00] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:27:03] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 68 packages available for upgrade (0 critical updates). [03:27:19] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:27:53] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:28:12] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [03:36:38] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44% [03:38:38] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 5% [04:40:56] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 13158MiB (5% inode=95%); [04:52:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.02, 17.92, 11.98 [04:54:08] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.76, 18.44, 12.90 [05:00:56] PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 19842MiB (8% inode=95%); [05:58:08] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 18.41, 11.49 [06:36:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.63, 22.67, 23.47 [06:42:08] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.36, 15.82, 19.95 [07:36:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.74, 18.77, 14.15 [07:38:08] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 20.60, 15.39 [07:40:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.09, 21.51, 16.37 [07:42:08] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 22.87, 17.49 [08:46:08] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 7.52, 16.73, 22.27 [08:48:08] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.76, 13.04, 20.25 [09:27:12] [02mw-config] 07SomeMWDev approved pull request #6013 13https://github.com/miraheze/mw-config/pull/6013#pullrequestreview-2974487918 [09:27:24] [02mw-config] 07SomeMWDev merged 07asko1's pull request #6013: T13922: Override wgTweekiSknGridNone for factoriopluswiki (07miraheze:03main...07asko1:03T13922) 13https://github.com/miraheze/mw-config/pull/6013 [09:27:25] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/53cd0ae012fae568becf7415275499873f30ee8c [09:27:25] 02mw-config/03main 07Skye 0353cd0ae T13922: Override wgTweekiSknGridNone for factoriopluswiki (#6013)… [09:27:32] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [09:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:50] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [09:27:51] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [09:27:52] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49855MiB (10% inode=99%); [09:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:59] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [09:28:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:28:23] miraheze/mw-config - SomeMWDev the build passed. [09:29:40] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49842MiB (10% inode=99%); [09:33:42] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Vector'} to test151 [09:33:44] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_skins': 'Vector'} to test151 - SUCCESS in 2s [09:33:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:33:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:34:12] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.43', 'upgrade_skins': 'Vector'} to all [09:34:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:34:38] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.43', 'upgrade_skins': 'Vector'} to all - SUCCESS in 26s [09:34:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:40] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=vocaloidlyricswiki --import-prefix=wikia:vocaloidlyrics --from=KunKunAguero --to=Osuka --no-run (END - exit=0) [09:44:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:50] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=vocaloidlyricswiki --import-prefix=wikia:vocaloidlyrics --from=KunKunAguero --to=Osuka (END - exit=0) [09:44:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:45:04] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=vocaloidlyricswiki --import-prefix=fandom:vocaloidlyrics --from=KunKunAguero --to=Osuka --no-run (END - exit=0) [09:45:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:45:13] !log [somerandomdeveloper@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:AssignImportedEdits --wiki=vocaloidlyricswiki --import-prefix=fandom:vocaloidlyrics --from=KunKunAguero --to=Osuka (END - exit=0) [09:45:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:51:43] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49937MiB (10% inode=99%); [11:08:56] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 13170MiB (5% inode=95%); [11:28:01] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:28:02] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:28:04] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:04] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:28:04] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:28:08] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:09] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [11:28:13] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [11:28:14] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:15] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:16] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:28:25] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:29] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:28:30] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:28:42] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:28:44] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [11:28:48] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [11:28:55] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:28:56] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [11:28:58] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:29:01] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:29:05] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:29:05] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.018 second response time [11:29:08] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:29:14] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [11:29:24] @Infrastructure Specialists i think we have a problem [11:29:25] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:29:27] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 144.48, 66.98, 28.29 [11:29:28] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:28] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:31] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:29:34] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:29:35] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:35] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:29:39] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:42] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:29:43] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:47] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:48] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [11:29:49] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:29:54] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:54] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:57] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:58] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:30:02] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:30:13] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [11:30:38] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [11:31:02] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:31:11] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: [11:32:01] PROBLEM - db161 MariaDB on db161 is UNKNOWN: [11:32:46] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 8.197 second response time [11:32:51] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [11:32:59] The fuck [11:33:00] @progeest Watch your language. [11:33:00] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [11:33:03] oh oops [11:33:05] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.023 second response time [11:33:07] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.013 second response time [11:33:07] My bad [11:33:13] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.019 second response time [11:33:14] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [11:33:14] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.014 second response time [11:33:20] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.227 second response time [11:33:25] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [11:33:27] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [11:33:32] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.020 second response time [11:33:37] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 1.401 second response time [11:33:42] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.245 second response time [11:33:46] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.550 second response time [11:33:48] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.293 second response time [11:33:54] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [11:33:56] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4034 bytes in 0.066 second response time [11:33:57] PROBLEM - db161 MariaDB on db161 is CRITICAL: Too many connections [11:33:58] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4034 bytes in 0.860 second response time [11:34:01] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.413 second response time [11:34:04] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [11:34:08] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.018 second response time [11:34:09] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.020 second response time [11:34:12] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [11:34:15] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 200 - Status line output matched "HTTP/2 200" - 271 bytes in 0.023 second response time [11:34:19]