[00:00:58] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/d0b86b526fa223c0e6c630760778c1bfb6d8715a [00:00:58] 02statichelp/03main 07WikiTideBot 03d0b86b5 Bot: Auto-update Tech namespace pages 2026-06-12 00:00:55 [00:03:07] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [00:13:45] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45250MiB (10.0% inode=100%); [00:31:07] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [01:05:17] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44585MiB (9.8% inode=100%); [01:05:40] !log [void@puppet181] Upgraded packages on mattermost2: libdbi-perl [01:05:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:56] !log [void@puppet181] Upgraded packages on bast161: libdbi-perl [01:05:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:03] RECOVERY - mattermost2 APT on mattermost2 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [01:06:05] RECOVERY - bast161 APT on bast161 is OK: APT OK: 83 packages available for upgrade (0 critical updates). [01:06:12] !log [void@puppet181] Upgraded packages on db172: libdbi-perl [01:06:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:28] !log [void@puppet181] Upgraded packages on db151: libdbi-perl [01:06:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:45] !log [void@puppet181] Upgraded packages on db171: libdbi-perl [01:06:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:02] !log [void@puppet181] Upgraded packages on db182: libdbi-perl [01:07:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:17] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 91456MiB (20.1% inode=100%); [01:07:21] !log [void@puppet181] Upgraded packages on db161: libdbi-perl [01:07:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:38] !log [void@puppet181] Upgraded packages on db181: libdbi-perl [01:07:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:46] RECOVERY - db181 APT on db181 is OK: APT OK: 85 packages available for upgrade (0 critical updates). [01:07:50] RECOVERY - db151 APT on db151 is OK: APT OK: 85 packages available for upgrade (0 critical updates). [01:07:53] RECOVERY - db161 APT on db161 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [01:07:56] !log [void@puppet181] Upgraded packages on bast181: libdbi-perl [01:07:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:00] RECOVERY - db182 APT on db182 is OK: APT OK: 85 packages available for upgrade (0 critical updates). [01:08:00] RECOVERY - db172 APT on db172 is OK: APT OK: 85 packages available for upgrade (0 critical updates). [01:08:12] !log [void@puppet181] Upgraded packages on mw151: libinput-bin and libinput10 [01:08:15] RECOVERY - db171 APT on db171 is OK: APT OK: 85 packages available for upgrade (0 critical updates). [01:08:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:18] RECOVERY - mw151 APT on mw151 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:08:30] !log [void@puppet181] Upgraded packages on mw153: libinput-bin and libinput10 [01:08:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:48] !log [void@puppet181] Upgraded packages on db192: libdbi-perl [01:08:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:01] RECOVERY - mw153 APT on mw153 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:09:10] !log [void@puppet181] Upgraded packages on db201: libdbi-perl [01:09:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:21] RECOVERY - db192 APT on db192 is OK: APT OK: 93 packages available for upgrade (0 critical updates). [01:09:28] !log [void@puppet181] Upgraded packages on mw162: libinput-bin and libinput10 [01:09:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:45] !log [void@puppet181] Upgraded packages on mon181: libdbi-perl [01:09:45] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 86372MiB (19.1% inode=100%); [01:09:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:49] RECOVERY - mon181 APT on mon181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [01:09:54] RECOVERY - bast181 APT on bast181 is OK: APT OK: 84 packages available for upgrade (0 critical updates). [01:10:03] !log [void@puppet181] Upgraded packages on mw172: libinput-bin and libinput10 [01:10:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:20] RECOVERY - mw162 APT on mw162 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:10:22] !log [void@puppet181] Upgraded packages on mw173: libinput-bin and libinput10 [01:10:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:40] !log [void@puppet181] Upgraded packages on mw171: libinput-bin and libinput10 [01:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:48] RECOVERY - db201 APT on db201 is OK: APT OK: 93 packages available for upgrade (0 critical updates). [01:10:58] RECOVERY - mw172 APT on mw172 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:10:59] !log [void@puppet181] Upgraded packages on mw152: libinput-bin and libinput10 [01:11:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:05] RECOVERY - mw173 APT on mw173 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:11:15] RECOVERY - mw152 APT on mw152 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:11:19] !log [void@puppet181] Upgraded packages on mw163: libinput-bin and libinput10 [01:11:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:45] !log [void@puppet181] Upgraded packages on mw161: libinput-bin and libinput10 [01:11:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:03] !log [void@puppet181] Upgraded packages on mw191: libinput-bin and libinput10 [01:12:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:21] !log [void@puppet181] Upgraded packages on mw181: libinput-bin and libinput10 [01:12:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:35] RECOVERY - mw171 APT on mw171 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:12:41] !log [void@puppet181] Upgraded packages on mw182: libinput-bin and libinput10 [01:12:41] RECOVERY - mw191 APT on mw191 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:12:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:56] RECOVERY - mw163 APT on mw163 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:13:02] !log [void@puppet181] Upgraded packages on mw183: libinput-bin and libinput10 [01:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:10] RECOVERY - mw161 APT on mw161 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:13:25] !log [void@puppet181] Upgraded packages on mw192: libinput-bin and libinput10 [01:13:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:41] !log [void@puppet181] Upgraded packages on phorge171: libdbi-perl [01:13:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:46] RECOVERY - mw192 APT on mw192 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:13:56] RECOVERY - mw183 APT on mw183 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:13:57] !log [void@puppet181] Upgraded packages on reports171: libdbi-perl [01:13:58] RECOVERY - reports171 APT on reports171 is OK: APT OK: 79 packages available for upgrade (0 critical updates). [01:14:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:08] RECOVERY - mw182 APT on mw182 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:14:12] RECOVERY - mw181 APT on mw181 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:14:15] !log [void@puppet181] Upgraded packages on mwtask161: libinput-bin and libinput10 [01:14:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:19] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:14:32] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [01:14:38] !log [void@puppet181] Upgraded packages on mw193: libinput-bin and libinput10 [01:14:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:55] !log [void@puppet181] Upgraded packages on mwtask171: libinput-bin and libinput10 [01:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:04] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [01:15:15] !log [void@puppet181] Upgraded packages on mwtask151: libinput-bin and libinput10 [01:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:36] !log [void@puppet181] Upgraded packages on mw202: libinput-bin and libinput10 [01:15:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:02] !log [void@puppet181] Upgraded packages on mw203: libinput-bin and libinput10 [01:16:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:21] !log [void@puppet181] Upgraded packages on mw201: libinput-bin and libinput10 [01:16:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:25] RECOVERY - mw193 APT on mw193 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:16:29] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:16:32] RECOVERY - mw201 APT on mw201 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:16:36] RECOVERY - mw202 APT on mw202 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:16:39] !log [void@puppet181] Upgraded packages on mwtask181: libinput-bin and libinput10 [01:16:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:55] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [01:16:59] !log [void@puppet181] Upgraded packages on test151: libinput-bin and libinput10 [01:17:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:17:16] RECOVERY - mw203 APT on mw203 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [01:17:50] RECOVERY - test151 APT on test151 is OK: APT OK: 118 packages available for upgrade (0 critical updates). [01:19:03] !log [void@puppet181] Upgraded packages on puppet181: libdbi-perl [01:19:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:12] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [01:21:13] !log [void@puppet181] Upgraded packages on kafka182: libjackson2-core-java and libjackson2-databind-java [01:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:47] RECOVERY - kafka182 APT on kafka182 is OK: APT OK: 77 packages available for upgrade (0 critical updates). [01:28:01] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 6.98, 5.85, 5.18 [01:30:01] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.02, 5.90, 5.28 [01:36:42] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.36, 17.77, 14.35 [01:37:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.93, 18.88, 14.28 [01:39:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.39, 17.34, 14.28 [01:40:42] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.25, 18.43, 15.49 [02:55:46] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 6.65, 5.79 [02:56:01] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.27, 6.50, 5.81 [02:57:46] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.96, 7.04, 6.04 [03:01:46] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.00, 5.60, 5.74 [03:02:01] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 2.45, 5.05, 5.49 [03:35:53] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.98, 21.04, 18.36 [03:37:47] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 14.62, 19.02, 17.97 [03:56:22] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.08, 6.13, 4.69 [03:58:19] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.56, 5.84, 4.76 [04:02:12] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.30, 6.80, 5.40 [04:04:09] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.25, 5.52, 5.10 [04:08:34] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [04:34:38] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:42] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.79, 19.41, 16.84 [04:38:42] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.93, 19.42, 17.17 [07:49:54] [02mw-config] 07whostacking opened pull request #6439: Change copyright icon for fischwiki (T15514) (07miraheze:03main...07whostacking:03fischy) 13https://github.com/miraheze/mw-config/pull/6439 [07:50:52] miraheze/mw-config - whostacking the build passed. [07:51:10] [02mw-config] 07whostacking retitled PR #6439: "Change copyright icon for fischwiki (T15514)" ➜ "T15514: Change copyright icon for fischwiki " 13https://github.com/miraheze/mw-config/pull/6439 [11:55:46] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.61, 13.85, 11.87 [11:57:46] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 12.67, 13.33, 11.93 [12:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f99139be89497de5fd47691011ff87c4c88c15d7 [12:30:21] 02ssl/03main 07WikiTideBot 03f99139b Bot: Auto-update domain lists [13:12:12] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.94, 15.95, 14.21 [13:14:11] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.77, 15.29, 14.20 [13:26:06] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.43, 14.63, 14.16 [13:28:05] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.29, 14.60, 14.20 [13:30:04] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 19.04, 16.18, 14.82 [13:32:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.30, 15.45, 14.73 [13:34:02] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.35, 15.79, 14.93 [13:38:00] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.13, 15.04, 14.94 [13:41:59] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.64, 15.68, 15.15 [13:45:57] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.02, 14.91, 15.02 [13:57:52] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.44, 14.14, 14.22 [13:59:51] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.02, 13.24, 13.89 [14:01:50] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 10.91, 12.49, 13.54 [14:03:07] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [14:07:46] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.74, 14.15, 14.03 [14:15:43] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.10, 12.70, 13.40 [14:23:39] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.39, 14.39, 13.72 [14:25:38] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.34, 14.16, 13.73 [14:27:37] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.75, 15.12, 14.13 [14:31:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.63, 14.05, 14.01 [14:35:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.47, 14.20, 13.95 [14:39:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.04, 15.17, 14.52 [14:53:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.47, 14.80, 14.20 [14:57:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.57, 14.55, 14.32 [15:01:07] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [15:01:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.76, 15.96, 14.90 [15:15:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.61, 15.49, 15.91 [15:23:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.38, 14.24, 14.78 [15:27:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.91, 14.59, 14.83 [15:31:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 19.02, 15.64, 15.08 [15:33:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.62, 14.92, 14.90 [15:35:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.81, 15.36, 15.04 [15:37:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.72, 14.82, 14.92 [15:39:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.65, 16.25, 15.43 [15:41:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 9.97, 13.85, 14.65 [15:43:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.63, 14.89, 14.92 [15:49:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.81, 15.34, 15.13 [15:51:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.80, 16.24, 15.48 [15:55:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.33, 14.29, 14.87 [15:59:36] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.08, 15.18, 15.07 [16:01:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.17, 14.67, 14.86 [16:03:36] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.16, 15.18, 15.03 [16:11:38] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.62, 15.95, 15.83 [16:19:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.72, 15.42, 15.21 [16:23:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.88, 14.53, 15.02 [16:33:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.68, 15.99, 15.28 [16:51:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.51, 14.98, 15.77 [16:53:36] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.18, 15.72, 15.93 [16:55:51] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.87, 14.70, 15.54 [16:57:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.68, 15.90, 15.90 [17:01:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.65, 15.24, 15.73 [17:05:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.63, 15.96, 15.81 [17:10:02] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.78, 14.76, 15.45 [17:13:36] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.21, 15.13, 15.37 [17:15:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.41, 15.51, 15.50 [17:17:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.74, 16.32, 15.80 [17:21:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 9.78, 14.23, 15.19 [17:27:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.65, 15.10, 15.22 [17:33:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.51, 15.66, 15.59 [17:34:42] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.98, 18.13, 15.71 [17:35:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.58, 16.59, 15.92 [17:38:42] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.40, 19.30, 16.87 [17:39:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.43, 15.38, 15.67 [17:41:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.12, 16.32, 15.97 [17:42:22] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.17, 19.12, 15.89 [17:44:16] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.77, 19.03, 16.27 [17:49:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.27, 15.64, 15.99 [17:51:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.56, 16.54, 16.28 [17:53:36] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 10.99, 14.38, 15.52 [17:56:26] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1781283350000&orgId=1&to=1781286986677 [17:59:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.26, 15.49, 15.53 [18:01:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.91, 14.20, 15.04 [18:07:35] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 19.01, 15.43, 15.13 [18:09:35] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.73, 13.98, 14.66 [18:11:26] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1781284190000&orgId=1&to=1781287790000 [18:13:36] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 9.18, 11.61, 13.56 [18:21:46] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 9.10, 7.64, 5.95 [18:22:01] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.68, 6.76, 5.85 [18:23:46] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.88, 7.40, 6.08 [18:25:46] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 7.71, 6.35 [18:27:46] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.80, 7.12, 6.29 [18:31:46] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 3.36, 5.97, 6.08 [18:32:01] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 4.31, 6.17, 6.21 [18:42:01] PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.73, 6.40, 6.00 [18:46:01] RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 5.15, 6.19, 6.05 [18:50:27] PROBLEM - cloud21 APT on cloud21 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [18:50:42] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [18:52:29] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [18:52:47] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [18:55:39] PROBLEM - cloud19 APT on cloud19 is CRITICAL: APT CRITICAL: 135 packages available for upgrade (1 critical updates). [18:58:05] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [18:59:28] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:00:15] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:01:38] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:02:10] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [19:03:00] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 119 packages available for upgrade (2 critical updates). [19:03:25] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 119 packages available for upgrade (2 critical updates). [19:04:28] PROBLEM - cloud20 APT on cloud20 is CRITICAL: APT CRITICAL: 135 packages available for upgrade (1 critical updates). [19:04:54] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:05:29] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [19:06:24] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:06:26] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:07:12] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:07:25] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.09, 13.32, 12.59 [19:07:44] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (1 critical updates). [19:09:24] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 10.79, 12.60, 12.43 [19:09:50] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:10:40] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:11:17] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:11:59] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:12:15] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 118 packages available for upgrade (2 critical updates). [19:13:41] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:14:34] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:14:41] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:15:08] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:18:47] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 135 packages available for upgrade (1 critical updates). [19:20:17] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 117 packages available for upgrade (2 critical updates). [19:37:12] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.08, 13.70, 12.86 [19:39:10] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 10.34, 12.94, 12.71 [19:42:30] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 16.61, 8.93, 4.74 [19:58:30] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 3.58, 9.71, 9.65 [19:59:58] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.55, 15.29, 13.88 [20:21:48] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 8.85, 14.64, 15.53 [20:27:46] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 7.06, 9.48, 12.90 [21:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/46e0a26e9ae5e3ebca9a8e0a33da5212f148dd84 [21:30:19] 02ssl/03main 07WikiTideBot 0346e0a26 Bot: Auto-update domain lists [23:05:33] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44037MiB (9.7% inode=100%); [23:15:33] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86176MiB (19.0% inode=100%); [23:25:21] !log [void@puppet181] Upgraded packages on cloud21: librabbitmq4 [23:25:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:37] !log [void@puppet181] Upgraded packages on cloud15: librabbitmq4 [23:25:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:53] !log [void@puppet181] Upgraded packages on cloud16: librabbitmq4 [23:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:05] RECOVERY - cloud21 APT on cloud21 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [23:26:11] !log [void@puppet181] Upgraded packages on cloud17: librabbitmq4 [23:26:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:23] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 134 packages available for upgrade (0 critical updates). [23:26:26] !log [void@puppet181] Upgraded packages on cloud19: librabbitmq4 [23:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:45] !log [void@puppet181] Upgraded packages on mw153: libjxl0.11 and librabbitmq4 [23:26:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:00] !log [void@puppet181] Upgraded packages on cloud20: librabbitmq4 [23:27:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:20] !log [void@puppet181] Upgraded packages on cloud18: librabbitmq4 [23:27:23] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [23:27:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:35] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 134 packages available for upgrade (0 critical updates). [23:27:37] !log [void@puppet181] Upgraded packages on mw151: libjxl0.11 and librabbitmq4 [23:27:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:44] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [23:27:45] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45243MiB (10.0% inode=100%); [23:27:57] RECOVERY - mw153 APT on mw153 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:27:57] !log [void@puppet181] Upgraded packages on mw152: libjxl0.11 and librabbitmq4 [23:28:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:05] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [23:28:16] !log [void@puppet181] Upgraded packages on mw161: libjxl0.11 and librabbitmq4 [23:28:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:22] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 134 packages available for upgrade (0 critical updates). [23:28:35] !log [void@puppet181] Upgraded packages on mw173: libjxl0.11 and librabbitmq4 [23:28:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:53] !log [void@puppet181] Upgraded packages on mw191: libjxl0.11 and librabbitmq4 [23:28:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:06] RECOVERY - mw173 APT on mw173 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:29:11] RECOVERY - mw152 APT on mw152 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:29:16] !log [void@puppet181] Upgraded packages on mw162: libjxl0.11 and librabbitmq4 [23:29:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:20] RECOVERY - mw151 APT on mw151 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:29:35] !log [void@puppet181] Upgraded packages on mw172: libjxl0.11 and librabbitmq4 [23:29:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:43] RECOVERY - mw162 APT on mw162 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:29:53] !log [void@puppet181] Upgraded packages on mw171: libjxl0.11 and librabbitmq4 [23:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:07] RECOVERY - mw161 APT on mw161 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:30:13] !log [void@puppet181] Upgraded packages on mw163: libjxl0.11 and librabbitmq4 [23:30:15] RECOVERY - mw172 APT on mw172 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:30:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:32] !log [void@puppet181] Upgraded packages on mw202: libjxl0.11 and librabbitmq4 [23:30:35] RECOVERY - mw171 APT on mw171 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:30:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:41] RECOVERY - mw191 APT on mw191 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:30:56] !log [void@puppet181] Upgraded packages on mw201: libjxl0.11 and librabbitmq4 [23:30:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:12] RECOVERY - mw163 APT on mw163 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:31:20] !log [void@puppet181] Upgraded packages on mw192: libjxl0.11 and librabbitmq4 [23:31:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:39] !log [void@puppet181] Upgraded packages on mw182: libjxl0.11 and librabbitmq4 [23:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:00] !log [void@puppet181] Upgraded packages on mw203: libjxl0.11 and librabbitmq4 [23:32:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:10] RECOVERY - mw201 APT on mw201 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:32:20] !log [void@puppet181] Upgraded packages on mw193: libjxl0.11 and librabbitmq4 [23:32:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:36] RECOVERY - mw202 APT on mw202 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:32:48] !log [void@puppet181] Upgraded packages on mw183: libjxl0.11 and librabbitmq4 [23:33:01] RECOVERY - mw192 APT on mw192 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:33:08] !log [void@puppet181] Upgraded packages on mw181: libjxl0.11 and librabbitmq4 [23:33:09] RECOVERY - mw183 APT on mw183 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:33:12] RECOVERY - mw181 APT on mw181 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:33:15] RECOVERY - mw182 APT on mw182 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:27] !log [void@puppet181] Upgraded packages on mwtask151: libjxl0.11 and librabbitmq4 [23:33:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:44] RECOVERY - mw203 APT on mw203 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:33:45] !log [void@puppet181] Upgraded packages on mwtask171: libjxl0.11 and librabbitmq4 [23:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:04] !log [void@puppet181] Upgraded packages on mwtask181: libjxl0.11 and librabbitmq4 [23:34:04] RECOVERY - mw193 APT on mw193 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:23] !log [void@puppet181] Upgraded packages on mwtask161: libjxl0.11 and librabbitmq4 [23:34:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:29] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:34:42] !log [void@puppet181] Upgraded packages on test151: libjxl0.11 and librabbitmq4 [23:34:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:55] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [23:35:14] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 116 packages available for upgrade (0 critical updates). [23:35:34] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [23:36:10] RECOVERY - test151 APT on test151 is OK: APT OK: 117 packages available for upgrade (0 critical updates).