[00:21:07] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 44198MiB (9.7% inode=100%); [00:29:07] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 87241MiB (19.2% inode=100%); [00:36:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44541MiB (9.8% inode=100%); [00:38:51] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 87853MiB (19.3% inode=100%); [00:40:59] return of the king [00:59:50] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44043MiB (9.7% inode=100%); [01:05:50] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 87092MiB (19.2% inode=100%); [02:50:21] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.29, 19.87, 17.03 [02:50:30] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.45, 20.13, 16.59 [02:51:29] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 20.74, 18.37, 15.18 [02:52:18] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.09, 18.81, 16.98 [02:52:29] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 12.67, 17.90, 16.24 [02:53:29] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 17.67, 17.79, 15.35 [03:01:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783044050000&orgId=1&to=1783047680034 [03:03:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.49, 19.10, 16.63 [03:03:58] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.05, 20.36, 18.51 [03:04:34] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.13, 20.17, 17.59 [03:05:03] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.96, 20.12, 18.15 [03:05:53] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.43, 18.51, 16.75 [03:06:52] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.09, 11.07, 8.66 [03:07:00] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.72, 19.56, 18.15 [03:08:34] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.23, 19.40, 17.84 [03:10:50] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.95, 12.46, 9.74 [03:11:00] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.77, 21.27, 19.18 [03:12:03] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.16, 21.14, 18.74 [03:12:34] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.47, 20.19, 18.54 [03:12:55] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 21.12, 20.02, 18.01 [03:13:09] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.28, 19.57, 17.19 [03:13:16] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.33, 21.02, 19.22 [03:13:29] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.92, 20.30, 18.25 [03:13:44] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.10, 20.35, 18.60 [03:15:09] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.79, 19.45, 17.45 [03:17:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.64, 20.11, 18.48 [03:19:00] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.97, 20.75 [03:19:09] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.97, 20.77, 18.44 [03:21:00] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.05, 22.98, 21.03 [03:21:31] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.86, 20.25, 19.47 [03:21:53] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.27, 19.44, 18.64 [03:22:34] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.53, 20.10, 19.67 [03:22:34] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 17.92, 20.00, 19.14 [03:23:10] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.81, 18.85, 18.37 [03:23:11] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 15.55, 19.45, 19.65 [03:23:29] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.02, 19.32, 19.10 [03:23:45] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 16.01, 19.53, 20.39 [03:23:50] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 16.94, 19.60, 19.54 [03:25:00] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.28, 19.40, 20.00 [03:36:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783045940000&orgId=1&to=1783049540000 [03:40:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 2.66, 6.84, 11.61 [03:44:35] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.24, 4.53, 9.55 [05:02:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783051310000&orgId=1&to=1783054940037 [05:07:45] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.02, 19.63, 17.39 [05:08:40] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.77, 18.90, 16.80 [05:09:03] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.67, 18.84, 16.84 [05:10:14] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 21.45, 19.31, 17.01 [05:12:14] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 20.07, 19.56, 17.39 [05:12:23] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 17.50, 12.78, 8.44 [05:13:00] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.69, 20.73, 18.49 [05:13:59] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.72, 19.89, 17.26 [05:14:23] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.92, 11.16, 8.38 [05:15:12] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.53, 21.05, 19.07 [05:15:25] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.64, 20.51, 18.54 [05:15:56] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.86, 18.92, 17.22 [05:16:14] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.94, 20.79, 18.32 [05:16:23] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.41, 21.11, 18.27 [05:16:23] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 8.61, 10.16, 8.34 [05:17:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.50, 20.35, 17.87 [05:17:11] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 17.26, 20.29, 19.08 [05:18:40] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.22, 20.33, 18.73 [05:19:01] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.48, 19.31, 17.81 [05:19:45] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.28, 20.16, 19.24 [05:20:14] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 16.78, 19.80, 18.57 [05:20:21] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 16.12, 19.14, 18.20 [05:21:00] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.13, 19.54, 19.18 [05:21:03] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 14.90, 18.34, 18.25 [05:21:19] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.46, 18.73, 18.54 [05:22:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783051310000&orgId=1&to=1783056020000 [05:40:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.28, 19.61, 18.14 [05:40:40] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.25, 20.44, 19.18 [05:41:45] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.41, 21.12, 19.66 [05:41:52] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.77, 20.84, 19.58 [05:41:57] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 23.29, 21.04, 19.68 [05:42:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.82, 20.54, 18.63 [05:42:37] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.97, 19.75, 17.16 [05:42:47] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 23.51, 21.65, 19.64 [05:43:00] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.59, 21.33, 19.85 [05:43:03] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.77, 20.93, 19.33 [05:43:05] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.50, 21.08, 19.29 [05:44:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.24, 20.40, 18.83 [05:44:35] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.68, 19.02, 17.18 [05:48:09] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.52, 20.37, 18.82 [05:49:44] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.15, 20.20, 19.92 [05:51:00] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 21.95, 20.32 [05:51:07] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.15, 20.11, 18.17 [05:51:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783054250000&orgId=1&to=1783057880033 [05:52:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.30, 20.32, 18.42 [05:52:59] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.58, 22.44, 20.70 [05:53:13] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.03, 21.67, 20.01 [05:53:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.84, 21.76, 20.62 [05:55:52] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 21.51, 20.84, 18.62 [05:57:50] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 18.88, 20.18, 18.65 [05:58:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.90, 21.14, 18.59 [06:00:56] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.75, 19.41, 19.18 [06:02:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.17, 18.96, 18.36 [06:03:46] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 24.54, 22.86, 21.70 [06:04:27] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.36, 19.06, 19.21 [06:05:45] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 13.75, 19.86, 20.77 [06:05:45] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.17, 18.34, 19.76 [06:06:14] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 16.15, 19.60, 20.34 [06:06:22] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.28, 18.91, 20.22 [06:07:00] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 15.70, 18.47, 20.01 [06:07:22] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.18, 18.54, 20.14 [06:07:44] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 17.60, 19.15, 20.39 [06:08:40] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.97, 19.38, 20.25 [06:08:40] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.54, 18.92, 20.13 [06:09:03] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.99, 18.87, 20.29 [06:10:48] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 17.99, 19.12, 20.22 [06:13:45] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.70, 18.70, 20.01 [06:21:20] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.24, 10.72, 8.37 [06:25:18] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.33, 11.95, 9.56 [06:27:17] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.04, 9.48, 8.95 [06:47:07] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.66, 12.08, 9.72 [06:51:05] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 7.81, 10.67, 9.77 [06:55:03] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 9.02, 9.45, 9.47 [07:09:33] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.14, 20.07, 19.10 [07:10:13] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.10, 18.66, 18.17 [07:10:51] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 20.54, 18.71, 18.23 [07:11:35] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.36, 20.23, 19.66 [07:12:12] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.82, 18.82, 18.28 [07:13:21] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.46, 20.66, 19.78 [07:13:57] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.87, 20.33, 19.73 [07:18:07] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.12, 20.96, 19.34 [07:18:09] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.17, 21.31, 20.09 [07:18:21] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.82, 19.97, 18.20 [07:18:31] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.29, 21.21, 19.97 [07:19:34] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.24, 20.66, 19.57 [07:19:53] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.93, 20.65, 19.22 [07:20:21] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.20, 20.70, 18.78 [07:20:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.56, 20.67, 18.37 [07:20:30] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 18.58, 20.25, 19.47 [07:21:12] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.12, 19.17, 17.11 [07:21:33] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 21.75, 19.85, 17.63 [07:23:34] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 21.79, 20.01, 18.02 [07:24:24] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 23.30, 21.76, 20.25 [07:25:13] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.80, 19.95, 17.52 [07:26:00] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.89, 20.26, 18.46 [07:26:12] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 22.55, 20.01 [07:26:19] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 24.02, 23.07, 21.30 [07:27:11] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.20, 19.95, 17.82 [07:28:09] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.57, 22.77, 20.40 [07:28:15] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.02, 23.05, 21.50 [07:29:12] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.61, 20.16, 18.66 [07:29:33] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 19.50, 19.94, 18.66 [07:29:34] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 19.24, 20.24, 18.90 [07:29:48] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.09, 19.52, 18.58 [07:32:44] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.39, 10.19, 8.59 [07:33:34] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.29, 21.05, 19.51 [07:33:36] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.55, 20.80, 19.27 [07:35:00] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.71, 21.03, 19.12 [07:35:30] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.65, 20.22, 19.26 [07:35:45] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 23.42, 22.17 [07:36:27] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.43, 20.15, 19.87 [07:36:42] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.34, 10.63, 9.19 [07:36:58] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.79, 18.81, 18.57 [07:37:34] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 12.07, 18.00, 18.79 [07:37:45] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 13.52, 20.11, 21.14 [07:37:53] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.25, 18.89, 20.31 [07:37:57] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 11.28, 17.64, 19.41 [07:37:58] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 10.83, 16.98, 19.04 [07:38:14] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 11.75, 18.55, 20.30 [07:38:41] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 8.29, 9.85, 9.08 [07:38:52] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 9.63, 16.98, 19.69 [07:39:00] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 10.21, 17.09, 19.85 [07:39:00] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 11.97, 17.97, 20.36 [07:39:28] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 11.35, 17.09, 19.75 [07:39:35] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 10.24, 16.45, 19.49 [07:39:45] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 11.30, 16.98, 19.86 [07:39:54] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 10.43, 16.18, 19.21 [07:39:56] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 11.87, 17.26, 19.92 [07:41:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1783060850000&orgId=1&to=1783064450000 [09:30:17] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3b473c9a818f08570e8a967dd686a045e8cbb708 [09:30:17] 02ssl/03main 07WikiTideBot 033b473c9 Bot: Auto-update domain lists [09:56:52] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=envidrazwikiwiki --new=indrawikiwiki --user=Reception123 (END - exit=256) [09:56:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:59:15] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=envidrazwiki --new=indrawiki --user=Reception123 (END - exit=0) [09:59:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:01:43] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=officalcorruptedcrossroadswiki --new=officialcorruptedcrossroadswiki --user=Reception123 (END - exit=0) [10:01:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:25] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=sonarwiki --new=musicapediawiki --user=Reception123 (END - exit=0) [10:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:08:56] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=lexingtonandconcordwiki --new=lncwiki --user=Reception123 (END - exit=0) [10:09:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:23] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 31.05, 18.12, 10.28 [13:44:23] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 4.66, 10.51, 9.65 [13:46:23] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 4.17, 8.41, 8.98 [14:19:41] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.57, 20.05, 16.27 [14:21:34] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 11.83, 17.00, 15.58 [19:30:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/aa6affcdf6730820a21bafe31d05689c844ae867 [19:30:19] 02ssl/03main 07WikiTideBot 03aa6affc Bot: Auto-update domain lists [20:42:36] PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui] [21:08:07] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/d34cbe322c9427d4857fe16179449013a0d2be12 [21:08:08] 02mw-config/03main 07Peter Li 03d34cbe3 T15558: move robots.php from puppet to mw-config (#6446) [21:08:08] [02mw-config] 07lihaohong6 merged pull request #6446: T15558: move robots.php from puppet to mw-config (07miraheze:03main...07lihaohong6:03robots2) 13https://github.com/miraheze/mw-config/pull/6446 [21:09:01] !log [petramagna@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [21:09:02] !log [petramagna@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [21:09:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:19] miraheze/mw-config - lihaohong6 the build passed. [21:10:36] RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [21:13:18] !log [petramagna@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:13:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:45] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [21:13:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3232b16105c1fa6b5159f77484e66e90d09d6a39 [23:00:17] 02ssl/03main 07WikiTideBot 033232b16 Bot: Auto-update domain lists [23:13:07] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45258MiB (10.0% inode=100%); [23:24:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44320MiB (9.8% inode=100%); [23:33:50] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45268MiB (10.0% inode=100%); [23:34:51] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 87165MiB (19.2% inode=100%); [23:53:50] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86171MiB (19.0% inode=100%);