[00:04:47] PROBLEM - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit update-static-tech-docs [00:13:18] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 75655MiB (16% inode=99%); [00:18:18] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 67612MiB (14% inode=99%); [00:52:10] RECOVERY - cp171 health.wikitide.net HTTPS on cp171 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 9.375 second response time [00:56:32] PROBLEM - cp171 health.wikitide.net HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:33:27] what is a health.wikitied.net [01:55:57] idk [01:56:07] i know what a health.wikitide.net is tho [01:56:28] https://tenor.com/view/squid-game-oh-my-god-bruh-saleswoman-oh-my-god-bruh-gif-2398788864743270124 [03:11:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.30, 17.22, 11.19 [03:11:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.70, 17.15, 11.48 [03:11:55] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.54, 17.05, 11.19 [03:12:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.08, 18.35, 11.89 [03:13:28] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.26, 18.91, 12.65 [03:13:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.09, 19.09, 12.95 [03:13:57] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.16, 19.20, 12.93 [03:14:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.95, 18.40, 12.70 [03:15:55] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 18.22, 18.94, 13.36 [03:17:19] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.85, 19.93, 14.48 [03:17:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.10, 19.68, 14.67 [03:17:50] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.06, 18.75, 14.23 [03:19:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.91, 19.82, 15.06 [03:25:38] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.09, 20.88, 16.75 [03:26:56] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.16, 20.87, 17.38 [03:29:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.88, 19.81, 17.09 [03:33:23] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.29, 20.28, 18.23 [03:35:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762742090000&orgId=1&to=1762745720038 [03:37:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.33, 20.30, 18.71 [03:38:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 14.19, 18.73, 18.57 [03:39:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.16, 17.85, 18.77 [03:45:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762742630000&orgId=1&to=1762746230000 [04:05:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.41, 17.09, 14.47 [04:06:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.42, 18.39, 15.61 [04:07:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.35, 18.21, 15.10 [04:07:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.41, 19.22, 16.57 [04:07:25] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.98, 17.09, 13.65 [04:07:36] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.67, 16.64, 12.89 [04:07:55] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.02, 18.15, 14.83 [04:08:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.39, 18.82, 15.13 [04:09:22] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.17, 17.64, 14.28 [04:09:36] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.28, 16.80, 13.37 [04:11:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.13, 19.54, 15.95 [04:11:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.84, 19.91, 16.14 [04:13:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.18, 18.89, 16.16 [04:13:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.33, 19.20, 16.34 [04:13:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.30, 20.27, 17.26 [04:17:11] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.78, 20.41, 16.71 [04:19:09] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.65, 19.64, 16.87 [04:19:55] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.49, 20.26, 18.31 [04:20:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.87, 20.25, 18.17 [04:21:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.78, 21.23, 18.31 [04:26:35] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 16.62, 19.16, 18.19 [04:26:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.85, 19.91, 19.52 [04:27:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.54, 20.30, 19.30 [04:30:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762745390000&orgId=1&to=1762749020041 [04:32:32] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.58, 10.19, 7.77 [04:35:24] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.61, 20.83, 19.95 [04:35:47] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.88, 11.12, 8.71 [04:36:31] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.13, 20.55, 18.94 [04:38:05] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.65, 20.95, 18.90 [04:39:14] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 17.21, 20.12, 19.95 [04:39:45] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 3.57, 8.19, 8.19 [04:40:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.46, 19.80, 19.09 [04:41:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.68, 19.24, 20.30 [04:41:57] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 15.26, 19.29, 18.78 [04:55:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762746710000&orgId=1&to=1762750310000 [05:20:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.43, 18.19, 15.25 [05:22:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.40, 17.81, 15.50 [05:38:14] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=eliminationwiki --startOver (END - exit=2) [05:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:49] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=eliminationwiki --startOver (END - exit=0) [05:39:18] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=eliminationwiki (END - exit=0) [05:39:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:39:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:22] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.50, 10.28, 7.35 [06:30:16] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.55, 8.53, 7.54 [06:33:06] RECOVERY - cp171 health.wikitide.net HTTPS on cp171 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 6.120 second response time [06:37:28] PROBLEM - cp171 health.wikitide.net HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:42:24] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [06:44:24] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 27% [06:49:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.38, 17.69, 14.71 [06:51:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 16.15, 17.33, 14.97 [06:57:24] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.94, 19.59, 16.60 [06:59:20] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 14.68, 17.43, 16.15 [07:01:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762754450000&orgId=1&to=1762758080043 [07:16:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762754450000&orgId=1&to=1762758920000 [07:36:18] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49760MiB (10% inode=99%); [07:45:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.65, 18.49, 16.32 [07:47:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.19, 18.68, 16.68 [07:57:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.33, 19.70, 17.80 [07:59:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 16.79, 18.85, 17.74 [08:01:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.50, 20.55, 18.18 [08:03:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.79, 20.13, 18.34 [08:08:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762758470000&orgId=1&to=1762762100043 [08:17:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.75, 18.60, 17.36 [08:19:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.21, 18.53, 17.47 [08:33:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762758470000&orgId=1&to=1762763300000 [08:34:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.70, 19.95, 18.37 [08:36:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.19, 17.99, 17.84 [09:00:38] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.44, 10.44, 8.94 [09:01:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.34, 19.05, 17.65 [09:01:25] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.00, 19.69, 17.80 [09:02:32] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 9.87, 10.15, 9.00 [09:04:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.93, 20.31, 18.28 [09:05:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.11, 20.04, 18.42 [09:05:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.87, 19.74, 18.33 [09:06:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.17, 19.76, 18.33 [09:18:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762762700000&orgId=1&to=1762766330035 [09:23:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762762970000&orgId=1&to=1762766570000 [10:05:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.06, 18.70, 16.22 [10:07:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.54, 18.80, 16.58 [10:07:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.51, 18.79, 16.41 [10:09:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.17, 18.39, 16.54 [10:13:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 10.26, 9.02, 7.92 [10:14:34] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49861MiB (10% inode=99%); [10:15:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.46, 20.07, 17.99 [10:15:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 9.92, 9.57, 8.28 [10:17:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.23, 18.70, 17.73 [10:53:43] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.34, 19.64, 18.21 [10:55:39] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.39, 19.58, 18.35 [11:02:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762768940000&orgId=1&to=1762772570037 [11:12:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762768940000&orgId=1&to=1762772960000 [11:15:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 15.45, 12.90, 9.43 [11:17:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.64, 11.14, 9.21 [11:19:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 6.20, 9.47, 8.83 [11:25:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.75, 19.01, 17.20 [11:27:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.56, 18.72, 16.51 [11:29:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.86, 19.82, 17.86 [11:29:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 20.24, 19.33, 17.02 [11:29:40] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.99, 20.04, 18.08 [11:31:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.01, 19.48, 17.99 [11:40:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762771190000&orgId=1&to=1762774820043 [11:43:19] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.50, 20.12, 18.55 [11:45:14] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.36, 19.49, 18.50 [11:48:45] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 22.29, 14.49, 10.53 [11:50:42] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.05, 11.29, 9.85 [11:52:39] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.64, 8.68, 9.06 [11:54:25] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:54:26] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:54:28] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:31] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:54:47] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:49] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:56] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [11:54:58] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [11:55:01] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:55:03] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:55:04] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:55:10] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [11:55:10] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [11:55:13] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [11:55:13] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:55:14] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:18] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:55:20] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [11:56:13] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:25] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:32] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:33] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:38] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:40] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:56:42] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:42] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:56:49] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [11:56:50] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [11:56:53] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:54] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:56:56] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:56:56] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [11:56:57] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:58] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [11:56:59] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:57:00] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:57:25] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:57:27] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:57:32] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [11:57:37] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:57:54] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [11:58:11] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.058 second response time [11:58:27] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4328 bytes in 2.116 second response time [11:58:29] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.553 second response time [11:58:30] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 2.432 second response time [11:58:35] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.661 second response time [11:58:35] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.482 second response time [11:58:39] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.804 second response time [11:58:40] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 3.797 second response time [11:58:43] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.401 second response time [11:58:43] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.216 second response time [11:58:46] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.228 second response time [11:58:46] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.530 second response time [11:58:49] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.213 second response time [11:58:51] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.061 second response time [11:58:51] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.996 second response time [11:58:53] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.580 second response time [11:58:53] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.531 second response time [11:58:54] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.292 second response time [11:58:55] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 1.123 second response time [11:58:55] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [11:58:56] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.039 second response time [11:58:58] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.285 second response time [11:58:59] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.058 second response time [11:59:00] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.294 second response time [11:59:01] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 27% [11:59:01] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.345 second response time [11:59:02] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [11:59:03] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.045 second response time [11:59:03] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.781 second response time [11:59:04] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.283 second response time [11:59:13] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.065 second response time [11:59:19] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.066 second response time [11:59:19] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.263 second response time [11:59:21] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.342 second response time [11:59:24] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.064 second response time [11:59:25] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.385 second response time [11:59:31] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.302 second response time [11:59:33] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.055 second response time [11:59:53] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.281 second response time [12:00:13] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.047 second response time [12:00:14] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 0.063 second response time [12:00:19] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4273 bytes in 0.066 second response time [12:01:54] !log restarted nginx + php-fpm on mw* [12:02:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:02] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [12:11:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.04, 19.04, 15.97 [12:15:43] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.78, 18.40, 16.51 [12:21:55] [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/ebde8b30c980302d2d1578066b3203937b9b71dd [12:21:55] 02CreateWiki/03main 07translatewiki.net 03ebde8b3 Localisation updates from https://translatewiki.net. [12:21:56] [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/41f4f1acf9901d0049310e19525b44b45778f1a9 [12:21:56] 02landing/03main 07translatewiki.net 0341f4f1a Localisation updates from https://translatewiki.net. [12:22:47] miraheze/landing - translatewiki the build passed. [12:25:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762771190000&orgId=1&to=1762777430000 [12:43:18] miraheze/CreateWiki - translatewiki the build passed. [14:25:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 20.15, 11.61, 7.69 [14:29:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.02, 11.73, 8.80 [14:31:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 6.48, 9.79, 8.43 [15:09:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762783730000&orgId=1&to=1762787360040 [15:13:51] [02mw-config] 07dependabot[bot] created 03dependabot/composer/justinrainbow/json-schema-6.6.1 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/6a41f0bd3a2c [15:13:51] 02mw-config/03dependabot/composer/justinrainbow/json-schema-6.6.1 07dependabot[bot] 036a41f0b Update justinrainbow/json-schema requirement from 6.6.0 to 6.6.1… [15:13:52] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6173 (Update justinrainbow/json-schema requirement from 6.6.0 to 6.6.1) 13https://github.com/miraheze/mw-config/pull/6173 [15:13:53] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6173 (Update justinrainbow/json-schema requirement from 6.6.0 to 6.6.1) 13https://github.com/miraheze/mw-config/pull/6173 [15:13:55] [02mw-config] 07dependabot[bot] opened pull request #6173: Update justinrainbow/json-schema requirement from 6.6.0 to 6.6.1 (03main...03dependabot/composer/justinrainbow/json-schema-6.6.1) 13https://github.com/miraheze/mw-config/pull/6173 [15:13:57] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6173 (Update justinrainbow/json-schema requirement from 6.6.0 to 6.6.1) 13https://github.com/miraheze/mw-config/pull/6173 [15:13:59] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6173 (Update justinrainbow/json-schema requirement from 6.6.0 to 6.6.1) 13https://github.com/miraheze/mw-config/pull/6173 [15:14:44] miraheze/mw-config - dependabot[bot] the build passed. [15:20:18] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27127MiB (5% inode=99%); [15:25:34] [02ImportDump] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 (+1 new commit) 13https://github.com/miraheze/ImportDump/commit/7dfcbac63805 [15:25:34] 02ImportDump/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 07dependabot[bot] 037dfcbac Bump eslint-config-wikimedia from 0.31.0 to 0.32.0… [15:25:35] [02ImportDump] 07dependabot[bot] added the label 'dependencies' to pull request #157 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/ImportDump/pull/157 [15:25:35] [02ImportDump] 07dependabot[bot] added the label 'javascript' to pull request #157 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/ImportDump/pull/157 [15:25:37] [02ImportDump] 07dependabot[bot] opened pull request #157: Bump eslint-config-wikimedia from 0.31.0 to 0.32.0 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0) 13https://github.com/miraheze/ImportDump/pull/157 [15:25:39] [02ImportDump] 07dependabot[bot] added the label 'dependencies' to pull request #157 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/ImportDump/pull/157 [15:25:41] [02ImportDump] 07dependabot[bot] added the label 'javascript' to pull request #157 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/ImportDump/pull/157 [15:25:52] [02ImportDump] 07coderabbitai[bot] commented on pull request #157: --- […] 13https://github.com/miraheze/ImportDump/pull/157#issuecomment-3512378036 [15:28:44] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.77, 11.81, 8.46 [15:30:41] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.54, 9.22, 7.91 [15:40:19] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.07, 20.15, 17.61 [15:42:07] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Video'} to all [15:42:15] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.10, 18.51, 17.28 [15:42:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:30] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Video'} to test151 [15:43:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:44:13] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Video'} to all - SUCCESS in 125s [15:44:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:44] miraheze/ImportDump - dependabot[bot] the build passed. [15:45:55] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Video'} to test151 - SUCCESS in 204s [15:46:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:41] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_skins': 'Cosmos'} to test151 [15:48:43] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_skins': 'Cosmos'} to all [15:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:45] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_skins': 'Cosmos'} to all - SUCCESS in 121s [15:50:46] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_skins': 'Cosmos'} to test151 - SUCCESS in 125s [15:51:06] [02ImportDump] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/81ff68dbfdf3c8d5393b025e9b8e6eb5a11765dc [15:51:07] 02ImportDump/03main 07dependabot[bot] 0381ff68d Bump eslint-config-wikimedia from 0.31.0 to 0.32.0 (#157)… [15:51:07] [02ImportDump] 07paladox merged 07dependabot[bot]'s pull request #157: Bump eslint-config-wikimedia from 0.31.0 to 0.32.0 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0) 13https://github.com/miraheze/ImportDump/pull/157 [15:51:08] [02ImportDump] 07paladox 04deleted 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 at 037dfcbac 13https://api.github.com/repos/miraheze/ImportDump/commit/7dfcbac [15:51:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:53] what updated [16:04:06] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_skins': 'Cosmos'} to all [16:04:07] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_skins': 'Cosmos'} to test151 [16:04:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:25] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_skins': 'Cosmos'} to all - SUCCESS in 79s [16:05:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:00] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_skins': 'Cosmos'} to test151 - SUCCESS in 112s [16:06:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:09] miraheze/ImportDump - paladox the build passed. [16:26:01] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.72, 20.16, 18.61 [16:27:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.94, 19.87, 18.38 [16:28:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.69, 19.97, 18.12 [16:29:02] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [16:29:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.25, 20.13, 18.66 [16:29:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.00, 18.40, 18.35 [16:32:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 17.72, 19.31, 18.35 [16:35:02] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [16:55:41] [02MediaWikiDebugJS] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 (+1 new commit) 13https://github.com/miraheze/MediaWikiDebugJS/commit/f2b1bfa0dd05 [16:55:41] 02MediaWikiDebugJS/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 07dependabot[bot] 03f2b1bfa Bump eslint-config-wikimedia from 0.31.0 to 0.32.0… [16:55:42] [02MediaWikiDebugJS] 07dependabot[bot] added the label 'dependencies' to pull request #27 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MediaWikiDebugJS/pull/27 [16:55:42] [02MediaWikiDebugJS] 07dependabot[bot] added the label 'javascript' to pull request #27 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MediaWikiDebugJS/pull/27 [16:55:44] [02MediaWikiDebugJS] 07dependabot[bot] opened pull request #27: Bump eslint-config-wikimedia from 0.31.0 to 0.32.0 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0) 13https://github.com/miraheze/MediaWikiDebugJS/pull/27 [16:55:46] [02MediaWikiDebugJS] 07dependabot[bot] added the label 'dependencies' to pull request #27 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MediaWikiDebugJS/pull/27 [16:55:48] [02MediaWikiDebugJS] 07dependabot[bot] added the label 'javascript' to pull request #27 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MediaWikiDebugJS/pull/27 [16:56:01] [02MediaWikiDebugJS] 07coderabbitai[bot] commented on pull request #27: --- […] 13https://github.com/miraheze/MediaWikiDebugJS/pull/27#issuecomment-3512824650 [16:58:18] [02MatomoAnalytics] 07dependabot[bot] created 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 (+1 new commit) 13https://github.com/miraheze/MatomoAnalytics/commit/c7da9f61c7bc [16:58:18] 02MatomoAnalytics/03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0 07dependabot[bot] 03c7da9f6 Bump eslint-config-wikimedia from 0.31.0 to 0.32.0… [16:58:19] [02MatomoAnalytics] 07dependabot[bot] added the label 'javascript' to pull request #172 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MatomoAnalytics/pull/172 [16:58:21] [02MatomoAnalytics] 07dependabot[bot] added the label 'dependencies' to pull request #172 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MatomoAnalytics/pull/172 [16:58:29] [02MatomoAnalytics] 07dependabot[bot] opened pull request #172: Bump eslint-config-wikimedia from 0.31.0 to 0.32.0 (03main...03dependabot/npm_and_yarn/eslint-config-wikimedia-0.32.0) 13https://github.com/miraheze/MatomoAnalytics/pull/172 [16:58:29] [02MatomoAnalytics] 07dependabot[bot] added the label 'dependencies' to pull request #172 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MatomoAnalytics/pull/172 [16:58:29] [02MatomoAnalytics] 07dependabot[bot] added the label 'javascript' to pull request #172 (Bump eslint-config-wikimedia from 0.31.0 to 0.32.0) 13https://github.com/miraheze/MatomoAnalytics/pull/172 [16:58:37] [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request #172: --- […] 13https://github.com/miraheze/MatomoAnalytics/pull/172#issuecomment-3512842996 [16:59:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.37, 20.60, 18.64 [17:01:18] miraheze/MatomoAnalytics - dependabot[bot] the build has errored. [17:01:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.97, 19.88, 18.60 [17:04:34] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27140MiB (5% inode=99%); [17:05:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.50, 21.10, 19.23 [17:07:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 15.38, 19.11, 18.74 [17:11:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.62, 19.97, 18.51 [17:13:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.15, 18.73, 18.26 [17:18:34] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 71171MiB (15% inode=99%); [17:24:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762791770000&orgId=1&to=1762795370000 [17:32:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.87, 18.85, 17.73 [17:34:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.07, 18.93, 17.93 [17:47:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762793240000&orgId=1&to=1762796873358 [17:48:45] PROBLEM - mattermost1 NTP time on mattermost1 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o