[00:01:29] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/8062c1001e2b4711af5ed5f461c03c7af4a24982 [00:01:29] 02statichelp/03main 07WikiTideBot 038062c10 Bot: Auto-update Tech namespace pages 2025-11-09 00:01:27 [00:16:50] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 65896MiB (14% inode=99%); [01:56:05] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:15] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [01:56:21] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:57:05] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44% [01:57:44] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [01:58:16] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4224 bytes in 2.866 second response time [01:58:20] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time [01:58:47] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [01:59:05] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 34% [01:59:40] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.202 second response time [02:00:03] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.738 second response time [02:06:47] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [03:03:41] PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 2241MiB (7% inode=80%); [03:07:41] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4509MiB (14% inode=80%); [03:31:43] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.01, 10.63, 5.40 [03:33:43] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.26, 7.42, 4.86 [03:45:43] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.60, 12.72, 7.31 [03:47:43] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.84, 8.84, 6.54 [05:47:02] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.47, 17.44, 10.63 [05:51:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.03, 18.94, 12.82 [05:57:56] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.83, 21.57, 16.14 [06:09:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 23.34, 19.82 [06:12:56] [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762665140000&orgId=1&to=1762668776684 [06:15:20] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.47, 23.97, 21.26 [06:17:16] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.73, 24.60, 21.80 [06:22:56] [Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762665500000&orgId=1&to=1762669100000 [06:47:02] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.46, 19.65, 23.49 [06:51:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.69, 12.49, 19.65 [09:01:03] [02mw-config] 07originalcloudcrafter un-readied pull request #6164: Add lawcommonswiki and lawpediawiki to GlobalUsage settings (07miraheze:03main...07originalcloudcrafter:03patch-1) 13https://github.com/miraheze/mw-config/pull/6164 [09:11:38] [02mw-config] 07originalcloudcrafter closed pull request #6164: Add lawcommonswiki and lawpediawiki to GlobalUsage settings (07miraheze:03main...07originalcloudcrafter:03patch-1) 13https://github.com/miraheze/mw-config/pull/6164 [09:17:13] [02mw-config] 07originalcloudcrafter drafted pull request #6172: Add microcommonswiki and related wikis to settings (07miraheze:03main...07originalcloudcrafter:03patch-1) 13https://github.com/miraheze/mw-config/pull/6172 [11:40:50] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49764MiB (10% inode=99%); [12:47:50] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.44/extensions/Math,1.45/extensions/Math'} to test151 [12:47:51] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.44/extensions/Math,1.45/extensions/Math'} to test151 - SUCCESS in 0s [12:47:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:48:46] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/Math'} to all [12:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:49:10] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/Math'} to all - SUCCESS in 23s [12:49:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:49:28] !log remove local patch for [[phab:T13808]] [12:49:29]

[12:49:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:50:01] !log (and upgraded the extension) [12:50:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:56] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49862MiB (10% inode=99%); [14:55:26] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49822MiB (10% inode=99%); [15:00:10] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.75, 16.14, 9.52 [15:02:05] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.62, 11.10, 8.45 [15:04:00] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.97, 7.79, 7.55 [15:23:01] [1/2] I'll help the first 10 people interested on how to start earning $100k or more within a week, but you will reimburse me 10% of your profits when you receive it. Note: only interested people should send a friend request or send me a dm! ask me (HOW) via Telegram username @Edward_Pryce1 [15:23:01] [2/2] Or use the telegram link in my bio [15:51:09] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.77, 10.46, 7.25 [15:53:09] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 8.77, 9.88, 7.43 [15:57:09] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 8.72, 10.95, 8.60 [15:59:09] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.75, 12.49, 9.47 [16:05:09] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 8.89, 11.99, 10.33 [16:07:57] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.78, 16.39, 11.64 [16:08:21] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.12, 16.43, 11.63 [16:08:24] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.60, 16.22, 11.37 [16:08:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.59, 17.27, 12.42 [16:08:45] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.61, 16.90, 12.01 [16:08:52] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.11, 16.54, 11.91 [16:09:09] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 4.45, 8.35, 9.24 [16:09:38] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 20.54, 16.40, 11.63 [16:10:22] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.27, 17.12, 11.83 [16:10:28] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:10:30] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.72, 17.60, 12.30 [16:10:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.29, 17.22, 11.80 [16:10:42] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.01, 18.17, 12.81 [16:10:55] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:11:43] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.43, 17.85, 12.15 [16:11:44] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.46, 17.81, 12.22 [16:11:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.53, 17.88, 12.20 [16:12:01] pls no [16:12:03] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 21.89, 18.14, 12.42 [16:12:22] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.68, 18.68, 12.90 [16:12:33] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.765 second response time [16:13:01] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.019 second response time [16:13:28] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.88, 19.57, 13.68 [16:13:47] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 21.32, 18.73, 13.02 [16:16:22] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.35, 20.11, 15.02 [16:16:30] [1/2] shoutout mw162 [16:16:30] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1437113631979339846/image.png?ex=69120fde&is=6910be5e&hm=b6e39b6281d72f87c479851127b564a458aaa228bdfeb8bf31a48a6f5bc20608& [16:17:22] nvm it exploded too [16:17:44] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.63, 19.83, 15.08 [16:17:47] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 18.51, 19.32, 14.62 [16:20:03] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.56, 13.78, 8.80 [16:20:15] miraheze/mw-config - originalcloudcrafter the build passed. [16:21:58] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.11, 11.65, 8.61 [16:23:53] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.92, 13.18, 9.42 [16:24:22] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.61, 20.13, 17.19 [16:28:12] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.80, 20.84, 18.02 [16:28:22] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.34, 21.07, 18.29 [16:29:43] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.32, 10.00, 9.41 [16:29:55] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.54, 20.17, 18.14 [16:30:03] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 19.65, 20.22, 18.15 [16:30:22] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.17, 19.91, 18.21 [16:31:43] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 19.04, 20.32, 18.50 [16:32:08] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.16, 20.25, 18.43 [16:32:43] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 20.38, 20.45, 18.14 [16:33:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.19, 21.35, 19.09 [16:34:03] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 20.03, 20.56, 18.79 [16:34:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762702430000&orgId=1&to=1762706060035 [16:34:42] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 14.22, 18.43, 17.70 [16:34:52] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 13.83, 19.17, 18.93 [16:35:38] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 15.82, 19.67, 19.23 [16:35:55] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.39, 18.17, 18.20 [16:35:58] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 12.37, 11.80, 9.50 [16:36:03] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 15.48, 18.30, 18.16 [16:36:30] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 19.59, 20.11, 19.40 [16:36:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.37, 19.97, 19.16 [16:36:42] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.32, 20.38, 19.61 [16:37:55] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 3.97, 8.90, 8.71 [16:39:14] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.69, 20.42, 19.03 [16:39:38] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.58, 20.99, 19.83 [16:39:44] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.98, 19.81, 18.14 [16:39:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.75, 20.15, 19.01 [16:39:59] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.66, 19.97, 18.77 [16:40:29] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.40, 20.88, 19.40 [16:40:30] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 23.19, 21.52, 20.12 [16:40:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.34, 21.67, 20.03 [16:40:42] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.41, 21.56, 20.28 [16:42:03] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 20.79, 20.47, 19.23 [16:42:36] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.48, 20.82, 18.95 [16:42:52] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.80, 21.47, 20.01 [16:44:35] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 17.87, 19.86, 18.84 [16:45:07] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.89, 19.96, 19.43 [16:45:53] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.76, 19.86, 19.19 [16:47:55] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.97, 19.65, 19.49 [16:48:03] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 19.74, 20.02, 19.44 [16:48:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.91, 20.38, 20.13 [16:49:47] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.09, 20.88, 19.72 [16:51:59] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 18.92, 20.23, 20.14 [16:52:03] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 23.28, 21.46, 20.13 [16:52:32] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.85, 20.86, 20.40 [16:54:03] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 17.54, 20.09, 19.80 [16:54:52] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.47, 20.00, 20.13 [16:57:39] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.87, 20.28, 19.89 [16:58:52] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.58, 21.19, 20.61 [16:59:20] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.89, 11.10, 9.60 [16:59:44] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.16, 20.33, 20.12 [16:59:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.88, 21.15, 20.26 [17:01:17] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 10.45, 11.05, 9.79 [17:01:33] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.32, 20.94, 20.26 [17:03:43] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 21.46, 20.47, 20.09 [17:03:55] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.82, 19.49, 19.78 [17:05:12] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 15.53, 12.02, 10.37 [17:05:29] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.88, 19.07, 19.73 [17:05:43] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 14.87, 18.91, 19.60 [17:06:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.37, 19.70, 20.30 [17:07:28] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.48, 19.11, 20.31 [17:08:52] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 13.76, 17.83, 19.67 [17:09:38] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 10.61, 17.06, 19.72 [17:09:43] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 20.52, 19.99, 19.90 [17:10:21] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 11.41, 17.60, 20.01 [17:10:24] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 12.82, 17.83, 20.33 [17:10:30] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 9.93, 16.12, 19.23 [17:10:42] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 11.17, 17.86, 20.31 [17:10:45] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 9.82, 17.19, 20.10 [17:11:09] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 4.68, 11.03, 10.88 [17:11:43] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 12.17, 17.04, 18.83 [17:11:57] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.24, 16.42, 19.71 [17:12:39] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.30, 16.84, 19.85 [17:14:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762704710000&orgId=1&to=1762708310000 [17:18:45] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.18, 19.42, 19.50 [17:19:09] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 5.04, 8.70, 10.06 [17:19:57] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.42, 22.70, 20.97 [17:20:45] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 20.31, 19.71, 19.60 [17:21:57] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.77, 19.76, 20.08 [17:33:38] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.33, 17.31, 16.06 [17:33:57] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.57, 18.69, 18.22 [17:34:24] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.54, 17.94, 16.91 [17:34:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.71, 18.77, 17.30 [17:34:42] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.04, 17.97, 16.53 [17:34:45] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.20, 19.14, 18.01 [17:36:23] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 20.91, 18.21, 16.04 [17:37:01] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.92, 18.54, 16.27 [17:37:07] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.92, 19.09, 16.86 [17:37:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.50, 19.40, 16.83 [17:38:01] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.22, 18.89, 15.95 [17:38:27] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.69, 19.78, 16.97 [17:39:46] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.18, 19.68, 16.20 [17:39:55] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 22.37, 19.90, 16.72 [17:40:00] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.77, 20.39, 16.73 [17:40:21] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.20, 20.20, 17.54 [17:41:49] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 20.23, 19.76, 16.96 [17:41:50] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 20.25, 20.00, 17.14 [17:43:13] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 18.47, 12.95, 10.36 [17:43:52] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.66, 19.91, 17.41 [17:45:07] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 8.12, 10.84, 9.89 [17:45:41] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.50, 19.95, 17.55 [17:45:47] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.08, 21.12, 17.89 [17:46:18] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.51, 20.27, 18.57 [17:47:01] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 16.55, 12.77, 10.67 [17:47:42] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.37, 20.95, 18.33 [17:49:42] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.71, 19.73, 18.20 [17:51:47] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.31, 19.67, 18.40 [17:51:51] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 21.01, 20.01, 18.11 [17:51:54] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 17.94, 20.20, 19.23 [17:52:37] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.18, 20.22, 19.32 [17:53:21] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 21.22, 19.63, 18.12 [17:54:18] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.46, 20.81, 19.45 [17:56:12] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=seilsarchiveswiki --new=seildirectorywiki --user=Skye (END - exit=0) [17:56:18] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.44, 19.67, 19.21 [17:56:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.44, 20.69, 19.71 [17:56:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:11] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 18.73, 19.79, 18.57 [17:57:12] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=seildirectorywiki (END - exit=0) [17:57:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:36] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.16, 20.22, 19.84 [17:57:37] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 19.23, 19.75, 18.61 [17:58:20] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.89, 20.25, 19.88 [17:59:01] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.28, 19.84, 19.64 [17:59:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762707530000&orgId=1&to=1762711160031 [17:59:56] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.61, 20.35, 20.11 [18:00:24] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.94, 20.39, 19.82 [18:02:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.59, 19.67, 20.33 [18:03:30] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 17.56, 19.48, 19.93 [18:03:54] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.48, 18.92, 20.02 [18:03:56] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 16.17, 18.99, 19.92 [18:05:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.73, 19.00, 19.95 [18:12:24] Guys, MH is a bit slower than usual [18:13:45] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 38.64, 26.49, 22.09 [18:19:08] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.98, 19.70, 18.88 [18:19:19] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.61, 19.65, 19.21 [18:19:39] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 19.77, 22.91, 22.22 [18:20:26] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.28, 19.62, 19.18 [18:21:08] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 34.75, 25.16, 20.98 [18:24:36] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.55, 20.29, 18.75 [18:26:12] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 2.13, 4.00, 11.76 [18:26:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.81, 20.74, 19.54 [18:26:49] PROBLEM - mw153 Current Load on mw153 is CRITICAL: LOAD CRITICAL - total load average: 35.56, 25.56, 19.67 [18:26:49] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.83, 20.25, 18.54 [18:27:26] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.90, 19.35, 17.59 [18:27:42] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.31, 18.79, 16.83 [18:28:29] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.12, 19.50, 18.87 [18:28:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.76, 19.60, 19.28 [18:28:49] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 19.18, 23.66, 19.76 [18:28:49] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.49, 18.31, 18.05 [18:29:08] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 15.32, 23.00, 22.69 [18:29:26] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 12.98, 17.12, 17.00 [18:29:42] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.46, 17.40, 16.56 [18:30:08] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.05, 19.71, 19.84 [18:30:12] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 1.93, 2.97, 9.57 [18:30:49] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 11.46, 19.64, 18.78 [18:34:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762707530000&orgId=1&to=1762713080000 [18:34:49] PROBLEM - mw153 Current Load on mw153 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 23.10, 20.22 [18:34:57] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.83, 22.30, 19.09 [18:35:19] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.24, 21.75, 20.85 [18:35:38] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 21.48, 18.43, 17.39 [18:35:54] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.67, 18.39, 16.53 [18:36:09] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.85, 20.92, 20.14 [18:36:42] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.70, 19.97, 18.85 [18:36:49] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 18.63, 21.71, 20.09 [18:36:51] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.28, 20.27, 18.75 [18:36:56] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.07, 21.06, 19.75 [18:37:08] PROBLEM - mw163 Current Load on mw163 is CRITICAL: LOAD CRITICAL - total load average: 26.06, 21.84, 18.10 [18:37:19] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.83, 21.54, 20.90 [18:37:43] oh my [18:37:49] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.46, 18.39, 16.77 [18:38:40] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.18, 19.25, 18.70 [18:38:44] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 13.84, 18.90, 19.25 [18:39:03] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 21.00, 21.87, 18.58 [18:39:05] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 14.46, 17.95, 20.10 [18:40:58] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 15.61, 19.41, 18.05 [18:41:02] @smallam this kind of shows if it's down or whatever [18:42:01] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.67, 18.75, 19.63 [18:42:33] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.55, 20.58, 19.37 [18:43:28] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.38, 21.26, 19.79 [18:43:35] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 19.07, 19.96, 18.71 [18:44:46] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.36, 19.17, 19.62 [18:45:24] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.88, 19.42, 19.28 [18:48:40] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.64, 11.78, 8.88 [18:50:25] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 15.15, 19.00, 19.42 [18:50:36] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.58, 10.09, 8.60 [18:50:36] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.34, 19.70, 20.30 [18:50:48] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.86, 20.35, 19.69 [18:51:56] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.69, 19.48, 18.13 [18:52:46] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.94, 19.40, 19.45 [18:52:53] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.75, 20.28, 18.40 [18:54:23] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.34, 21.71, 20.99 [18:55:56] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.88, 19.24, 18.43 [18:56:47] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.62, 20.29, 18.85 [18:57:25] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.07, 19.83, 19.54 [18:57:28] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.95, 20.13, 19.11 [18:57:35] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.16, 21.04, 20.26 [18:59:27] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 15.75, 18.88, 18.82 [19:01:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.46, 19.82, 19.63 [19:02:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762711310000&orgId=1&to=1762714940036 [19:05:19] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.86, 19.14, 17.00 [19:06:27] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.77, 20.03, 19.08 [19:08:18] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.85, 18.70, 20.34 [19:08:24] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.85, 18.76, 18.74 [19:09:15] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.50, 18.03, 17.16 [19:10:52] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.37, 11.07, 9.52 [19:11:04] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.33, 21.11, 20.03 [19:12:47] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.51, 9.48, 9.12 [19:13:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.70, 20.19, 19.86 [19:13:15] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 14.68, 18.24, 19.89 [19:17:12] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 14.26, 18.44, 20.33 [19:18:04] these servers are too evil [19:34:43] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.88, 19.66, 19.01 [19:35:56] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.41, 20.44, 19.34 [19:38:38] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.69, 20.39, 19.49 [19:39:50] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.32, 18.70, 18.98 [20:01:02] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.74, 20.90, 19.75 [20:02:59] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 21.98, 20.28 [20:03:56] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.42, 20.16, 18.04 [20:04:56] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.36, 22.35, 20.65 [20:06:04] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.92, 17.68, 17.42 [20:08:49] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.45, 19.86, 19.95 [20:09:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.71, 20.86, 19.52 [20:12:54] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.08, 19.86, 19.41 [20:13:31] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.04, 20.70, 19.37 [20:13:50] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.78, 20.78, 19.49 [20:15:29] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.28, 20.40, 19.41 [20:16:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.39, 20.24, 19.99 [20:17:42] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 15.70, 19.19, 19.25 [20:18:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.85, 18.96, 19.56 [20:27:13] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.98, 20.44, 19.39 [20:29:11] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.61, 18.30, 18.74 [20:30:22] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.13, 21.53, 20.14 [20:33:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.17, 20.52, 19.55 [20:35:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.46, 19.17, 19.17 [20:36:11] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 15.13, 18.68, 19.50 [20:42:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.94, 21.27, 20.06 [20:48:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 15.99, 19.57, 19.88 [20:57:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762711310000&orgId=1&to=1762721750000 [20:57:28] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.22, 21.13, 20.38 [21:01:19] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.22, 19.80, 20.03 [21:01:50] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.91, 20.57, 19.27 [21:03:47] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.50, 19.38, 18.98 [21:06:22] PROBLEM - cp171 health.wikitide.net HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:08:18] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27159MiB (5% inode=99%); [21:15:44] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.12, 20.38, 19.92 [21:17:39] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.16, 19.43, 19.63 [21:23:55] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.33, 19.01, 17.12 [21:25:55] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 15.77, 18.05, 17.03 [21:26:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762719980000&orgId=1&to=1762723610044 [21:30:33] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.33, 19.12, 17.60 [21:32:29] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 17.73, 18.70, 17.64 [21:33:11] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.30, 19.37, 18.04 [21:35:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.92, 18.44, 17.87 [21:36:37] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.27, 20.19, 18.84 [21:37:56] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.86, 11.81, 8.66 [21:39:53] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.75, 9.90, 8.34 [21:39:57] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.03, 20.47, 18.99 [21:40:18] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.76, 20.36, 18.66 [21:43:50] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.08, 19.29, 18.90 [21:48:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.88, 19.72, 19.69 [21:48:04] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 14.57, 19.11, 19.05 [21:53:12] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.63, 20.54, 19.49 [21:58:57] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 18.19, 19.77, 19.60 [21:59:23] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.99, 19.22, 18.60 [22:01:19] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.31, 19.54, 18.79 [22:04:44] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.98, 21.14, 20.23 [22:04:47] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.79, 20.50, 18.46 [22:05:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.61, 20.21, 18.45 [22:05:14] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.94, 21.01, 19.61 [22:05:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.21, 18.99, 16.55 [22:05:35] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.84, 18.92, 18.02 [22:06:36] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.96, 19.16, 17.52 [22:06:42] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 14.44, 18.70, 18.08 [22:07:10] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.25, 19.69, 19.27 [22:07:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 15.34, 17.59, 16.35 [22:07:32] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.93, 18.54, 17.98 [22:07:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.35, 19.99, 18.85 [22:08:34] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.26, 18.19, 17.36 [22:08:40] PROBLEM - os201 Current Load on os201 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 3.34, 2.21 [22:09:07] PROBLEM - os191 Current Load on os191 is WARNING: LOAD WARNING - total load average: 3.57, 3.42, 2.34 [22:09:07] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.55, 3.25, 2.31 [22:09:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.30, 19.66, 18.90 [22:10:34] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27106MiB (5% inode=99%); [22:10:40] RECOVERY - os201 Current Load on os201 is OK: LOAD OK - total load average: 1.80, 2.75, 2.13 [22:11:07] RECOVERY - os191 Current Load on os191 is OK: LOAD OK - total load average: 1.71, 2.77, 2.24 [22:11:07] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.01, 2.75, 2.24 [22:15:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.69, 20.24, 19.49 [22:18:34] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 72780MiB (16% inode=99%); [22:18:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 17.62, 19.46, 20.10 [22:25:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.54, 19.67, 18.53 [22:27:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 15.87, 18.25, 18.16 [22:46:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762724780000&orgId=1&to=1762728380000 [22:54:03]