[00:00:46] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.00, 17.93, 16.07 [00:12:34] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 75555MiB (16% inode=99%); [00:13:18] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 75208MiB (16% inode=99%); [00:13:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.16, 9.38, 7.39 [00:14:18] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 67052MiB (14% inode=99%); [00:15:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.40, 9.98, 7.83 [00:17:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.41, 10.38, 8.24 [00:19:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 6.86, 9.23, 8.10 [00:41:42] PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl [00:49:42] RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl [01:00:39] @smallam omg puppet from fazbear world tower defense! [01:09:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.02, 18.13, 15.77 [01:11:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.31, 18.22, 16.11 [01:23:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762993400000&orgId=1&to=1762997030039 [01:28:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1762993400000&orgId=1&to=1762997300000 [01:48:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.14, 19.58, 17.33 [01:49:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.60, 19.55, 16.86 [01:51:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.39, 19.76, 17.28 [01:52:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 15.41, 18.31, 17.45 [03:32:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763001140000&orgId=1&to=1763004770036 [03:47:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763001140000&orgId=1&to=1763005550000 [04:30:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.53, 19.15, 17.18 [04:32:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.92, 18.93, 17.32 [04:33:43] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.69, 19.89, 17.27 [04:35:41] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.37, 18.08, 16.91 [04:38:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.70, 20.87, 18.59 [04:45:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.94, 21.17, 18.60 [04:45:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.64, 19.21, 17.40 [04:46:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.42, 19.51, 19.13 [04:47:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.93, 18.36, 17.30 [04:49:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 14.47, 18.99, 18.45 [04:50:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763005790000&orgId=1&to=1763009420041 [05:20:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763005790000&orgId=1&to=1763011070000 [05:33:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.34, 19.85, 17.31 [05:35:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.71, 18.57, 17.18 [05:39:37] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.81, 10.24, 8.12 [05:41:09] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.13, 19.59, 17.16 [05:41:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.82, 20.50, 18.37 [05:42:25] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.69, 19.86, 17.57 [05:42:34] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.86, 18.76, 16.87 [05:43:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.60, 19.78, 17.97 [05:43:09] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.96, 18.71, 16.30 [05:43:15] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.91, 19.02, 15.50 [05:44:09] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.26, 20.28, 18.19 [05:44:34] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.28, 19.01, 17.20 [05:44:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.14, 19.62, 17.20 [05:44:42] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.97, 19.20, 15.90 [05:45:09] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 17.05, 18.34, 16.49 [05:45:09] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.57, 20.17, 18.00 [05:45:15] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 18.52, 19.08, 15.98 [05:46:05] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.58, 19.42, 18.10 [05:46:14] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.56, 19.05, 17.80 [05:46:36] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.34, 18.15, 15.89 [05:46:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 20.17, 19.86, 17.59 [05:48:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.56, 20.09, 18.87 [05:49:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 16.57, 19.14, 18.41 [05:50:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.98, 18.92, 18.59 [05:51:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.95, 19.72, 19.17 [05:53:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.71, 20.73, 19.23 [05:53:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.75, 11.37, 11.18 [05:54:27] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.87, 20.52, 19.33 [05:55:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763009720000&orgId=1&to=1763013350036 [05:59:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.12, 19.56, 19.35 [05:59:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 8.66, 9.15, 10.18 [06:06:27] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 20.29, 20.31, 20.09 [06:07:36] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.17, 10.25, 10.17 [06:09:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.51, 10.46, 10.25 [06:11:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 8.03, 9.33, 9.85 [06:20:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763011010000&orgId=1&to=1763014610000 [10:33:15] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.23, 8.60, 6.58 [10:35:15] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.00, 8.19, 6.66 [10:39:50] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.59, 19.73, 17.73 [10:39:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.94, 20.39, 18.09 [10:41:46] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.37, 19.78, 17.99 [10:42:21] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.20, 20.58, 17.73 [10:44:17] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.37, 20.29, 17.98 [10:44:50] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.54, 11.79, 9.23 [10:48:39] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.73, 10.66, 9.35 [10:49:47] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.76, 18.82, 18.81 [10:50:33] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 7.71, 9.79, 9.19 [10:55:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763027690000&orgId=1&to=1763031320036 [10:55:23] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.01, 20.79, 19.24 [10:58:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.70, 20.31, 19.53 [11:03:08] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.76, 20.17, 19.69 [11:05:36] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.41, 20.93, 19.35 [11:05:47] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.66, 20.77, 19.01 [11:07:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.05, 19.82, 19.16 [11:08:14] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.41, 21.03, 19.73 [11:08:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.51, 19.13, 19.77 [11:16:04] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.70, 20.40, 20.21 [11:20:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.38, 20.62, 20.02 [11:21:19] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 16.00, 18.76, 19.49 [11:22:08] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.86, 20.26, 19.99 [11:27:40] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.94, 21.31, 20.45 [11:31:36] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 10.19, 10.28, 9.86 [11:31:52] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.34, 21.14, 20.34 [11:33:35] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.75, 20.95, 20.29 [11:35:36] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 8.48, 9.96, 9.89 [11:37:29] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.90, 19.86, 20.03 [11:37:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 20.40, 20.46, 19.23 [11:37:45] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.31, 20.22, 20.29 [11:39:18] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.60, 19.90, 20.37 [11:39:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.37, 20.22, 19.30 [11:43:19] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.05, 20.87, 20.47 [11:45:16] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.09, 19.57, 20.03 [11:52:34] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.60, 18.01, 16.59 [11:53:09] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 22.26, 19.23, 17.35 [11:54:34] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.56, 17.57, 16.61 [11:55:00] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.86, 20.07, 19.83 [11:55:01] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.91, 11.50, 10.33 [11:55:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.19, 20.21, 19.63 [11:55:55] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.04, 20.04, 18.29 [11:56:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.41, 20.75, 19.99 [11:56:25] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.60, 20.37, 19.04 [11:56:37] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.52, 19.85, 17.79 [11:56:55] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 10.93, 11.48, 10.49 [11:57:09] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 20.31, 20.26, 18.25 [11:57:55] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 18.67, 19.21, 18.18 [11:58:21] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 15.89, 18.87, 18.67 [11:58:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.34, 18.94, 17.74 [11:58:49] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 6.83, 9.94, 10.05 [11:58:55] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.70, 20.08, 19.99 [12:01:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 12.03, 17.68, 19.04 [12:04:11] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.25, 18.75, 19.60 [12:29:03] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.97, 19.99, 18.68 [12:32:05] miraheze/MirahezeMagic - translatewiki the build passed. [12:38:14] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.66, 19.58, 17.80 [12:42:07] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 15.46, 19.22, 18.20 [12:45:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763034290000&orgId=1&to=1763037920045 [12:57:03] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.91, 20.34, 18.95 [12:59:02] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.20, 19.36, 18.74 [13:11:05] [02DataDump] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/DataDump/commit/054531d19c2d [13:11:06] 02DataDump/03paladox-patch-1 07paladox 03054531d Fix issue with connection going away when command runs for a long time… [13:11:09] [02DataDump] 07paladox opened pull request #112: Fix issue with connection going away when command runs for a long time (03main...03paladox-patch-1) 13https://github.com/miraheze/DataDump/pull/112 [13:11:16] [02DataDump] 07coderabbitai[bot] commented on pull request #112:
[…] 13https://github.com/miraheze/DataDump/pull/112#issuecomment-3527755898 [13:14:04] miraheze/DataDump - paladox the build has errored. [13:27:03] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.47, 18.56, 17.17 [13:28:27] [02DataDump] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/DataDump/commit/880519fe32d642f1b4f1b95a36a309a694603af1 [13:28:28] 02DataDump/03paladox-patch-1 07paladox 03880519f Update DataDumpGenerateJob.php [13:29:03] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.87, 17.99, 17.12 [13:31:16] miraheze/DataDump - paladox the build has errored. [13:41:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.70, 19.94, 17.13 [13:43:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.10, 18.16, 16.81 [13:46:16] [02DataDump] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/DataDump/commit/e4cb3b1a02624de0a54be3d9f099ee896d43fade [13:46:16] 02DataDump/03paladox-patch-1 07paladox 03e4cb3b1 Update DataDumpGenerateJob.php [13:49:06] miraheze/DataDump - paladox the build has errored. [13:54:24] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 48% [13:56:24] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 11% [13:57:43] [02DataDump] 07SomeMWDev created 03status-job (+1 new commit) 13https://github.com/miraheze/DataDump/commit/f3b47354c93b [13:57:43] 02DataDump/03status-job 07SomeRandomDeveloper 03f3b4735 Use job to update status after generating the dump… [13:57:56] [02DataDump] 07SomeMWDev opened pull request #113: Use job to update status after generating the dump (03main...03status-job) 13https://github.com/miraheze/DataDump/pull/113 [13:58:04] [02DataDump] 07coderabbitai[bot] commented on pull request #113:
[…] 13https://github.com/miraheze/DataDump/pull/113#issuecomment-3527956921 [13:59:30] [02DataDump] 07github-actions[bot] pushed 1 new commit to 03status-job 13https://github.com/miraheze/DataDump/commit/ff96c4373284318a96c56d87750a9a5e0873bc8a [13:59:30] 02DataDump/03status-job 07github-actions 03ff96c43 CI: lint code to MediaWiki standards… [14:01:06] miraheze/DataDump - SomeMWDev the build has errored. [14:02:35] [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-12.4.3 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/d2ae637c3b9e [14:02:35] 02mw-config/03dependabot/composer/phpunit/phpunit-12.4.3 07dependabot[bot] 03d2ae637 Update phpunit/phpunit requirement from 12.4.1 to 12.4.3… [14:02:36] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6176 (Update phpunit/phpunit requirement from 12.4.1 to 12.4.3) 13https://github.com/miraheze/mw-config/pull/6176 [14:02:36] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6176 (Update phpunit/phpunit requirement from 12.4.1 to 12.4.3) 13https://github.com/miraheze/mw-config/pull/6176 [14:02:38] [02mw-config] 07dependabot[bot] opened pull request #6176: Update phpunit/phpunit requirement from 12.4.1 to 12.4.3 (03main...03dependabot/composer/phpunit/phpunit-12.4.3) 13https://github.com/miraheze/mw-config/pull/6176 [14:02:40] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6176 (Update phpunit/phpunit requirement from 12.4.1 to 12.4.3) 13https://github.com/miraheze/mw-config/pull/6176 [14:02:42] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6176 (Update phpunit/phpunit requirement from 12.4.1 to 12.4.3) 13https://github.com/miraheze/mw-config/pull/6176 [14:02:44] [02mw-config] 07dependabot[bot] commented on pull request #6155: Superseded by #6176. 13https://github.com/miraheze/mw-config/pull/6155#issuecomment-3527978701 [14:02:46] [02mw-config] 07dependabot[bot] closed pull request #6155: Update phpunit/phpunit requirement from 12.4.1 to 12.4.2 (03main...03dependabot/composer/phpunit/phpunit-12.4.2) 13https://github.com/miraheze/mw-config/pull/6155 [14:02:48] [02mw-config] 07dependabot[bot] 04deleted 03dependabot/composer/phpunit/phpunit-12.4.2 at 03c9d35f6 13https://api.github.com/repos/miraheze/mw-config/commit/c9d35f6 [14:03:29] miraheze/mw-config - dependabot[bot] the build passed. [14:03:42] [02DataDump] 07SomeMWDev pushed 1 new commit to 03status-job 13https://github.com/miraheze/DataDump/commit/7f9dd4170929cd7b93466fd87b994f9325ffa1d9 [14:03:42] 02DataDump/03status-job 07SomeRandomDeveloper 037f9dd41 Trigger CI [14:05:28] [02DataDump] 07paladox approved pull request #113 13https://github.com/miraheze/DataDump/pull/113#pullrequestreview-3459965919 [14:06:13] [02DataDump] 07paladox closed pull request #112: Fix issue with connection going away when command runs for a long time (03main...03paladox-patch-1) 13https://github.com/miraheze/DataDump/pull/112 [14:06:21] [02DataDump] 07paladox 04deleted 03paladox-patch-1 at 03e4cb3b1 13https://api.github.com/repos/miraheze/DataDump/commit/e4cb3b1 [14:07:16] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/DataDump'} to test151 [14:07:17] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/DataDump'} to test151 - SUCCESS in 0s [14:07:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:07:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:49] miraheze/DataDump - SomeMWDev the build passed. [14:09:28] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.52, 20.36, 19.08 [14:15:20] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 15.15, 18.99, 19.14 [14:20:00] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/858bc1c257a4 [14:20:00] 02puppet/03paladox-patch-6 07paladox 03858bc1c Add syslog identifier to backup job configuration [14:20:04] [02puppet] 07paladox opened pull request #4595: Add syslog identifier to backup job configuration (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4595 [14:20:14] [02puppet] 07coderabbitai[bot] commented on pull request #4595:
[…] 13https://github.com/miraheze/puppet/pull/4595#issuecomment-3528048993 [14:20:17] PROBLEM - db171 Check unit status of sql-backup on db171 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [14:21:17] [02puppet] 07paladox merged pull request #4595: Add syslog identifier to backup job configuration (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4595 [14:21:17] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3a938f158e84ead54e602b3613ad37c7b2f212ef [14:21:17] 02puppet/03main 07paladox 033a938f1 Add syslog identifier to backup job configuration (#4595) [14:21:18] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03858bc1c 13https://api.github.com/repos/miraheze/puppet/commit/858bc1c [14:23:52] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.44/extensions/DataDump'} to test151 [14:23:53] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.44/extensions/DataDump'} to test151 - SUCCESS in 0s [14:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:45] [02DataDump] 07SomeMWDev merged pull request #113: Use job to update status after generating the dump (03main...03status-job) 13https://github.com/miraheze/DataDump/pull/113 [14:26:45] [02DataDump] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/f487ff0752365c17b99ac40ac0692c2cf5a3dfbb [14:26:45] 02DataDump/03main 07SomeRandomDeveloper 03f487ff0 Use job to update status after generating the dump (#113)… [14:26:46] [02DataDump] 07SomeMWDev 04deleted 03status-job at 037f9dd41 13https://api.github.com/repos/miraheze/DataDump/commit/7f9dd41 [14:26:58] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'DataDump'} to test151 [14:27:00] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'DataDump'} to test151 - SUCCESS in 1s [14:27:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:15] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DataDump'} to all [14:27:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:41] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'DataDump'} to all - SUCCESS in 26s [14:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:41] miraheze/DataDump - SomeMWDev the build passed. [14:32:30] [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1b93dd6029b7 [14:32:31] 02puppet/03paladox-patch-6 07paladox 031b93dd6 Add DataDumpStatusJob as its own queue in changeprop [14:32:34] [02puppet] 07paladox opened pull request #4596: Add DataDumpStatusJob as its own queue in changeprop (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4596 [14:32:41] [02puppet] 07coderabbitai[bot] commented on pull request #4596:
[…] 13https://github.com/miraheze/puppet/pull/4596#issuecomment-3528103125 [14:33:31] [02puppet] 07paladox retitled PR #4596: "Add DataDumpStatusJob as its own queue in changeprop" ➜ "Add DataDumpStatusUpdateJob as its own queue in changeprop" 13https://github.com/miraheze/puppet/pull/4596 [14:33:41] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/6a9dc54d3daddfa54b0a0880bd0887797b6512a2 [14:33:41] 02puppet/03paladox-patch-6 07paladox 036a9dc54 Update changeprop201.yaml [14:37:41] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 14.34, 11.27, 9.37 [14:37:44] [02puppet] 07paladox created 03paladox-patch-7 (+1 new commit) 13https://github.com/miraheze/puppet/commit/01b6622d8089 [14:37:46] 02puppet/03paladox-patch-7 07paladox 0301b6622 Refactor backup script to improve process handling… [14:37:52] [02puppet] 07paladox opened pull request #4597: Refactor backup script to improve process handling (03main...03paladox-patch-7) 13https://github.com/miraheze/puppet/pull/4597 [14:38:04] [02puppet] 07coderabbitai[bot] commented on pull request #4597:
[…] 13https://github.com/miraheze/puppet/pull/4597#issuecomment-3528127720 [14:39:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.71, 10.21, 9.21 [14:41:37] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.17, 11.09, 9.65 [14:43:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.60, 11.24, 9.88 [14:43:45] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:45:37] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 17.55, 13.41, 10.83 [14:47:30] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.52, 20.90, 19.40 [14:47:39] mooooooom are they supposed to do that [14:51:15] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.02, 19.42, 16.53 [14:51:24] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.38, 20.30, 19.50 [14:53:15] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 19.80, 19.23, 16.80 [14:53:32] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.91, 19.73, 17.88 [14:54:18] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49792MiB (10% inode=99%); [14:55:32] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 16.73, 18.59, 17.69 [14:55:45] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.32, 20.39, 17.82 [14:57:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.74, 20.14, 18.05 [14:59:02] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [14:59:08] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.72, 21.29, 20.21 [15:03:03] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.08, 20.06, 20.02 [15:03:46] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.43, 20.35, 19.19 [15:05:02] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [15:05:43] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.21, 19.97, 19.20 [15:11:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.85, 20.53, 18.91 [15:12:47] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.06, 21.34, 20.33 [15:13:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.00, 20.14, 18.99 [15:15:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c9302aaa6e97feb21b5bdfccee8c57bc48bce41f [15:15:21] 02ssl/03main 07WikiTideBot 03c9302aa Bot: Auto-update domain lists [15:16:14] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 21.47, 19.92 [15:17:09] the db161 load is a side effect of https://issue-tracker.miraheze.org/T14516#290891 [15:17:31] it shouldn't overload this time [15:18:11] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.11, 21.59, 20.15 [15:19:13] i was mostly worried about puppet181 https://discord.com/channels/407504499280707585/808001911868489748/1438539844052713492 [15:19:40] oh [15:20:37] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.37, 20.25, 20.28 [15:22:07] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.83, 19.02, 19.45 [15:25:33] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f3fc1d1bb2cdb302b73f8f503cc53de6c8cc5d4a [15:25:33] 02ssl/03main 07WikiTideBot 03f3fc1d1 Bot: Auto-update domain lists [15:27:32] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=cloudcraftercommonswiki --new=cloudcommonswiki --user=Skye (END - exit=0) [15:27:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:06] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=cloudcommonswiki (END - exit=0) [15:29:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:32:40] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=bakutenshootbeybladekaihenwiki --new=bsbeternalwiki --user=Skye (END - exit=0) [15:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:29] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=bsbeternalwiki (END - exit=0) [15:33:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:03] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.99, 19.79, 18.77 [15:36:58] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=gracerbxwiki --new=gracerobloxwiki --user=Skye (END - exit=0) [15:37:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:37:03] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.26, 18.68, 18.48 [15:38:29] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=gracerobloxwiki (END - exit=0) [15:38:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:28] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.60, 20.82, 19.68 [15:44:28] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.89, 19.86, 19.56 [15:54:17] RECOVERY - db171 Check unit status of sql-backup on db171 is OK: OK: Status of the systemd unit sql-backup [15:55:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763034290000&orgId=1&to=1763049320000 [15:55:37] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.72, 8.78, 11.75 [15:59:06] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6c4d21e2f9cf03aa2064ac428f0c61452afdd62c [15:59:06] 02ssl/03main 07MacFan4000 036c4d21e Update redirects.yaml [15:59:15] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.71, 9.62, 7.34 [16:00:17] PROBLEM - db181 Check unit status of sql-backup on db181 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [16:01:15] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.25, 11.55, 8.30 [16:02:49] PROBLEM - www.mwcosmos.com - Cloudflare on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80EB0DE2A6140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [16:04:17] RECOVERY - db181 Check unit status of sql-backup on db181 is OK: OK: Status of the systemd unit sql-backup [16:06:30] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/41b4e776de764e2b9cca4d0783a846dcb80b8d12 [16:06:31] 02ssl/03main 07WikiTideBot 0341b4e77 Bot: Auto-update domain lists [16:08:24] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=linguacommunalwiki --new=linguacomunalwiki --user=Skye (END - exit=0) [16:08:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:35] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=linguacomunalwiki (END - exit=0) [16:09:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:33] [02puppet] 07paladox created 03paladox-patch-8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/2d1c36fbbfdd [16:10:33] 02puppet/03paladox-patch-8 07paladox 032d1c36f wikitide-backup: isolate child processes using preexec_fn=os.setsid [16:10:37] [02puppet] 07paladox opened pull request #4598: wikitide-backup: isolate child processes using preexec_fn=os.setsid (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4598 [16:10:42] [02puppet] 07coderabbitai[bot] commented on pull request #4598:
[…] 13https://github.com/miraheze/puppet/pull/4598#issuecomment-3528547789 [16:12:28] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.86, 20.55, 19.02 [16:13:37] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 9.10, 9.51, 10.20 [16:15:20] [02puppet] 07paladox merged pull request #4598: wikitide-backup: isolate child processes using preexec_fn=os.setsid (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4598 [16:15:22] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/966b9a012f12ff881f64c148b09abf751cbce17f [16:15:24] [02puppet] 07paladox 04deleted 03paladox-patch-8 at 032d1c36f 13https://api.github.com/repos/miraheze/puppet/commit/2d1c36f [16:15:26] 02puppet/03main 07paladox 03966b9a0 wikitide-backup: isolate child processes using preexec_fn=os.setsid (#4598) [16:17:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a9822c7e7dae892e22558a52a94cd2225d2b9522 [16:17:19] 02ssl/03main 07WikiTideBot 03a9822c7 Bot: Auto-update domain lists [16:17:39] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=cpimaginedwiki --new=cpzerowiki --user=Skye (END - exit=0) [16:17:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:28] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.48, 20.17, 19.49 [16:20:16] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=cpzerowiki (END - exit=0) [16:20:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:25:09] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.24, 19.22, 16.76 [16:25:15] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.97, 11.49, 11.60 [16:26:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763047550000&orgId=1&to=1763051180039 [16:27:03] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.29, 20.86, 19.19 [16:27:09] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 15.42, 17.54, 16.43 [16:32:28] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.28, 21.28, 20.18 [16:34:28] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.68, 20.29, 19.95 [16:37:03] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.90, 19.54, 19.63 [16:37:15] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.11, 7.62, 9.63 [16:43:07] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49847MiB (10% inode=99%); [16:43:57] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.93, 21.14, 20.44 [16:45:55] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.60, 19.78, 20.03 [17:04:52] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 62638MiB (10% inode=99%); [17:11:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763050010000&orgId=1&to=1763053610000 [17:29:14] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49884MiB (10% inode=99%); [17:34:13] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 33752MiB (5% inode=99%); [18:01:52] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 164594MiB (27% inode=99%); [18:39:53] [02mw-config] 07toosplatched opened pull request #6177: Rephrasing $wmgSharedUploadClientDBname's description for clarity (07miraheze:03main...07toosplatched:03toosplatched-patch-1) 13https://github.com/miraheze/mw-config/pull/6177 [18:40:51] miraheze/mw-config - toosplatched the build passed. [19:08:10] [02mw-config] 07SomeMWDev approved pull request #6177 13https://github.com/miraheze/mw-config/pull/6177#pullrequestreview-3461271130 [19:08:17] [02mw-config] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/1980dcfdbf95cda6d00bfde94c3fe575f427ab71 [19:08:17] 02mw-config/03main 07stackd 031980dcf Rephrasing $wmgSharedUploadClientDBname's description for clarity (#6177) [19:08:17] [02mw-config] 07SomeMWDev merged 07toosplatched's pull request #6177: Rephrasing $wmgSharedUploadClientDBname's description for clarity (07miraheze:03main...07toosplatched:03toosplatched-patch-1) 13https://github.com/miraheze/mw-config/pull/6177 [19:08:23] !log [somerandomdeveloper@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [19:08:23] !log [somerandomdeveloper@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [19:08:27] !log [somerandomdeveloper@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:08:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:53] !log [somerandomdeveloper@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [19:08:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:19] miraheze/mw-config - SomeMWDev the build passed. [19:18:48] [02mw-config] 07pskyechology pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e2178ad38f5ae04ac76e579d10abe69ff7f1787c [19:18:49] 02mw-config/03main 07Skye 03e2178ad Typo: databse… [19:19:53] miraheze/mw-config - pskyechology the build passed. [19:26:07] [02puppet] 07paladox created 03paladox-patch-8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/430e3b38aff0 [19:26:07] 02puppet/03paladox-patch-8 07paladox 03430e3b3 Rename backup job to 'puppetdb-backup' [19:26:11] [02puppet] 07paladox opened pull request #4599: Rename backup job to 'puppetdb-backup' (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4599 [19:26:17] [02puppet] 07coderabbitai[bot] commented on pull request #4599:
[…] 13https://github.com/miraheze/puppet/pull/4599#issuecomment-3529375418 [19:26:20] [02puppet] 07paladox merged pull request #4599: Rename backup job to 'puppetdb-backup' (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4599 [19:26:20] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b5c925fd7b829baf9d8032a33b5b65c51d961e9d [19:26:20] 02puppet/03main 07paladox 03b5c925f Rename backup job to 'puppetdb-backup' (#4599) [19:26:21] [02puppet] 07paladox 04deleted 03paladox-patch-8 at 03430e3b3 13https://api.github.com/repos/miraheze/puppet/commit/430e3b3 [19:28:59] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:25] [02puppet] 07paladox created 03paladox-patch-8 (+1 new commit) 13https://github.com/miraheze/puppet/commit/959a2ea3bd42 [19:29:25] 02puppet/03paladox-patch-8 07paladox 03959a2ea Update syslog identifier for backup job [19:29:28] [02puppet] 07paladox opened pull request #4600: Update syslog identifier for backup job (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4600 [19:29:34] [02puppet] 07coderabbitai[bot] commented on pull request #4600:
[…] 13https://github.com/miraheze/puppet/pull/4600#issuecomment-3529388562 [19:29:50] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-8 13https://github.com/miraheze/puppet/commit/2fecafe3eb19428887cc79c4eef28c208365d289 [19:29:50] 02puppet/03paladox-patch-8 07paladox 032fecafe Rename backup job from 'puppetdb-backup' to 'puppetdb' [19:31:09] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/52bb01cfc182017a1b07f8914e8370535a936178 [19:31:09] 02puppet/03main 07paladox 0352bb01c Update syslog identifier for backup job (#4600) [19:31:11] [02puppet] 07paladox merged pull request #4600: Update syslog identifier for backup job (03main...03paladox-patch-8) 13https://github.com/miraheze/puppet/pull/4600 [19:31:13] [02puppet] 07paladox 04deleted 03paladox-patch-8 at 032fecafe 13https://api.github.com/repos/miraheze/puppet/commit/2fecafe [19:31:38] [02puppet] 07SomeMWDev approved pull request #4596 13https://github.com/miraheze/puppet/pull/4596#pullrequestreview-3461360638 [19:32:16] [02puppet] 07paladox merged pull request #4596: Add DataDumpStatusUpdateJob as its own queue in changeprop (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4596 [19:32:16] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0e86dd6c6b4f823d06a5e22c7e2ad9bdd13c65bc [19:32:16] 02puppet/03main 07paladox 030e86dd6 Add DataDumpStatusUpdateJob as its own queue in changeprop (#4596) [19:32:18] [02puppet] 07paladox 04deleted 03paladox-patch-6 at 036a9dc54 13https://api.github.com/repos/miraheze/puppet/commit/6a9dc54 [19:36:51] [02CreateWiki] 07SomeMWDev created 03special-page-writes (+2 new commits) 13https://github.com/miraheze/CreateWiki/compare/1752d3fce2e4^...cc991dc175de [19:36:51] 02CreateWiki/03special-page-writes 07SomeRandomDeveloper 031752d3f Set doesWrites to true for SpecialRequestWikiQueue… [19:36:51] 02CreateWiki/03special-page-writes 07SomeRandomDeveloper 03cc991dc Set doesWrites to true for SpecialRequestWiki [19:37:37] [02CreateWiki] 07SomeMWDev opened pull request #789: Set doesWrites to true for both special pages (03main...03special-page-writes) 13https://github.com/miraheze/CreateWiki/pull/789 [19:37:43] [02CreateWiki] 07coderabbitai[bot] commented on pull request #789:
[…] 13https://github.com/miraheze/CreateWiki/pull/789#issuecomment-3529416017 [19:38:09] [02CreateWiki] 07SomeMWDev retitled PR #789: "Set doesWrites to true for both special pages" ➜ "Set doesWrites to true for both wiki request special pages" 13https://github.com/miraheze/CreateWiki/pull/789 [19:41:01] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.64, 10.29, 7.52 [19:42:16] [02CreateWiki] 07paladox approved pull request #788 13https://github.com/miraheze/CreateWiki/pull/788#pullrequestreview-3461395973 [19:42:40] [02CreateWiki] 07codecov[bot] commented on pull request #789: :x: Patch coverage is `0%` with `4 lines` in your changes missing coverage. Please review. […] 13https://github.com/miraheze/CreateWiki/pull/789#issuecomment-3529431381 [19:42:47] [02CreateWiki] 07paladox approved pull request #789 13https://github.com/miraheze/CreateWiki/pull/789#pullrequestreview-3461397511 [19:43:01] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 11.97, 11.04, 8.15 [19:44:10] [02CreateWiki] 07SomeMWDev merged pull request #788: Avoid regenerating dblists more than once per server (03main...03local-dblist-cache-key) 13https://github.com/miraheze/CreateWiki/pull/788 [19:44:10] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/068a397c116b510590fb3ff23b1448823c2b44e9 [19:44:10] 02CreateWiki/03main 07SomeRandomDeveloper 03068a397 Avoid regenerating dblists more than once per server (#788)… [19:44:12] [02CreateWiki] 07SomeMWDev 04deleted 03local-dblist-cache-key at 0345586e7 13https://api.github.com/repos/miraheze/CreateWiki/commit/45586e7 [19:45:01] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 13.74, 12.18, 8.93 [19:45:38] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CreateWiki'} to test151 [19:45:40] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [19:45:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:01] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.34, 10.68, 8.78 [19:47:26] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all [19:47:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:52] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 26s [19:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:01] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 9.29, 10.13, 8.80 [19:54:02] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03special-page-writes 13https://github.com/miraheze/CreateWiki/commit/f9c10355c76261dc17ccc7f3a717d1595fc544c2 [19:54:02] 02CreateWiki/03special-page-writes 07SomeRandomDeveloper 03f9c1035 add return types + test [19:55:01] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 23.19, 15.48, 11.28 [19:56:42] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [20:03:32] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 77 packages available for upgrade (4 critical updates). [20:04:20] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 [20:04:21] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 - SUCCESS in 0s [20:04:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:43] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [20:05:25] miraheze/CreateWiki - SomeMWDev the build passed. [20:08:46] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (3 critical updates). [20:10:29] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 129 packages available for upgrade (3 critical updates). [20:11:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.11, 20.36, 18.23 [20:13:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.19, 19.88, 18.32 [20:13:29] PROBLEM - swiftobject201 APT on swiftobject201 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [20:14:18] PROBLEM - swiftobject191 APT on swiftobject191 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [20:15:24] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [20:15:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:36] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 128 packages available for upgrade (4 critical updates). [20:15:47] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (3 critical updates). [20:15:53] miraheze/CreateWiki - SomeMWDev the build passed. [20:21:01] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 8.03, 10.65, 11.96 [20:22:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763061740000&orgId=1&to=1763065370039 [20:29:35] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 [20:29:36] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.45/extensions/CreateWiki'} to test151 - SUCCESS in 0s [20:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:31:01] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 16.55, 11.38, 11.24 [20:31:35] [02CreateWiki] 07SomeMWDev merged pull request #789: Set doesWrites to true for both wiki request special pages (03main...03special-page-writes) 13https://github.com/miraheze/CreateWiki/pull/789 [20:31:35] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/16549e0715d8f5315817f20faa9713afd16305c4 [20:31:37] 02CreateWiki/03main 07SomeRandomDeveloper 0316549e0 Set doesWrites to true for both wiki request special pages (#789)… [20:31:39] [02CreateWiki] 07SomeMWDev 04deleted 03special-page-writes at 03f9c1035 13https://api.github.com/repos/miraheze/CreateWiki/commit/f9c1035 [20:31:45] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CreateWiki'} to test151 [20:31:46] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [20:31:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:32:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:04] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all [20:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.43, 19.11, 17.36 [20:33:35] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 31s [20:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:33:49] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.31, 18.29, 15.93 [20:34:19] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.15, 18.99, 16.80 [20:35:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.92, 18.36, 17.32 [20:35:49] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.80, 18.67, 16.38 [20:36:53] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [20:36:54] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [20:37:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:15] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.38, 19.59, 17.53 [20:41:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.33, 20.15, 18.30 [20:42:13] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.80, 19.19, 17.37 [20:44:10] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.97, 19.37, 17.65 [20:45:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.25, 19.16, 18.39 [20:49:06] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 20.85, 19.27, 17.41 [20:49:13] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.76, 10.19, 8.24 [20:50:55] miraheze/CreateWiki - SomeMWDev the build passed. [20:51:01] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.48, 18.07, 17.19 [20:51:12] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.36, 9.33, 8.16 [20:52:10] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.73, 20.14, 18.56 [20:52:50] miraheze/CreateWiki - SomeMWDev the build passed. [20:55:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/bdb829ae84b3edda8ce2963478a449ebe6cda8ca [20:55:24] 02ssl/03main 07WikiTideBot 03bdb829a Bot: Auto-update domain lists [20:55:52] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.61, 20.60, 18.86 [20:57:52] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.75, 19.86, 18.83 [20:57:52] [02CreateWiki] 07SomeMWDev merged 07lihaohong6's pull request #777: Improve dark mode styling (07miraheze:03main...07lihaohong6:03lihaohong6-patch-1) 13https://github.com/miraheze/CreateWiki/pull/777 [20:57:52] [02CreateWiki] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/b9d9a5fada6f338841f3c0f2fa2f85a4de759329 [20:57:53] 02CreateWiki/03main 07Peter Li 03b9d9a5f Improve dark mode styling (#777)… [20:58:00] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CreateWiki'} to test151 [20:58:02] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [20:58:05] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all [20:58:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:34] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 28s [20:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:36] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.06, 20.93, 19.66 [21:05:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.58, 20.30, 19.73 [21:08:10] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.18, 20.15, 19.90 [21:10:10] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.19, 20.91, 19.28 [21:10:27] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.94, 19.78, 19.65 [21:12:10] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.00, 19.99, 19.15 [21:15:07] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 22.55, 19.14, 17.06 [21:17:06] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 19.97, 19.51, 17.46 [21:19:56] miraheze/CreateWiki - SomeMWDev the build passed. [21:20:54] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.37, 11.80, 9.88 [21:22:21] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=0) [21:22:22] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=wixosswiki --update (END - exit=0) [21:22:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:40] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.57, 20.52, 19.03 [21:22:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:53] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.44, 11.61, 10.04 [21:24:10] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.17, 20.62, 20.12 [21:24:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.36, 20.34, 19.75 [21:26:15] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.09, 19.99, 19.71 [21:26:31] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.16, 20.08, 19.20 [21:26:51] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.47, 9.99, 9.71 [21:30:10] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 14.33, 18.53, 19.60 [21:31:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 9.92, 16.59, 19.29 [21:38:43] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.66, 11.04, 9.42 [21:40:10] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.42, 18.94, 18.03 [21:40:56] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.40, 18.82, 17.20 [21:41:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.84, 19.75, 19.24 [21:41:52] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.53, 19.37, 18.16 [21:42:10] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 20.21, 19.24, 18.25 [21:43:49] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.15, 19.56, 18.40 [21:44:40] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.09, 11.29, 10.41 [21:44:56] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.55, 19.07, 17.65 [21:45:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.84, 20.33, 19.63 [21:46:10] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.85, 20.02, 19.02 [21:48:38] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 9.20, 10.03, 10.06 [21:49:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.24, 21.40, 20.24 [21:49:49] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.50, 20.05, 18.91 [21:50:07] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.59, 20.85, 19.41 [21:53:49] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.09, 20.39, 19.34 [21:54:50] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.01, 20.72, 19.63 [21:55:59] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 19.44, 19.88, 19.48 [21:56:33] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.52, 12.20, 10.88 [21:56:47] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.06, 20.02, 19.52 [21:56:51] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.76, 21.10, 19.41 [21:58:47] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 17.74, 19.60, 19.06 [22:02:36] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.03, 21.07, 20.11 [22:07:52] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.77, 20.80, 20.03 [22:10:24] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.05, 20.21, 20.20 [22:11:52] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.28, 19.83, 19.86 [22:18:20] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 23.51, 20.46 [22:20:10] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 14.81, 19.34, 20.25 [22:20:20] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 12.60, 20.08, 19.63 [22:21:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.97, 18.48, 19.97 [22:21:52] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 4 backends are down. mw173 mw183 mw192 mw203 [22:22:35] @paladox [22:23:52] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [22:24:23] db171 is overloaded again for some reason [22:25:20] I started backups earlier after noticing them failing. But the load was higher before that [22:25:26] I really don’t know what’s caused that [22:26:25] [1/2] idk what happened to db171 because after I deployed my patch it was on load 1 for hours [22:26:26] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1438656277244018861/image.png?ex=6917ac91&is=69165b11&hm=2808264800625e7e6015312b72c1c31253bff0cfb238c35580dabda3cf0c7d78& [22:26:35] and since then it was constantly high load [22:26:45] but not the MF spikes anymore [22:27:57] Yeh higher queries as well and I don’t understand why [22:28:17] can you get a process list for db171 and share it on mattermost? [22:29:11] I had a look but it didn’t really have a notable query but it had a lot of dbs listed [22:29:24] hm [22:29:35] we should really consider a separate db server for mhglobal [22:29:41] we can't have individual wikis overloading the global db [22:34:23] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.51, 10.34, 11.75 [22:36:56] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.50, 20.23, 19.77 [22:40:52] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.41, 19.89, 19.76 [22:42:20] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.70, 19.97, 18.74 [22:42:23] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.62, 8.09, 10.08 [22:43:17] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.18, 20.93, 19.82 [22:44:20] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.72, 19.03, 18.58 [22:47:17] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 16.85, 19.18, 19.42 [22:55:31] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.01, 20.03, 19.72 [22:56:17] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [22:56:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:29] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 17.91, 19.69, 19.66 [22:58:01] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [22:58:02] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [22:58:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:42] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:00:43] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=wixosswiki --update (END - exit=2) [23:00:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:37] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:03:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:50] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [23:04:51] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:04:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:53] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:11:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:11] PROBLEM - db161 Check unit status of sql-backup on db161 is CRITICAL: CRITICAL: Status of the systemd unit sql-backup [23:14:15] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:14:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:29] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [23:15:30] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:15:35] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:15:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:36] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=wixosswiki --update (END - exit=2) [23:15:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:17:01] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.25, 9.64, 11.96 [23:20:30] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:20:32] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [23:20:33] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:20:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:37] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:20:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:06] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:22:08] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [23:22:09] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:22:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:10] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763072330000&orgId=1&to=1763075930000 [23:23:01] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 3.65, 6.19, 9.71 [23:26:25] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:26:27] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [23:26:28] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:26:29] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:26:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:25] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:28:27] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=256) [23:28:28] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:28:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:37] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (END - exit=2) [23:28:38] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/initSiteStats.php --wiki=wixosswiki --update (END - exit=0) [23:28:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:07] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (START) [23:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:28] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importDump.php --wiki=wixosswiki dump.xml --no-updates (END - exit=0) [23:30:29] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/rebuildall.php --wiki=wixosswiki (START) [23:30:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.50, 18.78, 16.52 [23:32:34] !log [void@puppet181] Upgraded packages on swiftproxy171: swift-proxy, swift, and python3-swift [23:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:55] !log [void@puppet181] Upgraded packages on swiftproxy161: swift-proxy, swift, and python3-swift [23:32:58] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.79, 10.09, 8.12 [23:33:01] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 16.22, 11.96, 10.53 [23:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:20] !log [void@puppet181] Upgraded packages on swiftobject161: swift-object, swift, and python3-swift [23:33:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:46] !log [void@puppet181] Upgraded packages on swiftobject191: swift-object, swift, and python3-swift [23:33:48] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:33:49] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.33, 19.03, 16.33 [23:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:59] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.60, 18.52, 16.69 [23:34:12] !log [void@puppet181] Upgraded packages on swiftobject171: swift-object, swift, and python3-swift [23:34:21] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [23:34:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:30] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.20, 19.35, 16.99 [23:34:34] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [23:34:39] !log [void@puppet181] Upgraded packages on swiftobject151: swift-object-expirer, swift-object, swift, and python3-swift [23:34:40] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:34:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:57] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.44, 9.22, 8.06 [23:35:07] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:35:13] !log [void@puppet181] Upgraded packages on swiftac171: swift-account, swift-container, swift, and python3-swift [23:35:17] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:35:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:40] !log [void@puppet181] Upgraded packages on swiftobject201: swift-object, swift, and python3-swift [23:35:44] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.44, 19.38, 17.55 [23:35:50] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.43, 19.00, 16.67 [23:35:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:58] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 20.23, 19.48, 17.29 [23:36:05] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 124 packages available for upgrade (0 critical updates). [23:36:08] !log [void@puppet181] Upgraded packages on swiftobject181: swift-object, swift, and python3-swift [23:36:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:15] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.59, 18.60, 17.02 [23:36:27] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 17.63, 19.16, 17.24 [23:37:01] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 9.94, 11.43, 10.72 [23:37:25] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:37:34] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [23:37:38] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.56, 18.87, 17.57 [23:42:10] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 20.58, 20.02, 17.66 [23:43:50] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.45, 20.21, 18.41 [23:44:10] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.43, 19.47, 17.72 [23:45:49] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 18.55, 19.67, 18.44 [23:47:01] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 7.24, 9.41, 10.15 [23:51:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw182 [23:53:36] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [23:53:45] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.69, 11.36, 9.57 [23:55:44] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.41, 10.38, 9.44 [23:55:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1763074520000&orgId=1&to=1763078150036 [23:57:43] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.88, 9.46, 9.21