[00:01:02] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/541755f46c05f58c7869958509d8e6b7ad7ec46a [00:01:02] 02statichelp/03main 07WikiTideBot 03541755f Bot: Auto-update Tech namespace pages 2026-06-30 00:00:59 [01:00:18] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/de02aa5eab6467723671d8209e4665c039c3f741 [01:00:18] 02ssl/03main 07WikiTideBot 03de02aa5 Bot: Auto-update domain lists [01:30:16] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/68223f5befd886820ce87f30cf73ab4171df22ff [01:30:16] 02ssl/03main 07WikiTideBot 0368223f5 Bot: Auto-update domain lists [02:00:19] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/804b55d34e7b9c5b758d15a0ed8f7819adbf6635 [02:00:19] 02ssl/03main 07WikiTideBot 03804b55d Bot: Auto-update domain lists [03:30:17] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/59205d62346b0fccf71e449fae5451fa597e03c3 [03:30:17] 02ssl/03main 07WikiTideBot 0359205d6 Bot: Auto-update domain lists [04:05:55] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CreateWiki:DeleteWikis --wiki=loginwiki --delete Reception123 (END - exit=0) [04:05:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:44] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php /srv/mediawiki/1.45/maintenance/MirahezeMagic:CheckWikiDatabases.php --wiki=loginwiki --delete (END - exit=256) [06:08:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:09:40] !log begin dropping previously deleted wikis [06:09:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782799340000&orgId=1&to=1782802970026 [07:57:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782799340000&orgId=1&to=1782806030000 [08:39:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782805160000&orgId=1&to=1782808790028 [09:03:35] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:31:35] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:42:31] !log [reception@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php MirahezeMagic:CheckWikiDatabases --wiki=loginwiki --delete (END - exit=256) [09:42:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:49:19] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.77, 17.96, 13.31 [09:49:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782805160000&orgId=1&to=1782812990000 [09:51:19] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 9.41, 14.85, 12.76 [10:09:11] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.39, 20.46, 16.89 [10:13:10] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.35, 19.96, 17.52 [10:37:38] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.49, 17.97, 15.14 [10:39:11] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.37, 19.68, 16.78 [10:39:18] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.36, 17.99, 15.20 [10:39:24] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.53, 17.83, 14.42 [10:40:12] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.97, 19.26, 15.85 [10:41:18] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 20.14, 18.84, 15.86 [10:41:24] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 16.19, 17.53, 14.76 [10:41:38] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.26, 19.84, 16.68 [10:42:12] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 16.27, 18.33, 15.95 [10:43:11] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 14.12, 18.00, 16.90 [11:45:21] [02puppet] 07dependabot[bot] created 03dependabot/bundler/modules/graylog/faraday-2.14.3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/bf4e047812f8 [11:45:21] 02puppet/03dependabot/bundler/modules/graylog/faraday-2.14.3 07dependabot[bot] 03bf4e047 build(deps-dev): bump faraday from 2.12.2 to 2.14.3 in /modules/graylog… [11:45:22] [02puppet] 07dependabot[bot] added the label 'dependencies' to pull request #4883 (build(deps-dev): bump faraday from 2.12.2 to 2.14.3 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4883 [11:45:23] [02puppet] 07dependabot[bot] added the label 'ruby' to pull request #4883 (build(deps-dev): bump faraday from 2.12.2 to 2.14.3 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4883 [11:45:25] [02puppet] 07dependabot[bot] opened pull request #4883: build(deps-dev): bump faraday from 2.12.2 to 2.14.3 in /modules/graylog (03main...03dependabot/bundler/modules/graylog/faraday-2.14.3) 13https://github.com/miraheze/puppet/pull/4883 [11:45:27] [02puppet] 07dependabot[bot] added the label 'dependencies' to pull request #4883 (build(deps-dev): bump faraday from 2.12.2 to 2.14.3 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4883 [11:45:29] [02puppet] 07dependabot[bot] added the label 'ruby' to pull request #4883 (build(deps-dev): bump faraday from 2.12.2 to 2.14.3 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4883 [14:08:43] [02puppet] 07dependabot[bot] created 03dependabot/bundler/modules/graylog/yard-0.9.44 (+1 new commit) 13https://github.com/miraheze/puppet/commit/cf119d945730 [14:08:43] 02puppet/03dependabot/bundler/modules/graylog/yard-0.9.44 07dependabot[bot] 03cf119d9 build(deps-dev): bump yard from 0.9.36 to 0.9.44 in /modules/graylog… [14:08:44] [02puppet] 07dependabot[bot] added the label 'dependencies' to pull request #4884 (build(deps-dev): bump yard from 0.9.36 to 0.9.44 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4884 [14:08:44] [02puppet] 07dependabot[bot] added the label 'ruby' to pull request #4884 (build(deps-dev): bump yard from 0.9.36 to 0.9.44 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4884 [14:08:46] [02puppet] 07dependabot[bot] opened pull request #4884: build(deps-dev): bump yard from 0.9.36 to 0.9.44 in /modules/graylog (03main...03dependabot/bundler/modules/graylog/yard-0.9.44) 13https://github.com/miraheze/puppet/pull/4884 [14:08:48] [02puppet] 07dependabot[bot] added the label 'dependencies' to pull request #4884 (build(deps-dev): bump yard from 0.9.36 to 0.9.44 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4884 [14:08:50] [02puppet] 07dependabot[bot] added the label 'ruby' to pull request #4884 (build(deps-dev): bump yard from 0.9.36 to 0.9.44 in /modules/graylog) 13https://github.com/miraheze/puppet/pull/4884 [14:12:29] [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-13.2.2 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/9ed727dff53a [14:12:29] 02mw-config/03dependabot/composer/phpunit/phpunit-13.2.2 07dependabot[bot] 039ed727d Update phpunit/phpunit requirement from 13.2.0 to 13.2.2… [14:12:30] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6451 (Update phpunit/phpunit requirement from 13.2.0 to 13.2.2) 13https://github.com/miraheze/mw-config/pull/6451 [14:12:30] [02mw-config] 07dependabot[bot] opened pull request #6451: Update phpunit/phpunit requirement from 13.2.0 to 13.2.2 (03main...03dependabot/composer/phpunit/phpunit-13.2.2) 13https://github.com/miraheze/mw-config/pull/6451 [14:12:32] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6451 (Update phpunit/phpunit requirement from 13.2.0 to 13.2.2) 13https://github.com/miraheze/mw-config/pull/6451 [14:12:34] [02mw-config] 07dependabot[bot] commented on pull request #6440: Superseded by #6451. 13https://github.com/miraheze/mw-config/pull/6440#issuecomment-4844413394 [14:12:36] [02mw-config] 07dependabot[bot] closed pull request #6440: Update phpunit/phpunit requirement from 13.2.0 to 13.2.1 (03main...03dependabot/composer/phpunit/phpunit-13.2.1) 13https://github.com/miraheze/mw-config/pull/6440 [14:12:38] [02mw-config] 07dependabot[bot] 04deleted 03dependabot/composer/phpunit/phpunit-13.2.1 at 03ce6e282 13https://api.github.com/repos/miraheze/mw-config/commit/ce6e282 [14:14:09] miraheze/mw-config - dependabot[bot] the build passed. [16:59:18] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.08, 19.57, 15.67 [16:59:24] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 23.10, 17.94, 13.51 [17:01:11] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.15, 19.24, 15.47 [17:01:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.17, 19.47, 14.81 [17:03:24] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 18.20, 19.01, 15.02 [17:03:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.75, 20.15, 15.63 [17:04:12] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.35, 19.76, 15.69 [17:05:11] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 25.66, 22.10, 17.47 [17:05:12] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.30, 20.99, 16.61 [17:05:59] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.66, 19.89, 16.08 [17:06:36] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.64, 20.34, 16.84 [17:08:09] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.51, 21.02, 16.84 [17:09:10] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.50, 22.88, 18.87 [17:09:20] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.81, 21.25, 17.50 [17:09:24] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.63, 20.87, 17.07 [17:09:25] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 20.70, 20.13, 16.87 [17:10:09] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.48, 19.11, 16.67 [17:11:18] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 24.25, 23.20, 19.82 [17:11:28] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.66, 18.51, 15.00 [17:13:15] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 17.40, 19.45, 17.36 [17:13:18] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 23.46, 23.23, 20.25 [17:13:24] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 15.75, 19.19, 17.35 [17:13:28] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.96, 18.38, 15.39 [17:14:23] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.79, 20.01, 18.20 [17:14:42] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.69, 19.11, 18.06 [17:17:02] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.88, 20.34, 18.72 [17:17:11] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.59, 20.36, 19.38 [17:19:24] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 20.42, 20.47, 18.50 [17:19:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.91, 20.36, 18.32 [17:20:15] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.53, 20.52, 18.97 [17:20:30] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.20, 21.15, 19.20 [17:20:55] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.68, 21.37, 19.49 [17:20:57] PROBLEM - mw153 Current Load on mw153 is WARNING: LOAD WARNING - total load average: 23.86, 21.17, 18.72 [17:20:57] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.66, 20.88, 17.94 [17:21:10] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.24, 21.78, 20.17 [17:21:59] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.25, 19.59, 18.26 [17:22:56] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 23.31, 21.54, 18.84 [17:24:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782836630000&orgId=1&to=1782840260031 [17:24:47] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.84, 20.08, 18.32 [17:24:55] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.73, 20.89, 18.80 [17:25:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.11, 20.51, 18.97 [17:28:39] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.04, 21.83, 19.44 [17:29:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.29, 21.02, 18.31 [17:29:29] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.15, 19.73, 17.51 [17:30:01] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.41, 19.63, 16.37 [17:31:19] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.41, 19.68, 17.43 [17:31:33] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 23.10, 20.97, 18.81 [17:31:38] uh this doesn't look good. idk what to do about it [17:31:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.89, 23.40, 20.67 [17:32:01] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.38, 18.59, 16.35 [17:32:30] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 23.42, 21.52 [17:33:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.47, 22.85, 20.80 [17:34:30] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.21, 23.42, 21.76 [17:34:38] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 23.29, 20.99 [17:34:53] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.90, 21.05, 18.03 [17:35:12] [1/2] https://cdn.discordapp.com/attachments/808001911868489748/1521569807453913199/image.jpg?ex=6a454fcf&is=6a43fe4f&hm=88be3f4fe6d591fca32b037434025ed5db9b29025803edf47bcefde4c25b9f10& [17:35:12] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1521569808234188871/image.jpg?ex=6a454fcf&is=6a43fe4f&hm=757e7d27b43b65336b4fca0920d2b146962ecc642e1ac8cd9c2a9b51f7a05e13& [17:35:28] <_arawynn> @Discord Moderators [17:36:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.59, 23.38, 21.29 [17:40:53] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.69, 19.93, 18.75 [17:41:59] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.04, 19.31, 20.15 [17:43:01] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 18.76, 19.97, 19.83 [17:43:08] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.71, 18.87, 19.33 [17:43:19] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.42, 18.53, 18.63 [17:43:28] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.50, 18.42, 18.91 [17:43:51] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 10.58, 17.25, 19.96 [17:44:02] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 11.56, 17.30, 20.05 [17:44:11] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 12.21, 17.62, 19.78 [17:44:12] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 12.42, 17.28, 19.94 [17:44:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.13, 17.10, 19.33 [17:44:21] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.24, 18.68, 20.19 [17:44:49] RECOVERY - mw153 Current Load on mw153 is OK: LOAD OK - total load average: 10.73, 16.09, 19.26 [17:45:11] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 13.55, 17.67, 20.30 [17:45:24] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 14.57, 17.05, 19.52 [17:48:30] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 10.75, 16.13, 19.37 [17:49:18] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 12.97, 16.53, 19.77 [17:50:02] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw181 mw153 mw201 mw203 [17:50:44] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [17:51:14] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [17:52:52] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.239 second response time [17:53:16] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.822 second response time [17:54:50] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw161 mw201 mw203 [17:55:11] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw181 [17:55:29] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [17:55:30] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [17:56:34] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [17:56:51] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [17:57:07] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [17:57:18] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [17:57:19] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw163 [17:57:21] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [17:57:42] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [17:58:37] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 6.812 second response time [17:58:48] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1485 bytes in 2.125 second response time [17:59:00] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [17:59:15] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.626 second response time [17:59:25] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.611 second response time [17:59:34] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.919 second response time [17:59:36] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 6.487 second response time [17:59:45] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.324 second response time [17:59:52] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43% [18:00:02] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [18:00:37] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:01:07] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 8.104 second response time [18:01:19] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [18:01:49] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 29% [18:04:02] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw193 mw201 [18:05:19] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw191 mw203 [18:07:06] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:09:08] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 8.172 second response time [18:10:02] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [18:10:42] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [18:11:08] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.329 second response time [18:13:19] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [18:14:13] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw161 [18:14:39] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 3 backends are down. mw191 mw193 mw203 [18:16:08] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:16:35] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [18:23:04] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:23:26] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 6 backends are down. mw153 mw163 mw191 mw193 mw201 mw203 [18:23:49] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 6 backends are down. mw161 mw181 mw153 mw191 mw193 mw201 [18:24:02] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw181 mw163 [18:24:11] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [18:25:06] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 8.068 second response time [18:26:15] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44% [18:27:40] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:28:12] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.219 second response time [18:28:15] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 23% [18:30:27] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [18:31:31] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 2 backends are down. mw181 mw163 [18:31:44] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.013 second response time [18:31:50] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw181 mw153 mw203 [18:32:29] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.342 second response time [18:33:08] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [18:33:26] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:33:35] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [18:35:58] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [18:36:44] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:03] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw191 mw193 [18:37:17] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw161 mw191 mw201 [18:37:30] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:38:00] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:38:02] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:38:05] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 8.588 second response time [18:38:34] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.012 second response time [18:38:59] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [18:39:12] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:39:30] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [18:39:30] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.304 second response time [18:39:37] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [18:39:57] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 1463 bytes in 0.082 second response time [18:40:00] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.395 second response time [18:40:02] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [18:40:33] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [18:40:44] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.519 second response time [18:50:35] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.55, 10.07, 8.52 [18:50:44] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw161 [18:51:19] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw163 [18:52:00] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:52:25] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [18:52:34] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 9.29, 9.80, 8.61 [18:52:47] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw203 [18:54:05] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.405 second response time [18:54:20] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.703 second response time [18:54:37] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [18:54:43] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [18:55:19] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [19:01:28] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 10.85, 10.03, 9.09 [19:03:27] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 6.48, 9.04, 8.86 [19:10:27] !log DELETED and DROPPED https://issue-tracker.miraheze.org/P597 [19:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:21] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 18.03, 11.26, 9.43 [19:15:20] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 8.27, 9.81, 9.11 [19:17:40] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [19:18:11] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.010 second response time [19:19:40] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 5.358 second response time [19:20:13] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.044 second response time [19:22:35] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5857 bytes in 0.011 second response time [19:24:40] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.080 second response time [19:24:47] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw163 [19:26:13] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 18.96, 13.72, 11.07 [19:28:12] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 6.92, 11.25, 10.52 [19:28:39] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [19:31:35] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:32:10] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 6.24, 8.48, 9.58 [21:12:13] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.45', '1.46'], 'upgrade_extensions': 'Chart'} to test151 [21:12:15] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.45', '1.46'], 'upgrade_extensions': 'Chart'} to test151 - SUCCESS in 1s [21:12:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:29] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'Chart'} to all [21:12:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:08] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'Chart'} to all - SUCCESS in 38s [21:13:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782836630000&orgId=1&to=1782854660065 [21:33:27] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (2 critical updates). [21:33:34] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (2 critical updates). [21:33:44] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:34:43] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 116 packages available for upgrade (2 critical updates). [21:34:53] PROBLEM - mw201 APT on mw201 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:35:23] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (2 critical updates). [21:36:33] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:36:40] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:37:19] PROBLEM - mw183 APT on mw183 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:38:02] PROBLEM - mattermost2 APT on mattermost2 is CRITICAL: APT CRITICAL: 76 packages available for upgrade (2 critical updates). [21:38:40] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 79 packages available for upgrade (2 critical updates). [21:39:27] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (2 critical updates). [21:40:32] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:40:32] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:41:50] PROBLEM - mw202 APT on mw202 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:42:12] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 82 packages available for upgrade (2 critical updates). [21:42:29] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:42:57] PROBLEM - mw203 APT on mw203 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:43:03] PROBLEM - llm191 APT on llm191 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (2 critical updates). [21:43:10] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 115 packages available for upgrade (2 critical updates). [21:44:18] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:44:36] PROBLEM - mw173 APT on mw173 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:44:40] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:44:52] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 6 packages available for upgrade (2 critical updates). [21:45:57] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 80 packages available for upgrade (2 critical updates). [21:48:35] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (2 critical updates). [21:49:46] PROBLEM - mw193 APT on mw193 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:50:25] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 12.70, 9.84, 7.62 [21:51:55] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:52:10] PROBLEM - mw153 APT on mw153 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:53:46] PROBLEM - mw191 APT on mw191 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:54:24] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 10.49, 10.30, 8.30 [21:57:23] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw161 mw163 mw193 mw201 [21:57:34] PROBLEM - cp161 APT on cp161 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (2 critical updates). [21:58:09] PROBLEM - mw192 APT on mw192 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:58:21] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (2 critical updates). [21:58:23] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 6.27, 9.53, 8.53 [21:59:31] PROBLEM - mw163 APT on mw163 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:59:53] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 78 packages available for upgrade (2 critical updates). [21:59:54] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 114 packages available for upgrade (2 critical updates). [21:59:59] PROBLEM - os202 APT on os202 is CRITICAL: APT CRITICAL: 80 packages available for upgrade (2 critical updates). [22:01:13] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [22:08:57] <4eva_metal> [1/2] https://cdn.discordapp.com/attachments/808001911868489748/1521638701913079948/image.jpeg?ex=6a458ff9&is=6a443e79&hm=8ce04444a1118c53dda87ec15c6e8d3720b71fb064e29b4704e885f2af367aec& [22:08:58] <4eva_metal> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1521638702575521853/image.jpeg?ex=6a458ff9&is=6a443e79&hm=4437a5ef6cb7924cafa57ec7945ae4e6a7f76a201d12f5b5ebb9a25ffdb2eb84& [23:44:20] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1782836630000&orgId=1&to=1782862880000