[00:04:46] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[00:04:50] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 57726MiB (12% inode=99%);
[00:05:34] <icinga-miraheze>	 RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 59258MiB (13% inode=99%);
[00:12:33] <MacFanBot>	 [02CreateWiki] 07Universal-Omega merged pull request #693: Add test for CheckLastWikiActivity (03master...03CheckLastWikiActivity-test) 13https://github.com/miraheze/CreateWiki/pull/693
[00:12:33] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/31665ea1ac3207177516010c681be6f61e5cfb65
[00:12:33] <MacFanBot>	 02CreateWiki/03master 07CosmicAlpha 0331665ea Add test for CheckLastWikiActivity (#693)
[00:12:34] <MacFanBot>	 [02CreateWiki] 07Universal-Omega 04deleted 03CheckLastWikiActivity-test at 03b445539 13https://github.com/miraheze/CreateWiki/commit/b445539
[00:24:25] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[00:33:52] <MacFanBot>	 [02CreateWiki] 07Universal-Omega created 03ManageInactiveWikis-test (+1 new commit) 13https://github.com/miraheze/CreateWiki/commit/de1bc53d2a72
[00:33:52] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03de1bc53 Add test for ManageInactiveWikis
[00:33:55] <MacFanBot>	 [02CreateWiki] 07Universal-Omega opened pull request #694: Add test for ManageInactiveWikis (03master...03ManageInactiveWikis-test) 13https://github.com/miraheze/CreateWiki/pull/694
[00:34:02] <MacFanBot>	 [02CreateWiki] 07coderabbitai[bot] commented on pull request #694: --- […] 13https://github.com/miraheze/CreateWiki/pull/694#issuecomment-2699352711
[00:34:24] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/a96991d03c3ac9c7a8d2db05c0816a3d7db1c02f
[00:34:24] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03a96991d Move
[00:36:12] <MacFanBot>	 [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/5d352bffe0148b73f8f81e02277ad283f63aeca7
[00:36:12] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07github-actions 035d352bf CI: lint code to MediaWiki standards…
[00:36:55] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/48ea940dbf4c8bb21dd8d2e488f6dac227420901
[00:36:55] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 0348ea940 -
[00:40:12] <MacFanBot>	 [02mw-config] 07BlankEclair opened pull request #5846: T13303: Force all Echo events to be a notice for softcellwiki (07miraheze:03master...07BlankEclair:03T13303) 13https://github.com/miraheze/mw-config/pull/5846
[00:40:36] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[00:40:38] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/ebce55a9dcfa8b331f6117ea887e3f052793b163
[00:40:39] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03ebce55a Split line
[00:41:03] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[00:41:14] <MacFanBot>	 [02mw-config] 07BlankEclair merged 07enews's pull request #5845: Add discord URL protool to cafe wiki  (07miraheze:03master...07enews:03patch-1) 13https://github.com/miraheze/mw-config/pull/5845
[00:41:14] <MacFanBot>	 [02mw-config] 07BlankEclair pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/5f959d4e4e9fdbfd425a774182866f96a170f115
[00:41:14] <MacFanBot>	 02mw-config/03master 07enews 035f959d4 Add discord URL protool to cafe wiki  (#5845)…
[00:41:38] <MacFanBot>	 [02mw-config] 07BlankEclair merged pull request #5846: T13303: Force all Echo events to be a notice for softcellwiki (07miraheze:03master...07BlankEclair:03T13303) 13https://github.com/miraheze/mw-config/pull/5846
[00:41:40] <MacFanBot>	 [02mw-config] 07BlankEclair pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/a7a961e5b20afb17c8787d4198fd234111869cb0
[00:41:42] <MacFanBot>	 02mw-config/03master 07Claire 03a7a961e T13303: Force all Echo events to be a notice for softcellwiki (#5846)…
[00:42:29] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[00:44:09] <MacFanBot>	 [02CreateWiki] 07BlankEclair approved pull request #694 13https://github.com/miraheze/CreateWiki/pull/694#pullrequestreview-2659534951
[00:44:57] <MacFanBot>	 [02CreateWiki] 07codecov[bot] commented on pull request #694: All modified and coverable lines are covered by tests :white_check_mark: […] 13https://github.com/miraheze/CreateWiki/pull/694#issuecomment-2699372578
[00:51:25] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[01:10:16] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/87335d3180e06e5612d25a6973494bda91ed27df
[01:10:16] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 0387335d3 Fix
[01:17:35] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/e75437119a1a2adc56ef7c7f098dd313a19809ca
[01:17:35] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03e754371 Fix
[01:19:16] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/setupStore.php --wiki=oversimplifiednrpwiki (END - exit=256)
[01:19:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:19:31] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=oversimplifiednrpwiki (END - exit=0)
[01:19:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:19:39] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aokpwiki (END - exit=0)
[01:19:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:20:23] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/564d96fb175bd1d1a00e96bc26a984353df860e6
[01:20:24] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03564d96f Fix
[01:30:23] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/05a94d0b46e747ef7dd4388bedf4afb85908aba5
[01:30:23] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 0305a94d0 Fix
[01:31:48] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/52102aaae54af133b70dca997e5dba40c365cdb0
[01:31:48] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 0352102aa Fix
[01:37:07] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/9b64b59476a562204dceb607376c0bf441cebab8
[01:37:07] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 039b64b59 Fix
[01:39:07] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/5ad6a453edbf2924077a2dfb96bdf80d79078a41
[01:39:07] <MacFanBot>	 02CreateWiki/03master 07CosmicAlpha 035ad6a45 Replace setMwGlobals with overrideConfigValue
[01:44:05] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/b1be6b569d1cc6ee849b40813e02ba7b4fe89619
[01:44:05] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03b1be6b5 Fix
[01:45:38] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/d4b42fd5f529aa9790c2b5f1c7ef2229cbae1345
[01:45:38] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03d4b42fd Fix
[01:50:14] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/d0c7574d324be65805521a430cc1eab70835d80c
[01:50:14] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03d0c7574 Remove path
[01:50:31] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/26802ca62e0e6d71a23d08ddbe62ec4c9110aa16
[01:50:31] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 0326802ca Merge branch 'master' into ManageInactiveWikis-test
[01:51:00] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[01:57:17] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/e18dc55e613fc7b192c9c0756ad2a653ea288119
[01:57:17] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03e18dc55 Fix
[02:07:59] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[02:13:13] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/33e46ed11763b6b2a0172158328e63fb13a88e8d
[02:13:13] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 0333e46ed Fix
[02:13:40] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/a5a9ff975e1cf00c05ae4c0dbe5385a67d22cbb5
[02:13:40] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03a5a9ff9 Fix
[02:18:12] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/9b086b0cc41df3839fe673a7083e8da498350ac3
[02:18:12] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 039b086b0 Fix
[02:29:25] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[02:32:00] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/bcef937f54677581d2011ea47738207c2d55e8c3
[02:32:00] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03bcef937 Fix
[02:40:33] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/0868299e5adc09c65e56cd2200154b48f1d7cb18
[02:40:33] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 030868299 Fix
[02:48:21] <MacFanBot>	 [02CreateWiki] 07Universal-Omega pushed 1 new commit to 03ManageInactiveWikis-test 13https://github.com/miraheze/CreateWiki/commit/f4b74f7a648a4184e05e6dbb30023e4aadb36139
[02:48:21] <MacFanBot>	 02CreateWiki/03ManageInactiveWikis-test 07CosmicAlpha 03f4b74f7 Fix
[02:59:12] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[03:34:14] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.73, 14.17, 6.38
[03:36:14] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 6.39, 11.60, 6.45
[03:50:14] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.39, 19.00, 13.08
[03:54:14] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 7.26, 15.92, 13.43
[04:01:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.72, 17.31, 9.58
[04:03:15] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.03, 12.86, 8.93
[04:05:53] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.41, 13.55, 6.73
[04:17:53] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 4.16, 14.55, 12.33
[04:44:42] <MacFanBot>	 [02mw-config] 07BlankEclair opened pull request #5847: T13303: Make every Echo event an alert on softcellwiki (07miraheze:03master...07BlankEclair:03T13303-2) 13https://github.com/miraheze/mw-config/pull/5847
[04:45:26] <MacFanBot>	 [02mw-config] 07BlankEclair merged pull request #5847: T13303: Make every Echo event an alert on softcellwiki (07miraheze:03master...07BlankEclair:03T13303-2) 13https://github.com/miraheze/mw-config/pull/5847
[04:45:28] <MacFanBot>	 [02mw-config] 07BlankEclair pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/3094def56b5e05e54d68bc04f2dbc70eec5ad334
[04:45:30] <MacFanBot>	 02mw-config/03master 07Claire 033094def T13303: Make every Echo event an alert on softcellwiki (#5847)…
[04:45:36] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[04:46:19] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[04:55:31] <MirahezeLSBot>	 !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[04:55:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:55:50] <MirahezeRelay>	 <agentisai> literally 1984
[04:55:53] <MirahezeLSBot>	 !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s
[04:55:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:55:59] <MirahezeRelay>	 <agentisai> 1985
[05:45:49] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[05:46:00] <icinga-miraheze>	 PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[05:46:01] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[05:46:03] <icinga-miraheze>	 PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[05:46:04] <icinga-miraheze>	 PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:46:05] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:46:09] <icinga-miraheze>	 PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[05:46:09] <icinga-miraheze>	 PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:46:14] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[05:46:14] <icinga-miraheze>	 PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[05:46:19] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[05:46:29] <icinga-miraheze>	 PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[05:46:30] <icinga-miraheze>	 PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[05:46:31] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:46:35] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[05:46:43] <icinga-miraheze>	 PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:46:50] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:46:52] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:46:52] <icinga-miraheze>	 PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:46:56] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:46:58] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki
[05:47:07] <icinga-miraheze>	 PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[05:47:13] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 272.87, 120.16, 47.62
[05:47:14] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:20] <icinga-miraheze>	 PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:47:24] <icinga-miraheze>	 PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[05:47:25] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:25] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 79%
[05:47:27] <icinga-miraheze>	 PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:31] <icinga-miraheze>	 PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:47:32] <icinga-miraheze>	 PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 42%
[05:47:34] <icinga-miraheze>	 PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:38] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:39] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:40] <icinga-miraheze>	 PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:41] <icinga-miraheze>	 PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:46] <icinga-miraheze>	 PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[05:47:46] <icinga-miraheze>	 PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[05:47:47] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[05:47:50] <icinga-miraheze>	 PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:51] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:47:54] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki
[05:47:55] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[05:47:59] <icinga-miraheze>	 PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[05:48:01] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki
[05:48:20] <icinga-miraheze>	 PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: <Timeout exceeded.><Terminated by signal 15 (Terminated).><Terminated with exit code 128 (0x80).>
[05:49:22] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is UNKNOWN: <Timeout exceeded.><Terminated by signal 15 (Terminated).><Terminated with exit code 128 (0x80).>
[05:49:32] <icinga-miraheze>	 RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 38%
[05:50:04] <icinga-miraheze>	 RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time
[05:50:10] <icinga-miraheze>	 RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.050 second response time
[05:50:11] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.151 second response time
[05:50:12] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 3.210 second response time
[05:50:17] <icinga-miraheze>	 RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 6.977 second response time
[05:50:17] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.153 second response time
[05:50:18] <icinga-miraheze>	 RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.181 second response time
[05:50:22] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.059 second response time
[05:50:27] <icinga-miraheze>	 RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.064 second response time
[05:50:36] <icinga-miraheze>	 RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.056 second response time
[05:50:38] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.226 second response time
[05:50:42] <icinga-miraheze>	 RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time
[05:50:44] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time
[05:50:50] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time
[05:50:55] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time
[05:50:56] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.056 second response time
[05:50:57] <icinga-miraheze>	 RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.058 second response time
[05:50:58] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy
[05:51:25] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 25%
[05:51:33] <icinga-miraheze>	 RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 5.659 second response time
[05:51:33] <icinga-miraheze>	 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 6.021 second response time
[05:51:33] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.660 second response time
[05:51:36] <icinga-miraheze>	 RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.853 second response time
[05:51:38] <icinga-miraheze>	 RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.054 second response time
[05:51:39] <icinga-miraheze>	 RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.154 second response time
[05:51:40] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is CRITICAL: Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1040 - Too many connections
[05:51:40] <icinga-miraheze>	 RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.165 second response time
[05:51:45] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.059 second response time
[05:51:45] <icinga-miraheze>	 PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:51:46] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.149 second response time
[05:51:46] <icinga-miraheze>	 RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time
[05:51:47] <icinga-miraheze>	 RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time
[05:51:48] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.169 second response time
[05:51:48] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.052 second response time
[05:51:51] <icinga-miraheze>	 RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.058 second response time
[05:51:51] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.054 second response time
[05:51:58] <icinga-miraheze>	 RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.159 second response time
[05:52:00] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.153 second response time
[05:52:01] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[05:52:01] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time
[05:52:17] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 6.962 second response time
[05:52:32] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:53:16] <icinga-miraheze>	 RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.053 second response time
[05:53:27] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time
[05:53:54] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy
[05:54:12] <icinga-miraheze>	 RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.058 second response time
[06:24:22] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.93, 17.00, 10.04
[06:26:08] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is UNKNOWN: <Timeout exceeded.><Terminated by signal 15 (Terminated).><Terminated with exit code 128 (0x80).>
[06:26:22] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.74, 17.36, 11.05
[06:27:46] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 57 minutes ago with 0 failures
[06:28:06] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is CRITICAL: Can't connect to server on 'db181.wikitide.net' (115)
[06:28:43] <icinga-miraheze>	 RECOVERY - db181 APT on db181 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[06:28:44] <MirahezeLSBot>	 !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[06:28:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:29:05] <MirahezeLSBot>	 !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s
[06:29:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:30:01] <icinga-miraheze>	 RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.3%Current connections: 3
[06:30:05] <icinga-miraheze>	 RECOVERY - db181 MariaDB on db181 is OK: Uptime: 126 Threads: 2 Questions: 90494 Slow queries: 0 Opens: 7272 Open tables: 7264 Queries per second avg: 718.206
[07:27:02] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.17, 0.34, 11.63
[07:31:02] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.38, 0.33, 9.04
[08:09:19] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 6.94, 12.57, 6.67
[08:11:19] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 1.48, 8.64, 5.94
[08:27:30] <MirahezeLSBot>	 !log [blankeclair@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[08:27:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:27:52] <MirahezeLSBot>	 !log [blankeclair@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s
[08:27:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:00:04] <MirahezeLSBot>	 !log [universalomega@db181] BlankEclair: you can bring up c4
[11:00:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:00:13] <BlankEclair>	 ohey, backups done
[11:47:21] <RhinosF1>	 Oh hey BlankEclair
[11:47:24] <RhinosF1>	 All good now?
[11:47:30] <BlankEclair>	 should be yeah
[11:47:46] <RhinosF1>	 Hey
[11:47:48] <RhinosF1>	 Yey
[11:47:52] <RhinosF1>	 Also fuck git
[11:54:39] <MirahezeRelay>	 <blankeclair, replying to RhinosF1> oh?
[12:07:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.94, 7.07, 3.17
[12:09:34] <icinga-miraheze>	 PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49939MiB (10% inode=99%);
[12:09:47] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 8.81, 7.53, 3.81
[12:13:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.00, 17.23, 9.11
[12:21:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.70, 11.33, 9.47
[12:26:36] <MirahezeRelay>	 <rhinosf1, replying to blankeclair> Bad conflict
[12:27:47] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.49, 9.18, 9.13
[12:31:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 6.20, 11.73, 10.50
[12:33:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 26.05, 16.54, 12.32
[12:43:53] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.60, 16.50, 8.69
[12:45:53] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 20.34, 17.75, 10.10
[12:49:53] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 21.71, 19.92, 12.72
[12:51:47] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.92, 8.73, 11.63
[12:53:53] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 7.95, 16.91, 13.40
[12:55:47] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.97, 4.38, 9.17
[13:21:02] <icinga-miraheze>	 PROBLEM - cloud20 IPMI Sensors on cloud20 is UNKNOWN: ID | Date | Time | Name | Type | State | Event1 | Feb-27-2025 | 00:26:48 | SEL | Event Logging Disabled | Nominal | Log Area Reset/Cleared2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 8
[13:21:03] <icinga-miraheze>	  91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 ipmi_sel_parse: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[13:23:03] <icinga-miraheze>	 PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [79 system event log (SEL) entries present]
[13:49:02] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 14.62, 28.30, 13.64
[13:50:33] <MirahezeRelay>	 <zppix, replying to icinga-miraheze> This doesnt look like a good alert
[13:55:02] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.14, 8.59, 9.30
[14:04:50] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49880MiB (10% inode=99%);
[14:19:09] <MirahezeRelay>	 <rhinosf1, replying to zppix> @orduin @cosmicalpha
[15:05:41] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 29.54, 18.98, 8.41
[15:21:43] <MacFan4000>	 Not sure which alert you’re referring to, but if it’s IPMI, that’s known (inlet temp issue) and IIRC the DC will be taking a look at that during the maintenance
[15:29:41] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 1.54, 16.46, 20.44
[15:30:14] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 29.18, 15.37, 6.50
[15:31:41] <icinga-miraheze>	 RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 0.49, 11.12, 17.99
[15:34:14] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.67, 22.12, 11.43
[15:36:14] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 3.55, 14.94, 10.09
[15:39:41] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 30.44, 21.63, 19.21
[16:14:55] <icinga-miraheze>	 PROBLEM - familiacorsi.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'familiacorsi.com' expires in 14 day(s) (Thu 20 Mar 2025 03:46:41 PM GMT +0000).
[16:15:12] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/30c9c7361dce08ee364f95d5835fb458eba9de63
[16:15:12] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 0330c9c73 Bot: Update SSL cert for familiacorsi.com
[16:15:41] <icinga-miraheze>	 PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 1.71, 15.58, 21.98
[16:17:41] <icinga-miraheze>	 RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 0.50, 10.56, 19.37
[16:41:36] <MacFan4000>	 !log (staffwiki) UPDATE echo_notification SET notification_read_timestamp=20250204083431 WHERE notification_user=24 and notification_event=215;
[16:41:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:44:25] <MacFan4000>	 !log (metawiki) DELETE FROM echo_unread_wikis WHERE euw_user=9736 AND euw_wiki="staffwiki";
[16:44:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:49:02] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/69ee6d757e0b895557a71ca89e6e5c30aff81c2a
[16:49:02] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 0369ee6d7 Bot: Update SSL cert for familiacorsi.com
[16:59:47] <icinga-miraheze>	 PROBLEM - echoes-wiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'echoes-wiki.com' expires in 14 day(s) (Thu 20 Mar 2025 04:41:07 PM GMT +0000).
[17:00:37] <icinga-miraheze>	 PROBLEM - kagaga.jp - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kagaga.jp' expires in 14 day(s) (Thu 20 Mar 2025 04:51:28 PM GMT +0000).
[17:00:48] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/c7259ffa4b41b84997ac33a1bc2c149c4ab4a5fa
[17:00:48] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 03c7259ff Bot: Update SSL cert for kagaga.jp
[17:13:45] <icinga-miraheze>	 RECOVERY - familiacorsi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'familiacorsi.com' will expire on Tue 03 Jun 2025 03:16:31 PM GMT +0000.
[17:29:15] <icinga-miraheze>	 RECOVERY - kagaga.jp - LetsEncrypt on sslhost is OK: OK - Certificate 'kagaga.jp' will expire on Tue 03 Jun 2025 04:02:12 PM GMT +0000.
[17:32:58] <MacFanBot>	 [02puppet] 07Universal-Omega closed pull request #4138: add cron to deploy Semantic MediaWiki (03master...03Reception123-patch-4) 13https://github.com/miraheze/puppet/pull/4138
[17:32:58] <MacFanBot>	 [02puppet] 07Universal-Omega commented on pull request #4138: We don't need to deploy this file anymore. 13https://github.com/miraheze/puppet/pull/4138#issuecomment-2701621414
[17:33:03] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03Reception123-patch-4 at 031dd9afe 13https://github.com/miraheze/puppet/commit/1dd9afe
[17:44:12] <MacFanBot>	 [02puppet] 07Universal-Omega left a file comment in pull request #4209 038d837fd: Maybe we could use `print('Error writing: {} Dropping this message: "{}"'.format(e, s))` instead? 13https://github.com/miraheze/puppet/pull/4209#discussion_r1981886145
[18:15:56] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4231: Revert "Revert "Temp allow cloud ssh without bastion"" (03master...03revert-4205-revert-4192-cloud-ssh) 13https://github.com/miraheze/puppet/pull/4231
[18:15:56] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/f52629da2a3b7a871da509ee2b295e0cc3661f0f
[18:15:58] <MacFanBot>	 02puppet/03master 07CosmicAlpha 03f52629d Revert "Revert "Temp allow cloud ssh without bastion"" (#4231)…
[18:16:00] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03revert-4205-revert-4192-cloud-ssh at 038dffa95 13https://github.com/miraheze/puppet/commit/8dffa95
[18:20:19] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4230: Revert "Revert "mariadb: default to read_only on startup"" (03master...03revert-4206-revert-4204-mariadb-read_only) 13https://github.com/miraheze/puppet/pull/4230
[18:20:19] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/162f6948fff01886a090a7fe138436bf71d7436a
[18:20:19] <MacFanBot>	 02puppet/03master 07CosmicAlpha 03162f694 Revert "Revert "mariadb: default to read_only on startup"" (#4230)…
[18:20:20] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03revert-4206-revert-4204-mariadb-read_only at 036f3fc26 13https://github.com/miraheze/puppet/commit/6f3fc26
[18:21:18] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/fb83b0c9aaf6ae1195b24cdd67af354455e90960
[18:21:18] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 03fb83b0c Bot: Update SSL cert for echoes-wiki.com
[18:30:12] <MirahezeRelay>	 <zppix, replying to MacFan4000> Thanks that worked macfan
[18:43:53] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 30.74, 19.24, 8.69
[18:45:53] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 13.65, 18.41, 9.79
[18:47:10] <icinga-miraheze>	 PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.069 second response time
[18:47:11] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.071 second response time
[18:47:13] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 8191 bytes in 0.125 second response time
[18:47:13] <icinga-miraheze>	 PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.080 second response time
[18:47:15] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.070 second response time
[18:47:16] <icinga-miraheze>	 PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.073 second response time
[18:47:19] <icinga-miraheze>	 PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error
[18:47:19] <icinga-miraheze>	 PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect
[18:47:19] <icinga-miraheze>	 n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 255 (0
[18:47:23] <icinga-miraheze>	 PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.071 second response time
[18:47:23] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.081 second response time
[18:47:24] <icinga-miraheze>	 PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect
[18:47:24] <icinga-miraheze>	 n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 255 (0
[18:47:25] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 77%
[18:47:27] <icinga-miraheze>	 PROBLEM - db172 MariaDB Connections on db172 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db172.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect
[18:47:27] <icinga-miraheze>	 n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db172.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 255 (0
[18:47:27] <icinga-miraheze>	 PROBLEM - db171 MariaDB on db171 is CRITICAL: Can't connect to server on 'db171.wikitide.net' (115)
[18:47:28] <icinga-miraheze>	 PROBLEM - db161 MariaDB on db161 is CRITICAL: Can't connect to server on 'db161.wikitide.net' (115)
[18:47:29] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.067 second response time
[18:47:32] <icinga-miraheze>	 PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect
[18:47:32] <icinga-miraheze>	 n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 255 (0
[18:47:33] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6895 bytes in 0.089 second response time
[18:47:37] <icinga-miraheze>	 PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.078 second response time
[18:47:40] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.074 second response time
[18:47:41] <icinga-miraheze>	 PROBLEM - db172 MariaDB on db172 is CRITICAL: Can't connect to server on 'db172.wikitide.net' (115)
[18:47:41] <icinga-miraheze>	 PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.071 second response time
[18:47:44] <icinga-miraheze>	 PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6895 bytes in 0.118 second response time
[18:47:48] <icinga-miraheze>	 PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6895 bytes in 0.098 second response time
[18:47:56] <icinga-miraheze>	 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 426 bytes in 0.042 second response time
[18:48:05] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.068 second response time
[18:48:06] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.077 second response time
[18:48:15] <icinga-miraheze>	 PROBLEM - db182 MariaDB on db182 is CRITICAL: Can't connect to server on 'db182.wikitide.net' (115)
[18:48:19] <icinga-miraheze>	 PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.074 second response time
[18:48:22] <icinga-miraheze>	 PROBLEM - db181 MariaDB on db181 is CRITICAL: Can't connect to server on 'db181.wikitide.net' (115)
[18:48:26] <icinga-miraheze>	 PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect
[18:48:26] <icinga-miraheze>	 n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 255 (0
[18:48:31] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.064 second response time
[18:48:32] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6895 bytes in 0.091 second response time
[18:48:35] <icinga-miraheze>	 PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.069 second response time
[18:48:43] <icinga-miraheze>	 PROBLEM - db151 MariaDB on db151 is CRITICAL: Can't connect to server on 'db151.wikitide.net' (115)
[18:48:43] <icinga-miraheze>	 PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4279 bytes in 0.020 second response time
[18:48:58] <icinga-miraheze>	 PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect
[18:48:58] <icinga-miraheze>	 n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66<Terminated with exit code 255 (0
[18:48:59] <icinga-miraheze>	 PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.071 second response time
[18:49:04] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4731 bytes in 0.078 second response time
[19:57:06] <icinga-miraheze>	 RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.311 second response time
[19:57:06] <icinga-miraheze>	 RECOVERY - db172 MariaDB Connections on db172 is OK: OK connection usage: 0.1%Current connections: 1
[19:57:06] <icinga-miraheze>	 RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 20.7%Current connections: 207
[19:57:06] <icinga-miraheze>	 RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time
[19:57:06] <icinga-miraheze>	 RECOVERY - db151 MariaDB on db151 is OK: Uptime: 63 Threads: 19 Questions: 30236 Slow queries: 0 Opens: 2604 Open tables: 2598 Queries per second avg: 479.936
[19:57:06] <icinga-miraheze>	 RECOVERY - db182 MariaDB Connections on db182 is OK: OK connection usage: 0.6%Current connections: 6
[19:57:09] <icinga-miraheze>	 RECOVERY - db182 MariaDB on db182 is OK: Uptime: 12 Threads: 5 Questions: 15712 Slow queries: 0 Opens: 55 Open tables: 48 Queries per second avg: 1309.333
[19:57:10] <icinga-miraheze>	 RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 7.8%Current connections: 78
[19:57:11] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time
[19:57:12] <icinga-miraheze>	 RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.112 second response time
[19:57:13] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.488 second response time
[19:57:14] <icinga-miraheze>	 RECOVERY - db171 MariaDB on db171 is OK: Uptime: 191 Threads: 124 Questions: 153096 Slow queries: 0 Opens: 3025 Open tables: 3019 Queries per second avg: 801.549
[19:57:14] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.534 second response time
[19:57:14] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time
[19:57:19] <icinga-miraheze>	 RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time
[19:57:19] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [5 system event log (SEL) entries present]
[19:57:21] <icinga-miraheze>	 RECOVERY - echoes-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'echoes-wiki.com' will expire on Tue 03 Jun 2025 05:22:42 PM GMT +0000.
[19:57:25] <icinga-miraheze>	 RECOVERY - db161 MariaDB on db161 is OK: Uptime: 335 Threads: 37 Questions: 201320 Slow queries: 0 Opens: 4842 Open tables: 4836 Queries per second avg: 600.955
[19:57:27] <icinga-miraheze>	 RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.177 second response time
[19:57:32] <icinga-miraheze>	 PROBLEM - cloud20 IPMI Sensors on cloud20 is CRITICAL: IPMI Status: Critical [88 system event log (SEL) entries present]
[19:57:33] <icinga-miraheze>	 RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time
[19:57:34] <icinga-miraheze>	 PROBLEM - cp38 Disk Space on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host
[19:57:35] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.184 second response time
[19:57:35] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time
[19:57:35] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.067 second response time
[19:57:36] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.499 second response time
[19:57:36] <icinga-miraheze>	 RECOVERY - db181 MariaDB on db181 is OK: Uptime: 57 Threads: 7 Questions: 72388 Slow queries: 0 Opens: 3592 Open tables: 3580 Queries per second avg: 1269.964
[19:57:37] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time
[19:57:37] <icinga-miraheze>	 RECOVERY - db151 MariaDB Connections on db151 is OK: OK connection usage: 1.3%Current connections: 13
[19:57:40] <icinga-miraheze>	 RECOVERY - db172 MariaDB on db172 is OK: Uptime: 204 Threads: 1 Questions: 219 Slow queries: 0 Opens: 42 Open tables: 36 Queries per second avg: 1.073
[19:57:41] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time
[19:57:42] <icinga-miraheze>	 RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.964 second response time
[19:57:42] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time
[19:57:46] <icinga-miraheze>	 RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17669 bytes in 0.053 second response time
[19:57:47] <icinga-miraheze>	 RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.9%Current connections: 9
[19:57:48] <icinga-miraheze>	 RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time
[19:58:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:58:49] <icinga-miraheze>	 RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19582 bytes in 0.064 second response time
[19:58:51] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for mw173 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host
[19:58:53] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:58:54] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for matomo151 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host
[19:58:54] <icinga-miraheze>	 PROBLEM - cp38 ferm_active on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host
[19:58:55] <icinga-miraheze>	 PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:58:55] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet last ran 1 hour ago
[19:58:56] <icinga-miraheze>	 PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:58:56] <icinga-miraheze>	 PROBLEM - Host mw193 is DOWN: CRITICAL - Host Unreachable (10.0.19.164)
[19:58:56] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:58:57] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:58:57] <icinga-miraheze>	 PROBLEM - cp38 Current Load on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host
[19:58:58] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:58:59] <icinga-miraheze>	 PROBLEM - bast181 NTP time on bast181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:58:59] <icinga-miraheze>	 PROBLEM - changeprop201 Puppet on changeprop201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:00] <icinga-miraheze>	 PROBLEM - swiftobject191 SSH on swiftobject191 is CRITICAL: connect to address 10.0.19.120 and port 22: No route to host
[19:59:00] <icinga-miraheze>	 PROBLEM - swiftobject191 Swift Object Service on swiftobject191 is CRITICAL: connect to address 10.0.19.120 and port 6000: No route to host
[19:59:01] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for mwtask151 on cp38 is CRITICAL: connect to address 10.0.19.146 port 5666: No route to hostconnect to host 10.0.19.146 port 5666: No route to host
[19:59:01] <icinga-miraheze>	 PROBLEM - cloud16 IPMI Sensors on cloud16 is CRITICAL: IPMI Status: Critical [2 system event log (SEL) entries present]
[19:59:01] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:14] <icinga-miraheze>	 PROBLEM - Host mw191 is DOWN: CRITICAL - Host Unreachable (10.0.19.160)
[19:59:15] <icinga-miraheze>	 PROBLEM - mw191 SSH on mw191 is CRITICAL: connect to address 10.0.19.160 and port 22: No route to host
[19:59:15] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for mw163 on cp38 is CRITICAL: connect to address localhost and port 8123: Connection refused
[19:59:16] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:16] <icinga-miraheze>	 PROBLEM - swiftobject201 ferm_active on swiftobject201 is CRITICAL: connect to address 10.0.20.145 port 5666: No route to hostconnect to host 10.0.20.145 port 5666: No route to host
[19:59:17] <icinga-miraheze>	 RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time
[19:59:17] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for swiftproxy161 on cp38 is CRITICAL: connect to address localhost and port 8206: Connection refused
[19:59:18] <icinga-miraheze>	 PROBLEM - mw192 php-fpm on mw192 is CRITICAL: connect to address 10.0.19.161 port 5666: No route to hostconnect to host 10.0.19.161 port 5666: No route to host
[19:59:18] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet last ran 1 hour ago
[19:59:19] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for mw183 on cp38 is CRITICAL: connect to address localhost and port 8127: Connection refused
[19:59:19] <icinga-miraheze>	 PROBLEM - cloud18 IPMI Sensors on cloud18 is CRITICAL: IPMI Status: Critical [2 system event log (SEL) entries present]
[19:59:20] <icinga-miraheze>	 PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:20] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for mw151 on cp38 is CRITICAL: connect to address localhost and port 8113: Connection refused
[19:59:21] <icinga-miraheze>	 PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:21] <icinga-miraheze>	 PROBLEM - cp38 Nginx Backend for mw161 on cp38 is CRITICAL: connect to address localhost and port 8115: Connection refused PROBLEM - bast161 NTP time on bast161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:59:22] <icinga-miraheze>	 PROBLEM - graylog161 graylog tls port 12210 ssl cert check on graylog161 is CRITICAL: connect to address localhost and port 12210: Connection refused
[19:59:22] <icinga-miraheze>	 PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:23] <icinga-miraheze>	 PROBLEM - bast161 PowerDNS Recursor on bast161 is CRITICAL: Domain 'wikitide.net' was not found by the server
[19:59:34] <icinga-miraheze>	 PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:35] <icinga-miraheze>	 PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:35] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:36] <icinga-miraheze>	 PROBLEM - swiftobject201 conntrack_table_size on swiftobject201 is CRITICAL: connect to address 10.0.20.145 port 5666: No route to hostconnect to host 10.0.20.145 port 5666: No route to host
[19:59:36] <icinga-miraheze>	 PROBLEM - swiftobject201 PowerDNS Recursor on swiftobject201 is CRITICAL: connect to address 10.0.20.145 port 5666: No route to hostconnect to host 10.0.20.145 port 5666: No route to host
[19:59:37] <icinga-miraheze>	 PROBLEM - swiftobject201 Current Load on swiftobject201 is CRITICAL: connect to address 10.0.20.145 port 5666: No route to hostconnect to host 10.0.20.145 port 5666: No route to host
[19:59:37] <icinga-miraheze>	 PROBLEM - swiftobject201 Disk Space on swiftobject201 is CRITICAL: connect to address 10.0.20.145 port 5666: No route to hostconnect to host 10.0.20.145 port 5666: No route to host
[19:59:38] <icinga-miraheze>	 PROBLEM - eventgate181 Puppet on eventgate181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:38] <icinga-miraheze>	 PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:39] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 1 backends are down. mw191
[19:59:39] <icinga-miraheze>	 PROBLEM - swiftobject201 Swift Object Service on swiftobject201 is CRITICAL: connect to address 10.0.20.145 and port 6000: No route to host
[19:59:40] <icinga-miraheze>	 PROBLEM - cloud17 IPMI Sensors on cloud17 is CRITICAL: IPMI Status: Critical [2 system event log (SEL) entries present]
[19:59:40] <icinga-miraheze>	 PROBLEM - mw193.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:59:41] <icinga-miraheze>	 PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:41] <icinga-miraheze>	 PROBLEM - cp38 NTP time on cp38 is CRITICAL: NTP CRITICAL: Offset 0.6193584204 secs
[19:59:42] <icinga-miraheze>	 PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:42] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:43] <icinga-miraheze>	 PROBLEM - cloud20 Puppet on cloud20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[19:59:54] <icinga-miraheze>	 PROBLEM - cp38 Puppet on cp38 is WARNING: WARNING: Puppet last ran 1 hour ago
[20:00:45] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw173 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8125
[20:00:49] <icinga-miraheze>	 RECOVERY - cp38 ferm_active on cp38 is OK: OK ferm input default policy is set
[20:00:50] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for matomo151 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8203
[20:00:53] <icinga-miraheze>	 RECOVERY - cp38 Current Load on cp38 is OK: LOAD OK - total load average: 0.90, 0.30, 0.11
[20:00:53] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 0 seconds ago with 2 failures. Failed resources (up to 3 shown)
[20:00:57] <icinga-miraheze>	 RECOVERY - Host mw193 is UP: PING OK - Packet loss = 0%, RTA = 1.05 ms
[20:00:59] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mwtask151 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8162
[20:01:00] <icinga-miraheze>	 RECOVERY - cp38 HTTP 4xx/5xx ERROR Rate on cp38 is OK: OK - NGINX Error Rate is 5%
[20:01:01] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw171 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8117
[20:01:03] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for puppet181 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8204
[20:01:05] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw203 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8134
[20:01:05] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw192 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8130
[20:01:05] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw152 on cp38 is OK: TCP OK - 0.001 second response time on localhost port 8114
[20:01:07] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw181 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8119
[20:01:08] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw163 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8123
[20:01:10] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw151 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8113
[20:01:10] <icinga-miraheze>	 RECOVERY - Host swiftobject191 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[20:01:10] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet last ran 1 hour ago
[20:01:11] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for swiftproxy161 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8206
[20:01:11] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw183 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8127
[20:01:11] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 17 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_mathoid],Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:01:13] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy
[20:01:13] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mwtask181 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8160
[20:01:14] <icinga-miraheze>	 RECOVERY - graylog161 graylog tls port 12210 ssl cert check on graylog161 is OK: OK - Certificate 'wikitide.net' will expire on Tue May 27 01:33:01 2025 +0000.TCP OK - 0.119 second response time on localhost port 12210
[20:01:15] <icinga-miraheze>	 RECOVERY - Host mw191 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:01:15] <icinga-miraheze>	 RECOVERY - cp38 Nginx Backend for mw161 on cp38 is OK: TCP OK - 0.000 second response time on localhost port 8115
[20:01:16] <icinga-miraheze>	 RECOVERY - graylog161 HTTPS on graylog161 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.106 second response time
[20:01:18] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 7 seconds ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet],Exec[git_pull_ssl],Exec[git_pull_mediawiki-repos],Exec[git_pull_pywikibot-config]
[20:01:18] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy
[20:01:18] <icinga-miraheze>	 PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 25 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:01:19] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:27] <icinga-miraheze>	 RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[20:01:31] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[20:01:32] <icinga-miraheze>	 RECOVERY - Host mw192 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms
[20:01:36] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:40] <icinga-miraheze>	 RECOVERY - cp38 NTP time on cp38 is OK: NTP OK: Offset -0.0001036822796 secs
[20:01:44] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[20:01:45] <icinga-miraheze>	 PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 51 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:01:45] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:45] <icinga-miraheze>	 PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 46 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:01:46] <icinga-miraheze>	 RECOVERY - Host swiftobject201 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms
[20:01:46] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[20:01:46] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:01:49] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 46 seconds ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:01:50] <icinga-miraheze>	 RECOVERY - cp38 Puppet on cp38 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[20:02:55] <icinga-miraheze>	 RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.153 second response time RECOVERY - swiftobject201 SSH on swiftobject201 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0)
[20:03:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:03:00] <icinga-miraheze>	 RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time
[20:03:00] <icinga-miraheze>	 RECOVERY - swiftobject191 SSH on swiftobject191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0)
[20:03:00] <icinga-miraheze>	 RECOVERY - swiftobject191 Swift Object Service on swiftobject191 is OK: TCP OK - 0.000 second response time on 10.0.19.120 port 6000
[20:03:00] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 2.22, 1.25, 0.51
[20:03:01] <icinga-miraheze>	 RECOVERY - ping on mw192 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[20:03:01] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:03:05] <icinga-miraheze>	 RECOVERY - mw192 ferm_active on mw192 is OK: OK ferm input default policy is set
[20:03:05] <icinga-miraheze>	 RECOVERY - swiftobject201 ferm_active on swiftobject201 is OK: OK ferm input default policy is set
[20:03:05] <icinga-miraheze>	 RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.052 second response time
[20:03:05] <icinga-miraheze>	 RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.154 second response time
[20:03:06] <icinga-miraheze>	 RECOVERY - mw191 conntrack_table_size on mw191 is OK: OK: nf_conntrack is 1 % full
[20:03:06] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:03:10] <icinga-miraheze>	 RECOVERY - swiftobject201 NTP time on swiftobject201 is OK: NTP OK: Offset 1.364946365e-05 secs
[20:03:10] <icinga-miraheze>	 RECOVERY - swiftobject201 Puppet on swiftobject201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:03:10] <icinga-miraheze>	 RECOVERY - mw192 php-fpm on mw192 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2'
[20:03:10] <icinga-miraheze>	 RECOVERY - mw191 SSH on mw191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u5 (protocol 2.0)
[20:03:13] <icinga-miraheze>	 RECOVERY - cloud19 Puppet on cloud19 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[20:03:20] <icinga-miraheze>	 RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:03:20] <icinga-miraheze>	 RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[20:03:20] <icinga-miraheze>	 RECOVERY - swiftobject201 Disk Space on swiftobject201 is OK: DISK OK - free space: / 460000MiB (34% inode=89%);
[20:03:25] <icinga-miraheze>	 RECOVERY - swiftobject201 PowerDNS Recursor on swiftobject201 is OK: DNS OK: 0.035 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:03:25] <icinga-miraheze>	 RECOVERY - ping on swiftobject201 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms
[20:03:25] <icinga-miraheze>	 RECOVERY - swiftobject201 Current Load on swiftobject201 is OK: LOAD OK - total load average: 0.53, 0.34, 0.14
[20:03:30] <icinga-miraheze>	 RECOVERY - swiftobject201 conntrack_table_size on swiftobject201 is OK: OK: nf_conntrack is 0 % full
[20:03:30] <icinga-miraheze>	 RECOVERY - swiftobject201 Swift Object Service on swiftobject201 is OK: TCP OK - 0.000 second response time on 10.0.20.145 port 6000
[20:03:30] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_3d2png]
[20:03:34] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[20:03:48] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:04:49] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[20:04:49] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:07:04] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[20:09:09] <icinga-miraheze>	 RECOVERY - bast161 PowerDNS Recursor on bast161 is OK: DNS OK: 0.328 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[20:09:14] <icinga-miraheze>	 RECOVERY - bast161 NTP time on bast161 is OK: NTP OK: Offset 2.631545067e-05 secs
[20:09:44] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[20:10:59] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:11:16] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:11:17] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:11:31] <icinga-miraheze>	 RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[20:11:36] <icinga-miraheze>	 RECOVERY - phorge171 phd on phorge171 is OK: PROCS OK: 1 process with args 'phd'
[20:11:37] <icinga-miraheze>	 RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[20:12:56] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[20:13:06] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:13:19] <icinga-miraheze>	 RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:13:45] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:15:15] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:15:21] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:15:32] <icinga-miraheze>	 PROBLEM - phorge171 phd on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd'
[20:16:33] <icinga-miraheze>	 RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:16:37] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[20:16:38] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[20:17:08] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:17:14] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:17:43] <icinga-miraheze>	 RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[20:18:15] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:18:39] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[20:19:18] <icinga-miraheze>	 RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:21:05] <icinga-miraheze>	 RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:21:06] <icinga-miraheze>	 RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[20:23:04] <icinga-miraheze>	 RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:23:12] <icinga-miraheze>	 RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:24:02] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:24:38] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[20:25:28] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:25:53] <icinga-miraheze>	 RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[20:26:16] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[20:26:22] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:26:25] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:26:50] <icinga-miraheze>	 RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:27:00] <icinga-miraheze>	 RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[20:27:17] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[20:27:36] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:27:41] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[20:28:05] <icinga-miraheze>	 RECOVERY - mw193.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000.
[20:28:31] <icinga-miraheze>	 PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 268 bytes in 0.030 second response time
[20:28:33] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[20:28:42] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:28:49] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[20:28:52] <icinga-miraheze>	 RECOVERY - mw191.wikitide.net SSL Check on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Tue 27 May 2025 01:33:01 AM GMT +0000.
[20:29:11] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:29:42] <icinga-miraheze>	 PROBLEM - db161 NTP time on db161 is CRITICAL: NTP CRITICAL: Offset 0.884301126 secs
[20:29:48] <icinga-miraheze>	 RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[20:30:05] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:30:16] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:30:24] <icinga-miraheze>	 PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[20:30:24] <icinga-miraheze>	 PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:30:24] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:30:24] <icinga-miraheze>	 PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[20:30:26] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[20:30:29] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[20:30:33] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:30:38] <icinga-miraheze>	 PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 48%
[20:30:40] <icinga-miraheze>	 PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:30:40] <icinga-miraheze>	 PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received
[20:30:42] <icinga-miraheze>	 PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki
[20:30:46] <icinga-miraheze>	 PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:30:47] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:30:48] <icinga-miraheze>	 PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[20:30:53] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 89%
[20:30:55] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:30:58] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:31:02] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:31:03] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[20:31:03] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:31:04] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[20:31:08] <icinga-miraheze>	 PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[20:31:14] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 388.85, 213.91, 86.82
[20:31:25] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 10 backends are down. mw151 mw161 mw162 mw172 mw181 mw153 mw163 mw183 mw201 mw203
[20:31:41] <icinga-miraheze>	 PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw181 mw163 mw191
[20:31:42] <icinga-miraheze>	 RECOVERY - db161 NTP time on db161 is OK: NTP OK: Offset 5.948543549e-05 secs
[20:32:04] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time
[20:32:17] <icinga-miraheze>	 RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.057 second response time
[20:32:18] <icinga-miraheze>	 RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.151 second response time
[20:32:18] <icinga-miraheze>	 RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.050 second response time
[20:32:20] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.149 second response time
[20:32:21] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.050 second response time
[20:32:24] <icinga-miraheze>	 PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100%
[20:32:26] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.054 second response time
[20:32:33] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4091 bytes in 0.044 second response time
[20:32:36] <icinga-miraheze>	 RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 4%
[20:32:37] <icinga-miraheze>	 RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time
[20:32:37] <icinga-miraheze>	 RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.049 second response time
[20:32:42] <icinga-miraheze>	 RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy
[20:32:43] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time
[20:32:46] <icinga-miraheze>	 RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4090 bytes in 0.057 second response time
[20:32:46] <icinga-miraheze>	 RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.054 second response time
[20:32:49] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.147 second response time
[20:32:53] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 5%
[20:32:56] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.143 second response time
[20:33:03] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.050 second response time
[20:33:06] <icinga-miraheze>	 RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3968 bytes in 0.056 second response time
[20:33:24] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[20:33:35] <icinga-miraheze>	 RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy
[20:33:40] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:34:22] <icinga-miraheze>	 RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 30.97 ms
[20:35:28] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:35:38] <MacFanBot>	 [02puppet] 07Universal-Omega created 03buffer (+1 new commit) 13https://github.com/miraheze/puppet/commit/c4cf583dc2b7
[20:35:38] <MacFanBot>	 02puppet/03buffer 07CosmicAlpha 03c4cf583 db: increase innodb buffer pool to 78G
[20:35:42] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4233: db: increase innodb buffer pool to 78G (03master...03buffer) 13https://github.com/miraheze/puppet/pull/4233
[20:35:49] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4233: --- […] 13https://github.com/miraheze/puppet/pull/4233#issuecomment-2702006857
[20:36:04] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03buffer 13https://github.com/miraheze/puppet/commit/986365a50fd8d22a413b858406d0b4038bf59b15
[20:36:04] <MacFanBot>	 02puppet/03buffer 07CosmicAlpha 03986365a Update
[20:36:21] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03buffer 13https://github.com/miraheze/puppet/commit/a0505d5f5e9a601600392d2b03ae3eb5d092c32e
[20:36:21] <MacFanBot>	 02puppet/03buffer 07CosmicAlpha 03a0505d5 Update
[20:36:41] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03buffer 13https://github.com/miraheze/puppet/commit/1c62db6a1859d988e5a5f3a5901d8c3e4af96d1c
[20:36:41] <MacFanBot>	 02puppet/03buffer 07CosmicAlpha 031c62db6 Update
[20:36:58] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[20:37:04] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[20:37:33] <MacFanBot>	 [02puppet] 07paladox merged 07Universal-Omega's pull request #4233: db: increase innodb buffer pool to 78G (03master...03buffer) 13https://github.com/miraheze/puppet/pull/4233
[20:37:33] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/7ea7c84c93f063554d25c87df064ae5744bad752
[20:37:33] <MacFanBot>	 02puppet/03master 07CosmicAlpha 037ea7c84 db: increase innodb buffer pool to 78G (#4233)
[20:37:34] <MacFanBot>	 [02puppet] 07paladox 04deleted 03buffer at 031c62db6 13https://github.com/miraheze/puppet/commit/1c62db6
[20:37:55] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[20:39:34] <MacFanBot>	 [02puppet] 07Universal-Omega created 03revert-4231-revert-4205-revert-4192-cloud-ssh (+1 new commit) 13https://github.com/miraheze/puppet/commit/9f3ed9a3151d
[20:39:34] <MacFanBot>	 02puppet/03revert-4231-revert-4205-revert-4192-cloud-ssh 07CosmicAlpha 039f3ed9a Revert "Revert "Revert "Temp allow cloud ssh without bastion"" (#4231)"…
[20:39:38] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4234: Revert "Revert "Revert "Temp allow cloud ssh without bastion""" (03master...03revert-4231-revert-4205-revert-4192-cloud-ssh) 13https://github.com/miraheze/puppet/pull/4234
[20:39:45] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4234: --- […] 13https://github.com/miraheze/puppet/pull/4234#issuecomment-2702014554
[20:39:48] <MacFanBot>	 [02puppet] 07Universal-Omega created 03revert-4230-revert-4206-revert-4204-mariadb-read_only (+1 new commit) 13https://github.com/miraheze/puppet/commit/65918da687ee
[20:39:48] <MacFanBot>	 02puppet/03revert-4230-revert-4206-revert-4204-mariadb-read_only 07CosmicAlpha 0365918da Revert "Revert "Revert "mariadb: default to read_only on startup"" (#4230)"…
[20:39:51] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4235: Revert "Revert "Revert "mariadb: default to read_only on startup""" (03master...03revert-4230-revert-4206-revert-4204-mariadb-read_only) 13https://github.com/miraheze/puppet/pull/4235
[20:39:58] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4235: --- […] 13https://github.com/miraheze/puppet/pull/4235#issuecomment-2702014968
[20:45:21] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/977899d53d43
[20:45:22] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 03977899d mariadb: increase table_open_cache and table_definition_cache to 200k
[20:45:25] <MacFanBot>	 [02puppet] 07paladox opened pull request #4236: mariadb: increase table_open_cache and table_definition_cache to 200k (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4236
[20:45:31] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4236: --- […] 13https://github.com/miraheze/puppet/pull/4236#issuecomment-2702031881
[20:45:34] <MacFanBot>	 [02puppet] 07paladox merged pull request #4236: mariadb: increase table_open_cache and table_definition_cache to 200k (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4236
[20:45:35] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/90d31d75715865ab1f5d326a84922b461244bdec
[20:45:36] <MacFanBot>	 02puppet/03master 07paladox 0390d31d7 mariadb: increase table_open_cache and table_definition_cache to 200k (#4236)
[20:45:38] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03977899d 13https://github.com/miraheze/puppet/commit/977899d
[20:45:50] <MacFanBot>	 [02puppet] 07paladox merged 07Universal-Omega's pull request #4235: Revert "Revert "Revert "mariadb: default to read_only on startup""" (03master...03revert-4230-revert-4206-revert-4204-mariadb-read_only) 13https://github.com/miraheze/puppet/pull/4235
[20:45:50] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/577c0bd88ac35f31480ca7b9546f046d410bf16c
[20:45:50] <MacFanBot>	 02puppet/03master 07CosmicAlpha 03577c0bd Revert "Revert "Revert "mariadb: default to read_only on startup""" (#4235)…
[20:45:51] <MacFanBot>	 [02puppet] 07paladox 04deleted 03revert-4230-revert-4206-revert-4204-mariadb-read_only at 0365918da 13https://github.com/miraheze/puppet/commit/65918da
[20:47:59] <icinga-miraheze>	 RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 2.294 second response time
[20:49:15] <icinga-miraheze>	 RECOVERY - phorge171 phd on phorge171 is OK: PROCS OK: 1 process with args 'phd'
[20:50:46] <icinga-miraheze>	 PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:50:51] <icinga-miraheze>	 PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:50:53] <icinga-miraheze>	 PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 87%
[20:50:55] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all
[20:50:57] <icinga-miraheze>	 PROBLEM - cp38 HTTPS on cp38 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[20:51:00] <icinga-miraheze>	 PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.077 second response time
[20:51:20] <icinga-miraheze>	 PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw173 mw191 mw203
[20:51:27] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 32s
[20:51:29] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4234: Revert "Revert "Revert "Temp allow cloud ssh without bastion""" (03master...03revert-4231-revert-4205-revert-4192-cloud-ssh) 13https://github.com/miraheze/puppet/pull/4234
[20:51:29] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/3b8a23f85418dc02435612c59b5c85d70e941576
[20:51:30] <MacFanBot>	 02puppet/03master 07CosmicAlpha 033b8a23f Revert "Revert "Revert "Temp allow cloud ssh without bastion""" (#4234)…
[20:51:30] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03revert-4231-revert-4205-revert-4192-cloud-ssh at 039f3ed9a 13https://github.com/miraheze/puppet/commit/9f3ed9a
[20:52:04] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.072 second response time
[20:52:05] <icinga-miraheze>	 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.069 second response time
[20:52:08] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.070 second response time
[20:52:12] <icinga-miraheze>	 PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 47%
[20:52:19] <icinga-miraheze>	 PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.077 second response time
[20:52:43] <icinga-miraheze>	 RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4091 bytes in 0.060 second response time
[20:52:48] <icinga-miraheze>	 RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4092 bytes in 0.054 second response time
[20:52:54] <icinga-miraheze>	 RECOVERY - cp38 HTTPS on cp38 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4091 bytes in 0.062 second response time
[20:53:07] <icinga-miraheze>	 PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.069 second response time
[20:53:13] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.078 second response time
[20:53:13] <icinga-miraheze>	 PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.078 second response time
[20:53:16] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.074 second response time
[20:53:19] <icinga-miraheze>	 RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy
[20:53:33] <icinga-miraheze>	 PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.069 second response time
[20:53:38] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all
[20:53:42] <icinga-miraheze>	 PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.078 second response time
[20:53:57] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 19s
[20:53:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:54:01] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.167 second response time
[20:54:02] <icinga-miraheze>	 RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.158 second response time
[20:54:03] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.163 second response time
[20:54:10] <icinga-miraheze>	 RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 8%
[20:54:18] <icinga-miraheze>	 RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.151 second response time
[20:54:53] <icinga-miraheze>	 RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 18%
[20:54:58] <icinga-miraheze>	 RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.144 second response time
[20:55:07] <icinga-miraheze>	 RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.137 second response time
[20:55:11] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.150 second response time
[20:55:13] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.154 second response time
[20:55:13] <icinga-miraheze>	 RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.157 second response time
[20:55:30] <icinga-miraheze>	 RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time
[20:55:36] <icinga-miraheze>	 RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.196 second response time
[20:57:28] <MirahezeRelay>	 <paladox> !log increase cp37 to 30gib
[20:57:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:08:27] <MacFanBot>	 [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/5ae8f5c46e14
[21:08:27] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 035ae8f5c Add mem201 to dns
[21:08:31] <MacFanBot>	 [02dns] 07paladox opened pull request #651: Add mem201 to dns (03master...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/651
[21:08:36] <MacFanBot>	 [02dns] 07paladox merged pull request #651: Add mem201 to dns (03master...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/651
[21:08:38] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/dns/commit/54f3a1dfed96a6ca0b9a8852e042655fffa8f0a5
[21:08:40] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-1 at 035ae8f5c 13https://github.com/miraheze/dns/commit/5ae8f5c
[21:08:41] <MacFanBot>	 02dns/03master 07paladox 0354f3a1d Add mem201 to dns (#651)
[21:09:03] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[21:09:40] <042AA82YN>	 miraheze/dns - paladox the build passed.
[21:09:40] <073AB1QT6>	 miraheze/dns - paladox the build passed.
[21:10:08] <MirahezeRelay>	 <cosmicalpha> !log increased db* RAM to 140GB
[21:10:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:29] <MirahezeLSBot>	 !log [universalomega@cloud15] Clear SEL
[21:23:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:24:31] <icinga-miraheze>	 RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK
[21:24:32] <icinga-miraheze>	 RECOVERY - cloud16 IPMI Sensors on cloud16 is OK: IPMI Status: OK
[21:24:32] <icinga-miraheze>	 PROBLEM - bast181 NTP time on bast181 is CRITICAL: connect to address 10.0.18.101 port 5666: No route to hostconnect to host 10.0.18.101 port 5666: No route to host
[21:24:41] <MirahezeLSBot>	 !log [universalomega@cloud16.wikitide.net] Clear SEL
[21:24:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:24:45] <icinga-miraheze>	 RECOVERY - bast181 PowerDNS Recursor on bast181 is OK: DNS OK: 0.244 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205
[21:25:13] <MirahezeLSBot>	 !log [universalomega@cloud17.wikitide.net] Clear SEL
[21:25:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:25:44] <MirahezeLSBot>	 !log [universalomega@cloud18.wikitide.net] Clear SEL
[21:25:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:26:21] <icinga-miraheze>	 RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[21:26:29] <MirahezeLSBot>	 !log [universalomega@cloud19.wikitide.net] Clear SEL
[21:26:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:26:32] <icinga-miraheze>	 RECOVERY - bast181 NTP time on bast181 is OK: NTP OK: Offset 0.0001050829887 secs
[21:26:33] <icinga-miraheze>	 RECOVERY - cloud19 IPMI Sensors on cloud19 is OK: IPMI Status: OK
[21:27:02] <icinga-miraheze>	 RECOVERY - cloud17 IPMI Sensors on cloud17 is OK: IPMI Status: OK
[21:27:02] <icinga-miraheze>	 RECOVERY - cloud18 IPMI Sensors on cloud18 is OK: IPMI Status: OK
[21:27:19] <MirahezeLSBot>	 !log [universalomega@cloud20.wikitide.net] Clear SEL
[21:27:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:27:38] <icinga-miraheze>	 RECOVERY - cloud20 IPMI Sensors on cloud20 is OK: IPMI Status: OK
[21:28:00] <icinga-miraheze>	 PROBLEM - fairytail.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fairytail.wiki' expires in 14 day(s) (Thu 20 Mar 2025 09:03:07 PM GMT +0000).
[21:28:11] <MacFanBot>	 [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/4e22b8ca69508ce9a2d74ba67af3034e888ee13a
[21:28:11] <MacFanBot>	 02ssl/03master 07WikiTideSSLBot 034e22b8c Bot: Update SSL cert for fairytail.wiki
[21:29:32] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4a299a06cd70
[21:29:32] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 034a299a0 Install mem201
[21:29:36] <MacFanBot>	 [02puppet] 07paladox opened pull request #4237: Install mem201 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4237
[21:29:42] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4237: --- […] 13https://github.com/miraheze/puppet/pull/4237#issuecomment-2702121584
[21:30:04] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/0372db76f4f2d89c1cb13992c884e555ac877763
[21:30:04] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 030372db7 Update site.pp
[21:30:14] <MacFanBot>	 [02puppet] 07paladox merged pull request #4237: Install mem201 (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4237
[21:30:15] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/29d171aaf89c9d1f0994932a102dff71455ee0b6
[21:30:15] <MacFanBot>	 02puppet/03master 07paladox 0329d171a Install mem201 (#4237)
[21:30:15] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 030372db7 13https://github.com/miraheze/puppet/commit/0372db7
[21:31:13] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.69, 0.66, 10.65
[21:32:18] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/39dc7945f882
[21:32:18] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 0339dc794 Add mem201 to mcrouter pool
[21:32:22] <MacFanBot>	 [02puppet] 07paladox opened pull request #4238: Add mem201 to mcrouter pool (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4238
[21:32:29] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4238: --- […] 13https://github.com/miraheze/puppet/pull/4238#issuecomment-2702126804
[21:33:13] <icinga-miraheze>	 RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.53, 0.59, 9.41
[21:33:58] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-6 13https://github.com/miraheze/puppet/commit/bd194a8ca22331e556a9d0d9b13d67c343f91069
[21:33:59] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 03bd194a8 Update common.yaml
[21:34:10] <MacFanBot>	 [02puppet] 07paladox merged pull request #4238: Add mem201 to mcrouter pool (03master...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4238
[21:34:10] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/8cce08297e49a2ebbd8145c3917d57de7ebdd5ff
[21:34:11] <MacFanBot>	 02puppet/03master 07paladox 038cce082 Add mem201 to mcrouter pool (#4238)
[21:34:13] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03bd194a8 13https://github.com/miraheze/puppet/commit/bd194a8
[21:48:58] <MirahezeRelay>	 <paladox> !log increase os151/161 storage size by 250gb each
[21:48:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:57:37] <icinga-miraheze>	 RECOVERY - fairytail.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fairytail.wiki' will expire on Tue 03 Jun 2025 08:29:35 PM GMT +0000.
[22:49:44] <BlankEclair>	 > [06/03/2025 08:27] <MirahezeLSBot> !log [universalomega@cloud20.wikitide.net] Clear SEL
[22:49:48] <BlankEclair>	 oh hey, fqdn as hostname
[22:55:22] <MacFanBot>	 [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/5520d85d503633904ca235b5c675e3ee37aeee2e
[22:55:22] <MacFanBot>	 02mw-config/03master 07paladox 035520d85 Add mem201 to wgMirahezeMagicMemcachedServers
[22:55:43] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[22:55:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:56:03] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s
[22:56:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:56:24] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[22:58:50] <MacFanBot>	 [02mw-config] 07paladox created 03paladox-patch-4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/6df3cfe94dda
[22:58:50] <MacFanBot>	 02mw-config/03paladox-patch-4 07paladox 036df3cfe Fix $wgMirahezeMagicMemcachedServers for beta
[22:58:55] <MacFanBot>	 [02mw-config] 07paladox opened pull request #5848: Fix $wgMirahezeMagicMemcachedServers for beta (03master...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/5848
[22:59:31] <MacFanBot>	 [02mw-config] 07github-actions[bot] pushed 1 new commit to 03paladox-patch-4 13https://github.com/miraheze/mw-config/commit/162ea2a4ab3c9b567d74f50f24ffc2faa7117533
[22:59:31] <MacFanBot>	 02mw-config/03paladox-patch-4 07github-actions 03162ea2a CI: lint code to MediaWiki standards…
[22:59:39] <MacFanBot>	 [02mw-config] 07paladox merged pull request #5848: Fix $wgMirahezeMagicMemcachedServers for beta (03master...03paladox-patch-4) 13https://github.com/miraheze/mw-config/pull/5848
[22:59:40] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[22:59:41] <MacFanBot>	 [02mw-config] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/mw-config/commit/afddc0aeee34a7159122878a85f6d2acdc360647
[22:59:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:59:42] <MacFanBot>	 02mw-config/03master 07paladox 03afddc0a Fix $wgMirahezeMagicMemcachedServers for beta (#5848)…
[22:59:44] <MacFanBot>	 [02mw-config] 07paladox 04deleted 03paladox-patch-4 at 03162ea2a 13https://github.com/miraheze/mw-config/commit/162ea2a
[22:59:56] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[22:59:57] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s
[22:59:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:00:00] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s
[23:00:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:00:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:00:03] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[23:00:31] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[23:15:01] <icinga-miraheze>	 PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100%
[23:17:27] <icinga-miraheze>	 RECOVERY - Host mattermost1 is UP: PING WARNING - Packet loss = 93%, RTA = 30.95 ms