[00:01:22] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/5cb45c5decf3a96a3cba0b9e059ae2da5de49443 [00:01:22] 02statichelp/03main 07WikiTideBot 035cb45c5 Bot: Auto-update Tech namespace pages 2025-09-26 00:01:20 [00:08:53] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 46341MiB (10% inode=99%); [00:10:57] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 47771MiB (10% inode=99%); [00:11:44] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 46061MiB (10% inode=99%); [00:15:28] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/e46d7909ac59f915507bb5a3c00beb9feb274df6 [00:15:28] 02ssl/03main 07WikiTideBot 03e46d790 Bot: Auto-update domain lists [00:25:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f41e0a30be856dc31d092cadc14a1be4aa954a4d [00:25:29] 02ssl/03main 07WikiTideBot 03f41e0a3 Bot: Auto-update domain lists [00:49:13] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [05:32:05] [02CreateWiki] 07AgentIsai created 03handleai from 03main (+0 new commit) 13https://github.com/miraheze/CreateWiki/compare/handleai [05:40:31] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03handleai 13https://github.com/miraheze/CreateWiki/commit/a7727f0653acd2cfffdf7d160d5c02609a695988 [05:40:31] 02CreateWiki/03handleai 07Agent Isai 03a7727f0 Add support for "handle with AI" button [05:41:10] [02CreateWiki] 07AgentIsai opened pull request #782: Add support for enqueuing wiki requests for AI evaluation (03main...03handleai) 13https://github.com/miraheze/CreateWiki/pull/782 [05:41:17] [02CreateWiki] 07coderabbitai[bot] commented on pull request #782:

[…] 13https://github.com/miraheze/CreateWiki/pull/782#issuecomment-3336852361 [05:45:45] miraheze/CreateWiki - AgentIsai the build has errored. [06:14:06] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03handleai 13https://github.com/miraheze/CreateWiki/commit/9e0eeb9ea6d8eb6d6a41ead1a26dcc0b6f49664d [06:14:06] 02CreateWiki/03handleai 07Agent Isai 039e0eeb9 Fix function call [06:18:27] miraheze/CreateWiki - AgentIsai the build has errored. [08:01:13] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [09:15:43] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 41% [09:27:22] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 28% [10:00:22] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44% [10:02:18] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 39% [11:07:04] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 48% [11:09:00] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 32% [11:38:09] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [11:42:09] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 39% [12:48:40] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'HAWelcome'} to all [12:48:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:49:00] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'HAWelcome'} to test151 [12:49:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:50:52] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'HAWelcome'} to all - SUCCESS in 131s [12:50:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:52:50] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Cargo'} to all [12:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:54:28] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 98s [12:54:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:59:30] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'HAWelcome'} to test151 - SUCCESS in 630s [12:59:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:08:25] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Cargo'} to test151 [13:08:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:10:11] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Cargo'} to test151 - SUCCESS in 105s [13:10:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:19] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1950MiB (10% inode=89%); [14:48:20] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.95, 8.89, 4.97 [14:50:19] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.04, 7.45, 4.98 [15:01:19] PROBLEM - matomo151 Disk Space on matomo151 is CRITICAL: DISK CRITICAL - free space: / 741MiB (4% inode=89%); [15:03:19] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6419MiB (35% inode=89%); [15:46:10] that's a lot of... problems [15:47:34] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27153MiB (5% inode=99%); [15:53:44] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [15:53:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:06] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 22s [15:54:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:06] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [15:55:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:27] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 20s [15:55:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:47] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [15:56:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:07] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 20s [15:57:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:08] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [15:58:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:28] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 20s [15:58:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:50] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [15:58:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:59:10] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 20s [15:59:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:59:42] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [15:59:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:03] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 21s [16:00:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:00:09] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27199MiB (5% inode=99%); [16:03:10] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [16:03:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:52] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [16:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:13] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 21s [16:04:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:04] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [16:05:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:24] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 20s [16:05:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:32] just a few cargo upgrades [16:09:47] !log [paladox@mwtask181] starting deploy of {'folders': '1.44/extensions/Cargo'} to all [16:09:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:08] !log [paladox@mwtask181] finished deploy of {'folders': '1.44/extensions/Cargo'} to all - SUCCESS in 20s [16:10:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:20] !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Cargo'} to all [16:36:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:39] !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Cargo'} to test151 [16:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:58] !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 97s [16:38:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:26] !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Cargo'} to test151 - SUCCESS in 106s [16:38:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:13] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27256MiB (5% inode=99%); [18:04:19] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.44, 12.06, 6.69 [18:06:19] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.18, 8.56, 6.05 [18:52:02] !log made rhinosf1 a superuser in matomo [18:52:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:31] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/69a1d85af72261f0a94f34d6bb22735c79e54f59 [20:05:31] 02ssl/03main 07WikiTideBot 0369a1d85 Bot: Auto-update domain lists [21:04:19] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 18.62, 8.07 [21:04:30] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.84, 16.41, 9.60 [21:06:25] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.98, 12.78, 9.07 [21:08:04] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 21.84, 11.34 [21:09:50] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:09:51] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [21:09:55] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 12 backends are down. mw151 mw161 mw162 mw172 mw181 mw173 mw183 mw191 mw192 mw201 mw202 mw203 [21:09:56] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [21:10:00] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 10.02, 17.10, 10.81 [21:10:15] Thats not good [21:10:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 15 backends are down. mw152 mw161 mw162 mw172 mw181 mw182 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [21:10:32] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 14 backends are down. mw152 mw161 mw162 mw171 mw181 mw182 mw153 mw163 mw183 mw192 mw193 mw201 mw202 mw203 [21:10:46] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 30.75, 17.88, 9.87 [21:10:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw152 mw162 mw182 [21:11:44] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.073 second response time [21:11:47] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [21:11:54] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4310 bytes in 0.069 second response time [21:12:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 9.93, 15.01, 9.85 [21:12:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [21:13:07] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.27, 23.32, 12.79 [21:13:44] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [21:14:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [21:14:32] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [21:15:06] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 5.58, 16.87, 11.66 [21:26:19] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 3.66, 6.69, 11.18 [21:28:19] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.32, 4.78, 9.94 [21:40:19] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 69.83, 35.95, 18.95 [21:40:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 13 backends are down. mw151 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [21:40:32] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw151 mw163 mw173 mw183 mw191 mw193 mw201 mw202 mw203 [21:41:00] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw161 mw172 mw163 mw191 [21:41:20] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 5 backends are down. mw152 mw161 mw172 mw181 mw182 [21:44:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [21:44:32] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [21:44:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [21:45:12] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [21:54:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw162 mw192 [21:54:32] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw162 [21:54:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw162 [21:56:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [21:56:32] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [21:56:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [22:02:19] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.66, 5.38, 11.51 [22:04:19] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.54, 3.78, 10.18 [22:14:20] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.49, 13.97, 10.78 [22:14:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 9 backends are down. mw152 mw161 mw162 mw182 mw173 mw191 mw201 mw202 mw203 [22:15:01] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:15:12] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [22:15:23] bad gateway [22:15:24] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:15:30] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:15:40] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.79, 17.25, 9.68 [22:15:50] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.08, 16.76, 9.32 [22:16:09] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43% [22:16:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw171 [22:16:32] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw171 [22:16:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [22:16:55] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [22:17:12] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [22:17:24] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.976 second response time [22:17:36] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 12.23, 15.04, 9.81 [22:17:48] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 13.12, 13.98, 9.11 [22:18:08] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.71, 20.38, 9.74 [22:18:09] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [22:18:09] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 19% [22:19:38] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.966 second response time [22:20:06] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 5.97, 14.73, 8.91 [22:20:10] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:21:08] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 29.60, 16.79 [22:22:08] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.062 second response time [22:22:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [22:22:32] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [22:23:02] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 10.67, 22.43, 15.65 [22:24:40] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:25:14] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 58.97, 37.65, 20.14 [22:25:28] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 67.40, 40.47, 23.28 [22:26:09] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [22:26:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw161 mw162 mw192 [22:26:30] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 26.85, 23.75, 15.02 [22:26:32] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 5 backends are down. mw151 mw161 mw162 mw192 mw193 [22:26:54] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw153 mw173 mw191 mw193 mw201 mw202 mw203 [22:27:12] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 14 backends are down. mw152 mw161 mw162 mw171 mw172 mw181 mw153 mw163 mw173 mw191 mw193 mw201 mw202 mw203 [22:27:24] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:32] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 34.17, 22.57, 14.68 [22:28:45] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time [22:28:53] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:29:06] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:10] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.82, 19.51, 13.04 [22:29:10] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:29:11] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:14] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:29:24] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.985 second response time [22:29:28] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 21.40, 23.60, 16.12 [22:30:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [22:30:32] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [22:30:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [22:30:58] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.682 second response time [22:31:06] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 10.65, 15.93, 12.51 [22:31:19] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 13.76, 23.75, 21.38 [22:32:05] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 35.95, 24.33, 17.22 [22:32:05] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [22:32:25] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [22:32:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 60.98, 31.76, 16.16 [22:32:33] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:46] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:32:53] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.057 second response time [22:33:09] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 6.40, 20.76, 20.74 [22:33:11] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.955 second response time [22:33:12] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.396 second response time [22:33:14] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time [22:33:18] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 11.04, 20.00, 20.33 [22:33:23] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4309 bytes in 2.847 second response time [22:33:27] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:33:59] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [22:34:32] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.229 second response time [22:34:43] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.054 seconds response time. mwtask181.fsslc.wtnet returns 10.0.18.106 [22:35:05] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 4.39, 15.34, 18.76 [22:35:12] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [22:35:26] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.066 second response time [22:35:59] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 10.14, 18.79, 16.96 [22:36:22] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 4.50, 18.42, 14.38 [22:38:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 12 backends are down. mw151 mw161 mw162 mw181 mw182 mw153 mw163 mw173 mw183 mw201 mw202 mw203 [22:38:32] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw161 mw162 mw181 mw182 mw173 mw183 mw201 mw202 mw203 [22:38:36] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:38:42] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:38:43] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:38:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 12 backends are down. mw152 mw161 mw172 mw181 mw153 mw163 mw183 mw191 mw192 mw193 mw201 mw203 [22:39:07] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [22:39:09] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:39:11] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 25.30, 21.07, 17.14 [22:39:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.81, 16.76, 13.44 [22:39:12] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw153 mw163 mw183 mw191 mw192 mw193 mw201 mw203 [22:39:18] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [22:39:34] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.65, 20.39, 16.20 [22:39:59] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:40:00] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:40:06] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [22:40:13] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:25] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 25.27, 29.41, 22.27 [22:40:40] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.209 second response time [22:40:56] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 30.49, 31.17, 18.37 [22:41:01] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 20.45, 15.27 [22:41:02] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:09] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.853 second response time [22:41:11] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 34.53, 22.91, 16.08 [22:41:19] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 33.43, 20.60, 14.19 [22:41:24] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:41:25] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:32] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.73, 22.52, 17.64 [22:41:37] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:43] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:54] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.69, 23.94, 16.74 [22:42:03] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:42:05] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.558 second response time [22:42:11] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.061 second response time [22:42:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.52, 23.05, 16.02 [22:42:32] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.059 second response time [22:42:34] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.134 second response time [22:42:59] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.461 second response time [22:43:05] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:07] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 17.77, 23.38, 19.33 [22:43:18] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.373 second response time [22:43:34] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.354 second response time [22:43:52] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 39.56, 29.21, 19.45 [22:44:00] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:44:03] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:05] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:44:05] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 6.583 second response time [22:44:09] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 14.54, 22.85, 22.77 [22:44:44] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 16.00, 22.58, 17.66 [22:45:08] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 35.16, 27.20, 21.15 [22:45:08] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 19.88, 22.20, 17.36 [22:45:10] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [22:45:14] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 13.15, 22.25, 16.82 [22:45:16] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.054 second response time [22:45:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.12, 19.21, 17.44 [22:45:54] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.058 second response time [22:46:03] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 4.679 second response time [22:46:06] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 27.13, 24.44 [22:46:13] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 14.87, 22.47, 21.70 [22:46:13] !log restart mariadb on db171 [22:46:15] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.562 second response time [22:46:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 7.47, 16.60, 15.09 [22:46:34] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:36] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:39] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 6.90, 17.78, 16.51 [22:47:03] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [22:47:06] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 4.82, 19.23, 17.81 [22:47:07] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 5.78, 16.97, 16.10 [22:47:11] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 3.29, 15.77, 15.10 [22:47:34] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:47:43] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:47:46] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 3.09, 17.56, 17.19 [22:48:08] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 16.37, 22.10, 22.87 [22:48:08] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 2.86, 15.78, 19.37 [22:48:08] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [22:48:41] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:48:41] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [22:48:42] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.016 second response time [22:48:46] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [22:48:47] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [22:48:47] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:52] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:48:52] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [22:48:55] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [22:49:04] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 3.58, 18.82, 19.60 [22:49:10] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:49:11] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [22:49:16] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:17] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:18] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:49:21] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:49:23] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:49:30] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [22:49:51] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [22:50:07] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.058 second response time [22:50:13] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 30.78, 24.71, 23.66 [22:50:17] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 2.241 second response time [22:50:22] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [22:50:24] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [22:50:29] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.087 second response time [22:50:30] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:39] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [22:50:41] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:49] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4310 bytes in 2.952 second response time [22:50:51] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43% [22:50:52] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.886 second response time [22:51:02] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 238s [22:51:04] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.066 second response time [22:51:05] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.057 second response time [22:51:13] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.070 second response time [22:51:16] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.067 second response time [22:51:21] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.058 second response time [22:51:29] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.052 second response time [22:51:45] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.072 second response time [22:52:07] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 10.48, 19.84, 22.06 [22:52:18] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.072 second response time [22:52:48] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 25% [22:53:03] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [22:53:05] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.092 second response time [22:53:12] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [22:53:25] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 22s [22:53:27] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [22:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:32] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [22:53:42] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time [22:53:57] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.162 second response time [22:54:06] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 4.00, 14.29, 19.76 [22:54:08] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [22:54:20] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [22:54:26] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.244 second response time [22:54:27] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [22:54:27] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.185 second response time [22:54:32] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [22:54:36] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [22:54:37] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [22:54:38] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time [22:54:42] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.167 second response time [22:54:43] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.158 second response time [22:54:43] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [22:54:55] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [22:55:02] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time [22:55:02] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.234 second response time [22:55:04] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.182 second response time [22:55:10] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.170 second response time [22:55:11] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time [22:56:09] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [23:00:16] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:04:47] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:48] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:04:49] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:04:50] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:55] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 9 backends are down. mw151 mw162 mw172 mw182 mw153 mw173 mw191 mw192 mw201 [23:04:55] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:12] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 11 backends are down. mw151 mw161 mw162 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw201 [23:05:14] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [23:05:15] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.60, 16.77, 14.58 [23:05:21] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:05:25] PROBLEM - mw182 SSH on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:37] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 24.83, 20.34, 14.82 [23:05:38] PROBLEM - mw182 PowerDNS Recursor on mw182 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:05:40] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:05:59] PROBLEM - mw201 PowerDNS Recursor on mw201 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:06:19] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 32.65, 25.11, 19.70 [23:06:20] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw162 mw171 mw181 mw182 mw153 mw163 mw173 mw191 mw201 mw203 [23:06:22] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:06:25] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 43% [23:06:31] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 47.87, 29.61, 19.05 [23:06:32]