[00:01:13] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/11a5984ee8616c87e8d5aeb2750ecb1b2f5720c3 [00:01:13] 02statichelp/03main 07WikiTideBot 0311a5984 Bot: Auto-update Tech namespace pages 2025-09-25 00:01:10 [00:09:23] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 54418MiB (11% inode=99%); [00:12:06] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 55547MiB (12% inode=99%); [00:12:07] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 54401MiB (11% inode=99%); [01:31:51] !log [blankeclair@mwtask181] Starting import for rubaldisbasicswiki (XML: None; Images: all_images/) (START) [01:31:52] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=rubaldisbasicswiki --sleep=1 '--comment=Importing images from https://baldis-basics-in-education-and-learning.fandom.com/ru ([[phorge:T14313|T14313]])' -- all_images/ (START) [01:31:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:59] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ollama 13https://github.com/miraheze/CreateWiki/commit/b19f56199313e2ce83206d88e2cd085209205f86 [02:27:00] 02CreateWiki/03ollama 07Agent Isai 03b19f561 Add JSON format request [02:48:32] miraheze/CreateWiki - AgentIsai the build passed. [02:53:23] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ollama 13https://github.com/miraheze/CreateWiki/commit/25d46defc2dd77ae6a0421addba6d5752855f135 [02:53:23] 02CreateWiki/03ollama 07Agent Isai 0325d46de Update [02:55:41] [02CreateWiki] 07github-actions[bot] pushed 1 new commit to 03ollama 13https://github.com/miraheze/CreateWiki/commit/9758497942aa04d2ee38667e4bceebeb6637ecdf [02:55:41] 02CreateWiki/03ollama 07github-actions 039758497 CI: lint code to MediaWiki standards… [02:57:53] miraheze/CreateWiki - AgentIsai the build has errored. [03:00:18] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ollama 13https://github.com/miraheze/CreateWiki/commit/32c8a05d47baeafb4d3f9dee2c423e2ac60be3a9 [03:00:18] 02CreateWiki/03ollama 07Agent Isai 0332c8a05 Fix [03:16:23] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03ollama 13https://github.com/miraheze/CreateWiki/commit/a8fa02d5dec62ff6909ceb1cbde59aa290b4d06f [03:16:23] 02CreateWiki/03ollama 07Agent Isai 03a8fa02d Fix API responses [03:19:46] [02CreateWiki] 07AgentIsai commented on pull request #781: @coderabbitai review 13https://github.com/miraheze/CreateWiki/pull/781#issuecomment-3331896541 [03:19:52] [02CreateWiki] 07coderabbitai[bot] commented on pull request #781:
[…] 13https://github.com/miraheze/CreateWiki/pull/781#issuecomment-3331897249 [03:27:18] !log [void@puppet181] Upgraded packages on bots171: libxslt1.1 [03:27:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:27:32] !log [void@puppet181] Upgraded packages on cloud16: libxslt1.1 and xsltproc [03:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:27:40] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 186 packages available for upgrade (0 critical updates). [03:27:46] !log [void@puppet181] Upgraded packages on cloud15: libxslt1.1 and xsltproc [03:27:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:08] !log [void@puppet181] Upgraded packages on cloud17: libxslt1.1 and xsltproc [03:28:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:14] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 189 packages available for upgrade (0 critical updates). [03:28:50] !log [void@puppet181] Upgraded packages on cp191: libxslt1.1 [03:28:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:54] RECOVERY - cp191 APT on cp191 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:29:07] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 186 packages available for upgrade (0 critical updates). [03:29:07] RECOVERY - bots171 APT on bots171 is OK: APT OK: 117 packages available for upgrade (0 critical updates). [03:29:09] !log [void@puppet181] Upgraded packages on cp171: libxslt1.1 [03:29:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:29:24] !log [void@puppet181] Upgraded packages on cloud20: libxslt1.1 and xsltproc [03:29:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:29:36] !log [void@puppet181] Upgraded packages on cloud19: libxslt1.1 and xsltproc [03:29:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:29:53] !log [void@puppet181] Upgraded packages on cp201: libxslt1.1 [03:29:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:29:57] RECOVERY - cp201 APT on cp201 is OK: APT OK: 82 packages available for upgrade (0 critical updates). [03:30:05] RECOVERY - cloud20 APT on cloud20 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [03:30:07] !log [void@puppet181] Upgraded packages on cloud18: libxslt1.1 and xsltproc [03:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:13] RECOVERY - cloud19 APT on cloud19 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:30:46] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 186 packages available for upgrade (0 critical updates). [03:30:59] !log [void@puppet181] Upgraded packages on mattermost1: libxslt1.1 [03:31:00] RECOVERY - cp171 APT on cp171 is OK: APT OK: 112 packages available for upgrade (0 critical updates). [03:31:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:11] !log [void@puppet181] Upgraded packages on matomo151: libxslt1.1 [03:31:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:41] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 114 packages available for upgrade (0 critical updates). [03:31:54] !log [void@puppet181] Upgraded packages on mon181: libxslt1.1 [03:31:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:00] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 74 packages available for upgrade (0 critical updates). [03:32:07] !log [void@puppet181] Upgraded packages on mw152: libxslt1.1 [03:32:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:11] RECOVERY - mon181 APT on mon181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:32:22] !log [void@puppet181] Upgraded packages on mw162: libxslt1.1 [03:32:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:36] !log [void@puppet181] Upgraded packages on mw151: libxslt1.1 [03:32:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:41] RECOVERY - mw162 APT on mw162 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:32:49] !log [void@puppet181] Upgraded packages on mw153: libxslt1.1 [03:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:04] !log [void@puppet181] Upgraded packages on mw161: libxslt1.1 [03:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:17] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=rubaldisbasicswiki --sleep=1 '--comment=Importing images from https://baldis-basics-in-education-and-learning.fandom.com/ru ([[phorge:T14313|T14313]])' -- all_images/ (END - exit=0) [03:33:17] RECOVERY - mw152 APT on mw152 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:33:18] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=rubaldisbasicswiki --update (START) [03:33:19] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=rubaldisbasicswiki --update (END - exit=0) [03:33:20] !log [blankeclair@mwtask181] Finished import for rubaldisbasicswiki (XML: None; Images: all_images/) (END - exit=0) [03:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:21] !log [void@puppet181] Upgraded packages on mw172: libxslt1.1 [03:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:33] !log [void@puppet181] Upgraded packages on mw173: libxslt1.1 [03:33:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:47] !log [void@puppet181] Upgraded packages on mw163: libxslt1.1 [03:33:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:56] RECOVERY - mw151 APT on mw151 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [03:34:00] !log [void@puppet181] Upgraded packages on mw171: libxslt1.1 [03:34:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:09] RECOVERY - mw161 APT on mw161 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:14] !log [void@puppet181] Upgraded packages on mw193: libxslt1.1 [03:34:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:23] RECOVERY - mw153 APT on mw153 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:28] !log [void@puppet181] Upgraded packages on mw181: libxslt1.1 [03:34:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:40] RECOVERY - mw193 APT on mw193 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:41] RECOVERY - mw172 APT on mw172 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:43] !log [void@puppet181] Upgraded packages on mw191: libxslt1.1 [03:34:43] RECOVERY - mw173 APT on mw173 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:49] RECOVERY - mw171 APT on mw171 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:52] RECOVERY - mw191 APT on mw191 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:54] RECOVERY - mw181 APT on mw181 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:34:56] !log [void@puppet181] Upgraded packages on mw192: libxslt1.1 [03:35:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:05] RECOVERY - mw163 APT on mw163 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:35:10] !log [void@puppet181] Upgraded packages on mw203: libxslt1.1 [03:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:25] !log [void@puppet181] Upgraded packages on mw182: libxslt1.1 [03:35:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:40] !log [void@puppet181] Upgraded packages on mw183: libxslt1.1 [03:35:42] RECOVERY - mw182 APT on mw182 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:35:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:54] !log [void@puppet181] Upgraded packages on mw202: libxslt1.1 [03:35:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:09] !log [void@puppet181] Upgraded packages on mw201: libxslt1.1 [03:36:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:22] RECOVERY - mw192 APT on mw192 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:36:24] !log [void@puppet181] Upgraded packages on mwtask151: libxslt1.1 [03:36:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:38] !log [void@puppet181] Upgraded packages on mwtask161: libxslt1.1 [03:36:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:52] RECOVERY - mw203 APT on mw203 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:36:52] !log [void@puppet181] Upgraded packages on mwtask171: libxslt1.1 [03:36:56] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [03:36:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:03] RECOVERY - mw201 APT on mw201 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:37:15] RECOVERY - mw183 APT on mw183 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:37:26] !log [void@puppet181] Upgraded packages on phorge171: libxslt1.1 [03:37:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:45] RECOVERY - mw202 APT on mw202 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [03:37:46] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [03:37:47] miraheze/CreateWiki - AgentIsai the build passed. [03:38:07] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [03:38:09] !log [void@puppet181] Upgraded packages on reports171: libxslt1.1 [03:38:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:19] [02CreateWiki] 07AgentIsai merged pull request #781: Refactor to use Ollama instead of OpenAI (03main...03ollama) 13https://github.com/miraheze/CreateWiki/pull/781 [03:38:19] [02CreateWiki] 07AgentIsai pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/e9071693e89d4f015984414b870fb56dece33455 [03:38:20] 02CreateWiki/03main 07Agent Isai 03e907169 Refactor to use Ollama instead of OpenAI (#781)… [03:38:22] [02CreateWiki] 07AgentIsai 04deleted 03ollama at 03a8fa02d 13https://api.github.com/repos/miraheze/CreateWiki/commit/a8fa02d [03:38:37] !log [void@puppet181] Upgraded packages on mwtask181: libxslt1.1 [03:38:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:50] !log [void@puppet181] Upgraded packages on puppet181: libxslt1.1 [03:38:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:02] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:39:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:12] !log [void@puppet181] Upgraded packages on swiftobject151: libxslt1.1 [03:39:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:17] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [03:39:23] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [03:39:24] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [03:39:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:35] RECOVERY - reports171 APT on reports171 is OK: APT OK: 115 packages available for upgrade (0 critical updates). [03:39:36] !log [void@puppet181] Upgraded packages on swiftac171: libxslt1.1 [03:39:39] !log [agent@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all [03:39:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:51] !log [void@puppet181] Upgraded packages on swiftproxy161: libxslt1.1 [03:39:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:03] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [03:40:04] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 126 packages available for upgrade (0 critical updates). [03:40:06] !log [void@puppet181] Upgraded packages on swiftobject191: libxslt1.1 [03:40:07] !log [agent@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 27s [03:40:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:15] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [03:40:20] !log [void@puppet181] Upgraded packages on swiftobject161: libxslt1.1 [03:40:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:35] !log [void@puppet181] Upgraded packages on swiftobject181: libxslt1.1 [03:40:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:50] !log [void@puppet181] Upgraded packages on swiftobject201: libxslt1.1 [03:40:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:58] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [03:40:59] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [03:41:05] !log [void@puppet181] Upgraded packages on swiftobject171: libxslt1.1 [03:41:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:20] !log [void@puppet181] Upgraded packages on swiftproxy171: libxslt1.1 [03:41:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:28] RECOVERY - swiftobject191 APT on swiftobject191 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [03:41:34] !log [void@puppet181] Upgraded packages on test151: libxslt1.1 [03:41:35] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [03:41:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:39] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 128 packages available for upgrade (0 critical updates). [03:42:34] RECOVERY - test151 APT on test151 is OK: APT OK: 120 packages available for upgrade (0 critical updates). [03:42:46] RECOVERY - swiftobject201 APT on swiftobject201 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [03:43:03] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [04:00:05] miraheze/CreateWiki - AgentIsai the build passed. [04:02:07] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49832MiB (10% inode=99%); [04:03:23] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49866MiB (10% inode=99%); [04:16:41] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:17:15] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 63.36, 33.39, 14.52 [04:17:32] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 60.17, 32.18, 14.60 [04:18:39] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [04:21:11] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 23.12, 20.53, 10.50 [04:22:35] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 15.07, 21.56, 12.38 [04:24:34] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 15.82, 19.33, 12.63 [04:25:05] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 10.94, 17.22, 11.71 [04:29:00] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:01] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:29:03] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:04] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:29:52] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:29:55] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [04:29:59] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:30:29] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 23.25, 16.22 [04:30:35] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [04:30:49] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:30:50] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:31:04] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [04:31:05] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.416 seconds response time. mwtask181.fsslc.wtnet returns 10.0.18.106 [04:31:09] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 41.77, 26.82, 16.76 [04:31:37] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:32:00] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:06] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.358 second response time [04:32:27] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 5.98, 16.58, 14.61 [04:32:30] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.509 second response time [04:32:44] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.071 second response time [04:32:48] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [04:33:07] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.035 second response time [04:33:09] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.526 second response time [04:33:38] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [04:33:52] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.038 second response time [04:33:55] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044 [04:33:57] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [04:35:03] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 8.75, 20.55, 16.89 [04:37:03] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 12.98, 19.02, 16.78 [04:37:04] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [04:38:22] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 22.65, 23.60, 18.22 [04:39:04] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.077 second response time [04:40:21] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 5.17, 16.63, 16.32 [04:42:10] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 23.22, 22.25, 14.42 [04:44:10] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 5.77, 16.05, 13.10 [04:45:33] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:47:28] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.066 second response time [04:53:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 2.95, 11.26, 22.54 [04:54:32] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:54:33] PROBLEM - mwtask151 PowerDNS Recursor on mwtask151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:54:49] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:55:34] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:55:36] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 3.95, 8.65, 20.20 [04:56:06] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:25] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:56:40] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 74.42, 42.74, 23.88 [04:56:57] PROBLEM - mwtask151 conntrack_table_size on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:56:58] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:58] PROBLEM - mwtask151 php-fpm on mwtask151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:57:10] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:57:24] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:59:53] PROBLEM - mwtask151 mathoid on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 10044: Connection refused [05:01:00] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 8.348 second response time [05:01:04] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [05:01:04] RECOVERY - mwtask151 php-fpm on mwtask151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [05:01:46] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [05:01:51] RECOVERY - mwtask151 mathoid on mwtask151 is OK: TCP OK - 0.000 second response time on 10.0.15.150 port 10044 [05:02:04] RECOVERY - mwtask151 conntrack_table_size on mwtask151 is OK: OK: nf_conntrack is 0 % full [05:02:06] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49911MiB (10% inode=99%); [05:02:27] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.061 second response time [05:02:38] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [05:02:41] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [05:02:42] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [05:02:44] RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 0.359 seconds response time. mwtask151.fsslc.wtnet returns 10.0.15.150 [05:03:15] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:03:48] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 90.54, 41.75, 27.52 [05:03:54] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:04:02] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:04:15] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:04:39] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:04:41] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:05:55] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [05:06:07] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 1.029 second response time. mwtask181.fsslc.wtnet returns 10.0.18.106 [05:06:07] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:06:19] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:06:40] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 1.185 second response time [05:06:40] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 2.713 second response time [05:06:46] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:07:01] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:07:13] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:07:16] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:08:04] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [05:08:33] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 42.06, 24.96, 17.23 [05:08:41] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.133 second response time [05:08:58] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.907 second response time [05:09:02] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:09:08] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.356 second response time [05:10:33] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.698 second response time [05:10:39] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:10:44] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time [05:11:02] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [05:11:14] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [05:11:24] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.171 second response time [05:11:30] PROBLEM - mwtask161 mathoid on mwtask161 is CRITICAL: connect to address 10.0.16.157 and port 10044: Connection refused [05:11:54] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:11:55] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044 [05:11:58] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:21] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [05:13:25] RECOVERY - mwtask161 mathoid on mwtask161 is OK: TCP OK - 0.000 second response time on 10.0.16.157 port 10044 [05:13:48] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:51] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [05:14:00] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.227 second response time [05:15:12] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [05:15:39] PROBLEM - mwtask151 mathoid on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 10044: Connection refused [05:15:46] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.266 second response time [05:16:16] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time [05:16:17] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:17:37] RECOVERY - mwtask151 mathoid on mwtask151 is OK: TCP OK - 0.000 second response time on 10.0.15.150 port 10044 [05:18:11] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:18:13] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:18:13] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:18:16] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [05:20:06] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [05:20:08] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [05:20:10] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [05:22:34] PROBLEM - mwtask161 Current Load on mwtask161 is WARNING: LOAD WARNING - total load average: 3.40, 19.92, 22.59 [05:22:44] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/opt/mcrouter_2023.07.17.00-1_amd64.deb],File[/etc/ferm/conf.d/02_main] [05:23:10] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 2.47, 12.34, 22.96 [05:24:32] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 2.56, 13.99, 20.08 [05:26:09] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:17] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 33.55, 23.38, 15.03 [05:27:06] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 27.71, 13.42, 20.49 [05:28:13] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 6.79, 13.82, 22.64 [05:28:52] PROBLEM - mwtask151 PowerDNS Recursor on mwtask151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:30:16] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.674 second response time [05:30:24] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 15.47, 22.12, 16.51 [05:30:50] RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 1.992 second response time. mwtask151.fsslc.wtnet returns 10.0.15.150 [05:32:07] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 32.72, 17.14, 21.51 [05:32:11] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:32:22] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 3.51, 15.41, 14.71 [05:33:03] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 4.17, 15.56, 20.34 [05:33:28] PROBLEM - mwtask161 SSH on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:36] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:33:36] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:40] PROBLEM - mwtask161 PowerDNS Recursor on mwtask161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:34:01] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:34:23] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:34:55] PROBLEM - mwtask161 mathoid on mwtask161 is CRITICAL: connect to address 10.0.16.157 and port 10044: Connection refused [05:35:14] PROBLEM - mwtask161 Current Load on mwtask161 is CRITICAL: LOAD CRITICAL - total load average: 72.75, 46.29, 29.56 [05:35:39] PROBLEM - mwtask161 conntrack_table_size on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:36:19] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.404 second response time [05:36:21] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.046 second response time [05:36:33] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.22, 22.21, 22.77 [05:36:55] RECOVERY - mwtask161 mathoid on mwtask161 is OK: TCP OK - 0.000 second response time on 10.0.16.157 port 10044 [05:37:31] RECOVERY - mwtask161 SSH on mwtask161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [05:37:38] RECOVERY - mwtask161 conntrack_table_size on mwtask161 is OK: OK: nf_conntrack is 0 % full [05:37:43] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [05:37:45] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.065 second response time [05:37:50] RECOVERY - mwtask161 PowerDNS Recursor on mwtask161 is OK: DNS OK: 0.327 seconds response time. mwtask161.fsslc.wtnet returns 10.0.16.157 [05:38:00] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [05:38:30] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 31.99, 28.34, 25.08 [05:39:01] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:25] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 25.66, 22.30, 20.86 [05:41:22] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 6.67, 16.18, 18.79 [05:42:02] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:42:22] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.67, 21.16, 16.97 [05:43:13] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.476 second response time [05:44:02] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:38] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:44:54] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:44:56] PROBLEM - mwtask171 mathoid on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 10044: Connection refused [05:44:57] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:12] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:17] PROBLEM - mwtask151 mathoid on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 10044: Connection refused [05:45:35] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 59.42, 32.94, 24.28 [05:45:35] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:45:42] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:46] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:46:10] PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:46:41] PROBLEM - mwtask151 PowerDNS Recursor on mwtask151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:46:49] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:47:00] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:47:07] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:47:18] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:47:21] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:23] PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [05:47:38] PROBLEM - mwtask161 PowerDNS Recursor on mwtask161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:47:38] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:47:39] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:47:43] PROBLEM - mwtask171 PowerDNS Recursor on mwtask171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:47:43] PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:48:02] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:48:26] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:48:55] PROBLEM - mwtask161 mathoid on mwtask161 is CRITICAL: connect to address 10.0.16.157 and port 10044: Connection refused [05:48:57] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:59] PROBLEM - mwtask171 php-fpm on mwtask171 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:49:14] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.720 second response time [05:49:18] PROBLEM - mwtask151 conntrack_table_size on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:49:28] PROBLEM - mwtask151 ferm_active on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:49:46] PROBLEM - mwtask161 APT on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:50:09] PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:50:12] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:50:16] PROBLEM - mwtask171 conntrack_table_size on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:50:30] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [05:50:46] PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [05:52:44] RECOVERY - mwtask171 ferm_active on mwtask171 is OK: OK ferm input default policy is set [05:52:44] RECOVERY - mwtask171 conntrack_table_size on mwtask171 is OK: OK: nf_conntrack is 0 % full [05:52:45] PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:52:50] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [05:52:54] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:52:57] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [05:53:04] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.753 second response time [05:53:06] RECOVERY - mwtask171 php-fpm on mwtask171 is OK: PROCS OK: 24 processes with command name 'php-fpm8.2' [05:53:10] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:53:11] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:53:20] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.644 second response time [05:53:21] PROBLEM - mwtask161 SSH on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:34] PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:53:53] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 7.385 second response time [05:54:00] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:08] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:11] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [05:55:12] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.193 second response time [05:55:27] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.077 second response time [05:56:03] RECOVERY - mwtask171 PowerDNS Recursor on mwtask171 is OK: DNS OK: 2.176 seconds response time. mwtask171.fsslc.wtnet returns 10.0.17.144 [05:56:30] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:56:43] RECOVERY - mwtask171 mathoid on mwtask171 is OK: TCP OK - 0.000 second response time on 10.0.17.144 port 10044 [05:57:23] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:57:28] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [05:58:09] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [05:59:35] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [05:59:36] RECOVERY - mwtask161 SSH on mwtask161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [05:59:38] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [05:59:40] PROBLEM - mwtask151 php-fpm on mwtask151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:59:45] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:59:50] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:12] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:00:23] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:28] PROBLEM - mwtask171 PowerDNS Recursor on mwtask171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:00:36] PROBLEM - mwtask171 mathoid on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 10044: Connection refused [06:00:37] PROBLEM - mwtask171 php-fpm on mwtask171 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:02:23] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:02:59] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:03:11] PROBLEM - mwtask161 php-fpm on mwtask161 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:03:45] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 9.131 second response time [06:03:45] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.029 second response time [06:04:02] PROBLEM - mwtask161 SSH on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:25] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:04:55] PROBLEM - mwtask181 conntrack_table_size on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:12] RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 6.108 second response time [06:06:27] PROBLEM - mwtask161 ferm_active on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:06:28] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 8.719 second response time [06:06:53] RECOVERY - mwtask171 php-fpm on mwtask171 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:06:53] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 6.352 second response time [06:07:00] RECOVERY - mwtask181 conntrack_table_size on mwtask181 is OK: OK: nf_conntrack is 0 % full [06:07:07] RECOVERY - mwtask151 ferm_active on mwtask151 is OK: OK ferm input default policy is set [06:08:07] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:08:11] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:08:11] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.046 second response time [06:09:22] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:09:35] PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:09:35] PROBLEM - mwtask161 conntrack_table_size on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:10:08] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.784 second response time [06:10:11] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 1.325 second response time [06:10:15] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [06:10:21] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [06:10:49] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:51] PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:11:14] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:11:22] PROBLEM - mwtask171 php-fpm on mwtask171 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:12:27] RECOVERY - mwtask151 conntrack_table_size on mwtask151 is OK: OK: nf_conntrack is 0 % full [06:12:28] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.868 second response time [06:12:30] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.017 second response time [06:12:31] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.430 second response time [06:12:32] RECOVERY - mwtask151 php-fpm on mwtask151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:12:37] PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [06:13:03] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:17] RECOVERY - mwtask151 mathoid on mwtask151 is OK: TCP OK - 0.000 second response time on 10.0.15.150 port 10044 [06:13:18] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:13:21] PROBLEM - mwtask171 conntrack_table_size on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:13:53] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:14:29] PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:14:36] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:15:04] RECOVERY - mwtask161 ferm_active on mwtask161 is OK: OK ferm input default policy is set [06:15:04] RECOVERY - mwtask161 conntrack_table_size on mwtask161 is OK: OK: nf_conntrack is 0 % full [06:15:18] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 2.453 second response time [06:15:39] RECOVERY - mwtask161 php-fpm on mwtask161 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:16:32] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044 [06:16:51] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:17:10] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:16] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:17] PROBLEM - mwtask151 mathoid on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 10044: Connection refused [06:17:46] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 3.244 second response time [06:18:02] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.244 second response time [06:18:07] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:18:07] PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:19:14] RECOVERY - mwtask171 conntrack_table_size on mwtask171 is OK: OK: nf_conntrack is 0 % full [06:19:38] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:20:05] PROBLEM - mwtask161 php-fpm on mwtask161 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:20:22] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [06:20:56] RECOVERY - mwtask171 ferm_active on mwtask171 is OK: OK ferm input default policy is set [06:21:17] RECOVERY - mwtask151 mathoid on mwtask151 is OK: TCP OK - 0.000 second response time on 10.0.15.150 port 10044 [06:22:05] PROBLEM - mwtask161 conntrack_table_size on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:22:41] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 5.072 seconds response time. mwtask181.fsslc.wtnet returns 10.0.18.106 [06:24:00] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [06:24:00] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:03] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:24:07] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:24:16] RECOVERY - mwtask161 php-fpm on mwtask161 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:24:54] RECOVERY - mwtask161 PowerDNS Recursor on mwtask161 is OK: DNS OK: 6.927 seconds response time. mwtask161.fsslc.wtnet returns 10.0.16.157 [06:24:57] RECOVERY - mwtask161 SSH on mwtask161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:25:21] PROBLEM - mwtask151 mathoid on mwtask151 is CRITICAL: connect to address 10.0.15.150 and port 10044: Connection refused [06:25:23] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:26:08] PROBLEM - mwtask171 conntrack_table_size on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:27:16] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:27:27] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:27:49] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.49, 16.83, 10.14 [06:27:53] PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:34] PROBLEM - mwtask161 ferm_active on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:50] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-expired-blocks on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:53] PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:55] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:56] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-parsercache on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:57] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:23] PROBLEM - mwtask161 SSH on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:30] PROBLEM - mwtask181 Check unit status of mediawiki_job_generate-sitemaps on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:32] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:35] PROBLEM - mwtask161 PowerDNS Recursor on mwtask161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:29:41] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-checkuser on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:44] PROBLEM - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:44] PROBLEM - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:44] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-abusefilter on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:44] PROBLEM - mwtask181 conntrack_table_size on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:46] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 6.01, 12.68, 9.45 [06:30:43] RECOVERY - mwtask161 ferm_active on mwtask161 is OK: OK ferm input default policy is set [06:31:48] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-special-pages [06:31:48] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:31:49] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:32:14] RECOVERY - mwtask181 Check unit status of mediawiki_job_generate-sitemaps on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps [06:32:14] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-checkuser on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser [06:32:14] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table [06:32:32] PROBLEM - mwtask151 conntrack_table_size on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:37] RECOVERY - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia [06:32:37] RECOVERY - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is OK: OK: Status of the systemd unit update-static-tech-docs [06:32:40] PROBLEM - mwtask151 php-fpm on mwtask151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:33:58] PROBLEM - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:34:02] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-loginnotify on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:34:31] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics [06:34:32] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-expired-blocks on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks [06:34:32] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-parsercache on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache [06:34:36] RECOVERY - mwtask171 conntrack_table_size on mwtask171 is OK: OK: nf_conntrack is 0 % full [06:34:40] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:34:50] RECOVERY - mwtask151 php-fpm on mwtask151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:34:56] RECOVERY - mwtask171 php-fpm on mwtask171 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:35:16] PROBLEM - mwtask161 php-fpm on mwtask161 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:35:47] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:36:03] RECOVERY - mwtask171 ferm_active on mwtask171 is OK: OK ferm input default policy is set [06:36:09] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:36:48] PROBLEM - mwtask151 ferm_active on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:37:36] PROBLEM - mwtask161 ferm_active on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:38:39] PROBLEM - mwtask181 php-fpm on mwtask181 is UNKNOWN: NRPE: Unable to read output [06:39:04] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:10] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:39:25] PROBLEM - mwtask171 php-fpm on mwtask171 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:39:33] PROBLEM - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:39:36] PROBLEM - mwtask151 php-fpm on mwtask151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:39:37] PROBLEM - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:40:08] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:46] RECOVERY - mwtask171 PowerDNS Recursor on mwtask171 is OK: DNS OK: 1.272 second response time. mwtask171.fsslc.wtnet returns 10.0.17.144 [06:41:00] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:41:12] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [06:41:22] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-abusefilter on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter [06:41:23] RECOVERY - mwtask171 php-fpm on mwtask171 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:41:30] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-parsercache on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:42:14] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw172 [06:42:29] RECOVERY - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia [06:42:30] RECOVERY - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis [06:42:52] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 18.91, 22.25, 13.01 [06:43:09] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.056 second response time [06:43:51] RECOVERY - mwtask171 mathoid on mwtask171 is OK: TCP OK - 0.000 second response time on 10.0.17.144 port 10044 [06:44:08] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:44:11] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [06:44:51] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 5.46, 16.09, 11.88 [06:45:02] RECOVERY - mwtask151 ferm_active on mwtask151 is OK: OK ferm input default policy is set [06:46:49] RECOVERY - mwtask151 conntrack_table_size on mwtask151 is OK: OK: nf_conntrack is 0 % full [06:46:50] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.258 second response time [06:47:31] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet last ran 1 hour ago [06:47:32] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:48:22] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:48:31] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-loginnotify on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify [06:48:31] RECOVERY - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is OK: OK: Status of the systemd unit update-static-tech-docs [06:48:37] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-checkuser on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:21] PROBLEM - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:23] PROBLEM - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:58] RECOVERY - mwtask161 php-fpm on mwtask161 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [06:50:46] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [06:50:51] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table [06:50:52] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-statistics on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics [06:51:10] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:51:25] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-checkuser on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser [06:51:41] PROBLEM - mwtask171 mathoid on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 10044: Connection refused [06:51:56] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:07] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-parsercache on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache [06:53:45] PROBLEM - mwtask151 conntrack_table_size on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:54:07] PROBLEM - mwtask151 ferm_active on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:54:31] PROBLEM - mwtask161 php-fpm on mwtask161 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:55:07] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:08] RECOVERY - mwtask161 ferm_active on mwtask161 is OK: OK ferm input default policy is set [06:55:17] RECOVERY - mwtask181 Check unit status of mediawiki_job_backup-all-wikis-ia on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia [06:55:46] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:56:21] PROBLEM - mwtask181 Check unit status of mediawiki_job_purge-abusefilter on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:57:15] RECOVERY - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis [06:57:24] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 7.806 second response time [06:57:38] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:57:40] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 27.18, 18.49, 11.87 [06:57:42] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:45] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 16.37, 11.33 [06:57:45] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:57:45] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:05] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:58:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 8 backends are down. mw151 mw152 mw161 mw162 mw182 mw192 mw201 mw202 [06:58:15] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 8 backends are down. mw151 mw152 mw161 mw162 mw182 mw192 mw201 mw202 [06:58:25] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet last ran 1 hour ago [06:58:29] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 42.12, 21.11, 11.76 [06:58:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 8 backends are down. mw151 mw152 mw162 mw182 mw192 mw193 mw201 mw202 [06:58:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw162 mw182 mw193 mw201 [06:58:58] PROBLEM - mw201 SSH on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:00] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 26.60, 18.53, 11.25 [06:59:10] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:59:37] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 9.55, 14.73, 11.28 [06:59:42] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.32, 20.05, 13.43 [06:59:45] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [06:59:50] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.633 second response time [07:00:00] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.065 second response time [07:00:01] PROBLEM - mw201 APT on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:00:58] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 10.54, 15.13, 10.85 [07:01:22] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:01:40] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 9.39, 15.92, 12.68 [07:01:40] PROBLEM - mwtask171 PowerDNS Recursor on mwtask171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:01:46] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.430 second response time [07:01:48] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 4.234 second response time [07:01:49] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.776 second response time [07:01:50] PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:01:59] RECOVERY - mw201 APT on mw201 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [07:02:03] PROBLEM - mwtask161 ferm_active on mwtask161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:02:21] RECOVERY - mwtask181 Check unit status of mediawiki_job_purge-abusefilter on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter [07:02:22] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 28.24, 24.24, 15.08 [07:02:27] PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 12.69, 11.93, 23.58 [07:02:58] PROBLEM - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:02:59] RECOVERY - mw201 SSH on mw201 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:03:12] PROBLEM - mwtask181 Check unit status of mediawiki_job_refreshlinks on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:03:25] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 6.559 second response time [07:03:36] RECOVERY - mwtask171 PowerDNS Recursor on mwtask171 is OK: DNS OK: 0.983 seconds response time. mwtask171.fsslc.wtnet returns 10.0.17.144 [07:03:57] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet last ran 1 hour ago [07:04:07] PROBLEM - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:04:20] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 12.44, 19.72, 14.54 [07:04:30] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 12.10, 19.59, 14.58 [07:05:34] RECOVERY - mwtask171 mathoid on mwtask171 is OK: TCP OK - 0.000 second response time on 10.0.17.144 port 10044 [07:05:36] RECOVERY - mwtask181 Check unit status of mediawiki_job_refreshlinks on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks [07:05:36] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-wikibase-sites-table on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table [07:05:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:06:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [07:06:47] RECOVERY - mwtask181 Check unit status of mediawiki_job_manage-inactive-wikis on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis [07:06:54] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:08:23] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 7.87, 9.94, 19.01 [07:08:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [07:08:57] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [07:09:16] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:26] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:37] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:49] RECOVERY - mwtask161 php-fpm on mwtask161 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [07:10:23] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:30] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:11:30] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:11:31] PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:11:31] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:11:31] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:11:32] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:43] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:11:47] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:47] RECOVERY - mwtask151 ferm_active on mwtask151 is OK: OK ferm input default policy is set [07:11:53] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:11:54] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:11:57] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:11:58] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:58] PROBLEM - mwtask171 PowerDNS Recursor on mwtask171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:12:00] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:12:03] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [07:12:07] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:12:08] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:12:08] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:08] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:12] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:13] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [07:12:15] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:19] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:19] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:12:21] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:24] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.620 seconds response time. mwtask181.fsslc.wtnet returns 10.0.18.106 [07:12:27] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [07:12:27] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:27] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:27] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:12:28] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [07:12:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [07:12:38] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:12:52] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [07:12:53] RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [07:12:53] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [07:12:54] seems fine to me 💔 [07:12:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw192 mw201 mw202 mw203 [07:13:02] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw192 mw201 mw202 mw203 [07:13:08] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [07:13:10] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [07:13:25] RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.064 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123 [07:13:28] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.051 second response time [07:13:29] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.063 second response time [07:13:31] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 3.564 second response time [07:13:31] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.594 second response time [07:13:31] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.291 second response time [07:13:32] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.527 second response time [07:13:34] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.379 second response time [07:13:34] RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [07:13:41] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.177 second response time [07:13:43] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.208 second response time [07:13:43] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:13:43] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 0.047 second response time [07:13:50] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [07:13:52] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [07:13:53] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.063 second response time [07:13:55] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.000 second response time on 10.0.18.106 port 10044 [07:13:57] RECOVERY - mwtask171 PowerDNS Recursor on mwtask171 is OK: DNS OK: 2.813 seconds response time. mwtask171.fsslc.wtnet returns 10.0.17.144 [07:13:57] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [07:13:59] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.062 second response time [07:14:00] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.057 second response time [07:14:02] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.076 second response time [07:14:04] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.183 second response time [07:14:05] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [07:14:06] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.054 second response time [07:14:07] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.068 second response time [07:14:09] RECOVERY - mwtask181 conntrack_table_size on mwtask181 is OK: OK: nf_conntrack is 1 % full [07:14:10] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:14:10] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [07:14:15] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.156 second response time [07:14:15] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.470 second response time [07:14:18] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.766 second response time [07:14:20] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.188 second response time [07:14:21] RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [07:14:22] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [07:14:26] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.197 second response time [07:14:26] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.186 second response time [07:14:27] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.054 second response time [07:14:30] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.85, 1.98, 0.72 [07:14:36] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.062 second response time [07:14:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [07:14:49] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.045 second response time [07:14:53] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 9% [07:14:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [07:14:59] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [07:15:04] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.181 second response time [07:15:04] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet last ran 1 hour ago [07:15:57] !log [agent@mwtask181] starting deploy of {'config': True} to all [07:16:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:04] !log [agent@mwtask181] starting deploy of {'config': True, 'force': True} to all [07:16:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:49] !log [agent@mwtask181] starting deploy of {'config': True, 'force': True} to all [07:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:53] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.071 second response time [07:17:04] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:18:31] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:18:48] PROBLEM - mwtask151 ferm_active on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:19:04] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.092 second response time [07:19:04] PROBLEM - mwtask161 php-fpm on mwtask161 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:19:10] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.097 second response time [07:19:19] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.093 second response time [07:19:22] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.087 second response time [07:19:25] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.094 second response time [07:19:28] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.093 second response time [07:19:31] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.083 second response time [07:19:31] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.098 second response time [07:19:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758781160000&orgId=1&to=1758784793356 [07:20:01] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:03] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:03] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:15] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:18] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:18] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.014 second response time [07:20:19] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:20] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [07:20:24] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [07:20:27] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:20:49] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [07:22:10] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.953 second response time [07:22:11] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.776 second response time [07:22:14] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:22:16] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.087 second response time [07:22:16] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.092 second response time [07:22:19] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.019 second response time [07:22:21] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.979 second response time [07:22:24] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.954 second response time [07:22:34] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:22:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [07:22:45] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw193 [07:22:50] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.370 second response time [07:22:53] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 80% [07:22:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 2 backends are down. mw173 mw193 [07:23:09] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.688 second response time [07:23:12] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:23:13] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.428 second response time [07:23:18] RECOVERY - mwtask161 php-fpm on mwtask161 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [07:23:19] RECOVERY - mwtask161 PowerDNS Recursor on mwtask161 is OK: DNS OK: 0.896 seconds response time. mwtask161.fsslc.wtnet returns 10.0.16.157 [07:23:22] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.398 second response time [07:23:22] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:23:24] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:23:33] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.289 second response time [07:23:33] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.553 second response time [07:24:06] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 9 backends are down. mw172 mw182 mw163 mw183 mw192 mw193 mw201 mw202 mw203 [07:24:08] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:24:09] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.345 second response time [07:24:12] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 4.017 second response time [07:24:34] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.050 second response time [07:24:41] PROBLEM - mwtask171 PowerDNS Recursor on mwtask171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:24:53] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 55% [07:24:55] RECOVERY - mwtask161 mathoid on mwtask161 is OK: TCP OK - 0.000 second response time on 10.0.16.157 port 10044 [07:25:07] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.063 second response time [07:25:20] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.057 second response time [07:25:21] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.066 second response time [07:25:25] RECOVERY - mwtask161 ferm_active on mwtask161 is OK: OK ferm input default policy is set [07:25:29] RECOVERY - mwtask161 APT on mwtask161 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [07:26:02] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.052 second response time [07:26:05] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [07:26:10] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.089 second response time [07:26:10] RECOVERY - mwtask161 SSH on mwtask161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:26:18] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.096 second response time [07:26:19] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.085 second response time [07:26:28] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.086 second response time [07:26:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [07:26:39] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [07:26:44] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.069 second response time [07:26:49] RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [07:26:49] PROBLEM - db171 MariaDB on db171 is CRITICAL: Can't connect to server on 'db171.fsslc.wtnet' (115) [07:26:53] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [07:26:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [07:27:03] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.088 second response time [07:27:05] RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.064 second response time [07:27:07] RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [07:27:11] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.096 second response time [07:27:12] RECOVERY - mwtask161 conntrack_table_size on mwtask161 is OK: OK: nf_conntrack is 0 % full [07:27:15] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.091 second response time [07:27:20] RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [07:27:25] RECOVERY - mwtask151 php-fpm on mwtask151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [07:27:26] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:27:26] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [07:27:26] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.fsslc.wtn...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.085 second response time [07:27:30] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [07:27:31] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.084 second response time [07:27:34] PROBLEM - mwtask171 mathoid on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 10044: Connection refused [07:27:38] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [07:27:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [07:27:53] PROBLEM - mwtask171 SSH on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:00] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.095 second response time [07:28:07] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:28:08] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.098 second response time [07:28:18] RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 0.047 seconds response time. mwtask151.fsslc.wtnet returns 10.0.15.150 [07:28:21] RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 7.148 second response time [07:28:24] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.093 second response time [07:28:27] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.086 second response time [07:28:30] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [07:28:33] RECOVERY - mwtask151 conntrack_table_size on mwtask151 is OK: OK: nf_conntrack is 0 % full [07:28:42] RECOVERY - mwtask171 PowerDNS Recursor on mwtask171 is OK: DNS OK: 0.045 seconds response time. mwtask171.fsslc.wtnet returns 10.0.17.144 [07:28:46] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 10 Threads: 25 Questions: 8972 Slow queries: 0 Opens: 89 Open tables: 83 Queries per second avg: 897.200 [07:29:03] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 0.54, 0.24, 0.09 [07:29:18] RECOVERY - mwtask151 mathoid on mwtask151 is OK: TCP OK - 0.000 second response time on 10.0.15.150 port 10044 [07:29:20] RECOVERY - mwtask161 Current Load on mwtask161 is OK: LOAD OK - total load average: 0.93, 0.31, 0.11 [07:29:22] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [07:29:26] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 4.8%Current connections: 48 [07:29:27] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.189 second response time [07:29:30] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.176 second response time [07:29:32] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [07:29:34] RECOVERY - mwtask171 mathoid on mwtask171 is OK: TCP OK - 0.000 second response time on 10.0.17.144 port 10044 [07:29:39] RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.001 second response time [07:29:48] RECOVERY - mwtask171 SSH on mwtask171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [07:29:49] RECOVERY - mwtask151 ferm_active on mwtask151 is OK: OK ferm input default policy is set [07:29:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758781760000&orgId=1&to=1758785360000 [07:29:58] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.174 second response time [07:30:01] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.158 second response time [07:30:08] RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [07:30:08] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.355 second response time [07:30:17] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.650 second response time [07:30:20] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 135 packages available for upgrade (0 critical updates). [07:30:33] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.988 second response time [07:30:34] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.899 second response time [07:30:39] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.131 second response time [07:30:53] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 18% [07:31:02] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [07:31:11] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.145 second response time [07:31:12] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.063 second response time [07:31:14] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.347 second response time [07:31:22] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.543 second response time [07:31:30] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.604 second response time [07:31:57] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.516 second response time [07:32:08] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.000 second response time [07:32:09] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:28] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 11 backends are down. mw151 mw161 mw171 mw181 mw173 mw191 mw192 mw193 mw201 mw202 mw203 [07:32:34] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4212 bytes in 0.064 second response time [07:32:35] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.006 second response time [07:32:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 11 backends are down. mw161 mw171 mw181 mw182 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [07:32:52] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:32:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 12 backends are down. mw152 mw161 mw172 mw181 mw182 mw173 mw191 mw192 mw193 mw201 mw202 mw203 [07:33:32] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.947 second response time [07:34:02] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 4 backends are down. mw152 mw162 mw192 mw193 [07:34:13] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 20.71, 20.44, 12.42 [07:34:25] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.181 second response time [07:34:32] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.179 second response time [07:34:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [07:34:44] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.20, 23.94, 14.95 [07:34:45] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.175 second response time [07:34:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [07:35:19] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.201 second response time [07:35:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [07:36:01] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [07:36:11] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 9.40, 17.21, 12.26 [07:36:22] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [07:36:42] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 5.44, 17.26, 13.61 [07:36:56] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.046 second response time [07:45:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758782720000&orgId=1&to=1758786353366 [07:55:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758783260000&orgId=1&to=1758786860000 [07:58:36] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:58:51] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [07:59:04] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 12 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw182 mw163 mw192 mw193 mw201 mw202 [07:59:23] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 9.844 second response time [07:59:24] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 29.08, 18.50, 11.91 [07:59:33] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 13 backends are down. mw151 mw152 mw162 mw171 mw172 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 [07:59:52] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 14 backends are down. mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw192 mw193 mw201 mw203 [08:00:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 9 backends are down. mw162 mw181 mw182 mw153 mw163 mw173 mw191 mw201 mw203 [08:00:13] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 22.18, 13.66 [08:00:22] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 29.46, 28.00, 17.22 [08:00:30] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.067 second response time [08:00:33] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:48] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.065 second response time [08:01:21] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.910 second response time [08:01:22] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 18.29, 18.15, 12.62 [08:01:37] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 17.62, 10.81 [08:01:41] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:22] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.51, 19.64, 12.39 [08:02:31] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.644 second response time [08:03:25] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [08:03:34] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 12.97, 17.24, 11.61 [08:03:36] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.189 second response time [08:03:51] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [08:04:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [08:04:10] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 7.99, 18.94, 14.61 [08:04:20] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 6.93, 14.62, 11.39 [08:04:22] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 6.16, 17.98, 15.72 [08:04:48] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [08:15:57] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:09] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.016 second response time [08:16:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw181 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [08:16:20] PROBLEM - mw201 SSH on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:31] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 45.17, 23.18, 15.47 [08:16:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw171 mw172 mw182 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [08:16:58] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 8 backends are down. mw171 mw172 mw182 mw173 mw191 mw192 mw193 mw201 [08:17:05] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 26.75, 22.77, 15.43 [08:17:27] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 21.13, 22.96, 15.18 [08:17:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [08:17:56] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.159 second response time [08:18:05] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [08:18:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [08:18:16] RECOVERY - mw201 SSH on mw201 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:18:29] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 9.09, 16.85, 14.10 [08:18:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [08:18:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [08:19:01] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 6.90, 16.87, 14.15 [08:19:22] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 6.01, 16.69, 13.79 [08:23:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.52, 4.72, 10.83 [08:25:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.80, 3.43, 9.61 [08:25:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [08:29:03] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [08:29:04] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:29:07] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:29:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.24, 9.21, 10.79 [08:29:17] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:29:19] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [08:29:24] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.57, 15.35, 10.72 [08:29:24] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 34.46, 20.34, 14.68 [08:29:27] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 17.47, 11.75 [08:29:31] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:29:32] PROBLEM - mw191 SSH on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:33] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:33] PROBLEM - mw191 php-fpm on mw191 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:29:33] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 15.07, 10.14 [08:29:41] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw192 mw193 mw201 mw203 [08:29:44] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 12 backends are down. mw151 mw152 mw171 mw172 mw181 mw173 mw191 mw192 mw193 mw201 mw202 mw203 [08:30:18] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 36.01, 22.21, 12.88 [08:30:24] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:30:26] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.04, 16.95, 10.61 [08:30:30] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 38.65, 25.90, 17.16 [08:30:33] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 2.971 second response time [08:30:36] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.10, 17.32, 11.24 [08:30:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw191 mw192 mw193 mw201 mw202 mw203 [08:30:37] PROBLEM - mw192 SSH on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:39] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:30:43] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:44] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 46.50, 30.44, 18.18 [08:30:45] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:45] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [08:30:47] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 42.68, 21.89, 14.01 [08:30:50] PROBLEM - mw193 SSH on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:57] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 15 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw191 mw192 mw193 mw201 mw202 mw203 [08:30:58] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:58] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.441 second response time [08:30:59] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:31:03] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:31:03] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:31:03] PROBLEM - mw191 PowerDNS Recursor on mw191 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:31:07] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:31:08] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 3.501 second response time [08:31:20] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 48.72, 28.38, 18.23 [08:31:21] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.74, 10.70, 11.11 [08:31:25] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 33.84, 21.35, 13.43 [08:31:32] RECOVERY - mw191 SSH on mw191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:31:32] RECOVERY - mw191 php-fpm on mw191 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [08:31:38] PROBLEM - mw151 PowerDNS Recursor on mw151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:31:42] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [08:31:50] PROBLEM - mw151 APT on mw151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:14] PROBLEM - mw171 APT on mw171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:15] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:18] PROBLEM - mw151 SSH on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:21] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.287 second response time [08:32:28] PROBLEM - mw202 SSH on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:30] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.880 second response time [08:32:31] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 34.79, 35.54, 21.70 [08:32:31] RECOVERY - mw192 SSH on mw192 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:32:34] PROBLEM - mw171 SSH on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:36] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 10.66, 14.83, 11.09 [08:32:37] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [08:32:38] PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:32:40] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.129 second response time [08:32:41] PROBLEM - mw152 APT on mw152 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:41] PROBLEM - mw172 SSH on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:41] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:32:44] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 40.05, 29.45, 16.68 [08:32:46] RECOVERY - mw193 SSH on mw193 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:32:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 36.64, 26.29, 15.12 [08:32:54] PROBLEM - mw171 PowerDNS Recursor on mw171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:32:55] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.372 second response time [08:32:58] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time [08:32:59] RECOVERY - mw191 PowerDNS Recursor on mw191 is OK: DNS OK: 0.325 seconds response time. mw191.fsslc.wtnet returns 10.0.19.160 [08:33:00] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 0.045 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [08:33:02] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [08:33:10] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.367 second response time [08:33:11] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:21] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.15, 18.77, 13.43 [08:33:22] PROBLEM - mw201 APT on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:22] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.219 second response time [08:33:26] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:28] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.58, 19.02, 13.82 [08:33:33] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [08:33:34] RECOVERY - mw151 PowerDNS Recursor on mw151 is OK: DNS OK: 0.338 seconds response time. mw151.fsslc.wtnet returns 10.0.15.114 [08:33:34] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.97, 17.17, 12.36 [08:33:38] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [08:33:41] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 24% [08:33:51] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 9.789 second response time [08:33:53] RECOVERY - mw151 APT on mw151 is OK: APT OK: 133 packages available for upgrade (0 critical updates). [08:34:16] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.32, 20.74, 14.55 [08:34:27] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 17.83, 22.31, 17.88 [08:34:28] RECOVERY - mw202 SSH on mw202 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:34:42] RECOVERY - mw152 APT on mw152 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:34:46] PROBLEM - mw172 APT on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758785660000&orgId=1&to=1758789293357 [08:34:53] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.023 second response time [08:35:24] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:04] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [08:36:05] RECOVERY - mw201 APT on mw201 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:36:08] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [08:36:17] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 32.97, 23.88, 16.35 [08:36:20] RECOVERY - mw151 SSH on mw151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:36:25] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 36.84, 27.74, 20.36 [08:36:25] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.854 second response time [08:36:33] RECOVERY - mw171 SSH on mw171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:36:37] RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.055 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123 [08:36:44] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.234 second response time [08:36:44] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.46, 20.04, 15.06 [08:36:45] RECOVERY - mw172 APT on mw172 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:36:48] RECOVERY - mw172 SSH on mw172 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:37:03] RECOVERY - mw171 PowerDNS Recursor on mw171 is OK: DNS OK: 4.986 seconds response time. mw171.fsslc.wtnet returns 10.0.17.122 [08:37:15] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 14.76, 21.77, 18.96 [08:37:17] RECOVERY - mw171 APT on mw171 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:37:22] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [08:37:28] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.20, 21.16, 15.97 [08:37:34] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.052 second response time [08:37:42] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.256 second response time [08:38:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [08:38:16] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 7.69, 17.31, 14.86 [08:38:23] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 7.54, 21.00, 18.42 [08:38:24] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 8.40, 19.80, 18.34 [08:38:27] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 6.43, 18.54, 18.60 [08:38:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [08:38:51] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 7.21, 23.41, 19.42 [08:38:53] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [08:38:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [08:39:09] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 6.66, 22.03, 18.34 [08:39:12] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 6.75, 16.45, 17.34 [08:39:22] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 7.61, 23.68, 21.79 [08:39:25] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 5.59, 15.73, 14.60 [08:39:37] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [08:40:16] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 5.29, 16.00, 16.89 [08:40:50] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 4.37, 17.06, 17.58 [08:40:51] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 4.94, 17.77, 18.67 [08:41:08] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 3.62, 15.78, 16.49 [08:41:18] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 4.01, 17.08, 19.61 [08:45:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.57, 5.85, 10.77 [08:47:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.05, 4.26, 9.58 [08:51:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.53, 9.83, 10.43 [08:51:32] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 17 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [08:51:34] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:47] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:49] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:51:51] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:53] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:51:53] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:51:55] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:51:59] PROBLEM - mw203 SSH on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 mw163 mw191 mw192 mw193 mw201 mw202 mw203 [08:52:17] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:52:21] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:52:24] PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:52:24] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:52:24] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:25] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [08:52:25] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.036 second response time [08:52:25] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:30] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 6.899 second response time [08:52:33] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:52:35] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [08:52:35] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 2.265 second response time [08:52:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 18.90, 13.91 [08:52:36] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 12 backends are down. mw151 mw162 mw171 mw172 mw153 mw173 mw191 mw192 mw193 mw201 mw202 mw203 [08:52:40] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 47.27, 27.49, 18.67 [08:52:41] PROBLEM - mw191 Current Load on mw191 is CRITICAL: LOAD CRITICAL - total load average: 33.40, 23.73, 18.39 [08:52:50] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:51] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:52:55] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [08:52:55] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 37.78, 26.29, 18.55 [08:52:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [08:52:56] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 26.37, 21.34, 16.49 [08:53:00] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 18.23, 14.10 [08:53:01] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:53:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.64, 8.59, 9.90 [08:53:18] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 39.92, 28.53, 21.07 [08:53:22] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 43.70, 26.06, 18.63 [08:53:23] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:25] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 49% [08:53:45] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 45.52, 27.85, 19.01 [08:54:02] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:14] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.368 second response time [08:54:19] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 2.639 second response time [08:54:19] PROBLEM - mw162 APT on mw162 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:22] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 1.097 second response time [08:54:26] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:28] PROBLEM - mw171 APT on mw171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:33] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.11, 18.53, 14.42 [08:54:35] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.357 second response time [08:54:37] PROBLEM - mw201 APT on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:38] PROBLEM - mw191 APT on mw191 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:45] PROBLEM - mw202 APT on mw202 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:50] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:52] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:54:55] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 49.14, 27.41, 18.64 [08:54:57] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.44, 17.42, 14.24 [08:54:58] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 5.697 second response time [08:55:04] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:55:31] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.294 second response time [08:55:37] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.003 second response time [08:55:37] PROBLEM - mw172 APT on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:47] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.208 second response time [08:55:55] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [08:55:57] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [08:56:01] RECOVERY - mw203 SSH on mw203 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [08:56:25] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [08:56:26] RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 4.302 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123 [08:56:31] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.126 second response time [08:56:31] RECOVERY - mw171 APT on mw171 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:56:31] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.811 second response time [08:56:32] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [08:56:34] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.976 second response time [08:56:35] RECOVERY - mw201 APT on mw201 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:56:42] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.586 second response time [08:56:46] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.226 second response time [08:56:48] RECOVERY - mw202 APT on mw202 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:56:49] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [08:57:05] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.080 second response time [08:57:05] RECOVERY - mw162 APT on mw162 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:57:08] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.889 second response time [08:57:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.22, 14.25, 11.95 [08:57:23] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 24% [08:57:27] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.81, 22.11, 15.51 [08:57:37] RECOVERY - mw172 APT on mw172 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [08:57:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [08:58:06] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/mediawiki-firejail-espeak] [08:58:17] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 8.826 second response time [08:58:17] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.391 second response time [08:58:37] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.935 second response time [08:58:43] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.066 second response time [08:58:49] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.702 second response time [08:58:55] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 14.65, 22.67, 19.85 [08:58:56] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 0.054 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [08:58:58] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 14.97, 22.74, 19.23 [08:59:22] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.41, 23.87, 20.49 [08:59:37] RECOVERY - mw191 APT on mw191 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:00:53] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.021 second response time [09:00:54] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 39.07, 28.82, 21.81 [09:01:01] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 12.61, 21.37, 21.48 [09:01:21] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 7.67, 18.08, 18.79 [09:01:41] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 9.74, 18.29, 15.67 [09:01:45] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.02, 18.80, 19.23 [09:01:46] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.060 second response time [09:01:57] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.215 second response time [09:02:37] PROBLEM - mw192 SSH on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:02:39] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:02:43] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 5.34, 16.41, 18.23 [09:02:48] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 3.382 second response time [09:02:50] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 9.11, 21.01, 19.79 [09:02:56] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 5.71, 18.50, 19.05 [09:02:57] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 5.87, 15.99, 19.50 [09:03:20] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 6.42, 19.66, 21.38 [09:04:31] RECOVERY - mw192 SSH on mw192 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:04:46] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 9.975 second response time [09:04:46] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 9.11, 17.31, 18.59 [09:05:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:06:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [09:06:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [09:06:45] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.178 second response time [09:06:54] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 6.14, 21.99, 23.23 [09:06:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [09:07:16] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 5.67, 15.77, 19.81 [09:07:27] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [09:10:51] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 3.72, 12.00, 18.84 [09:18:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [09:18:19] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.92, 18.09, 13.54 [09:18:23] PROBLEM - mw193 SSH on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:32] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:34] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:18:35] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:18:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 16 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw192 mw193 mw201 mw202 mw203 [09:18:40] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:18:41] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 7.463 second response time [09:18:45] PROBLEM - mw203 Current Load on mw203 is CRITICAL: LOAD CRITICAL - total load average: 60.33, 29.74, 19.33 [09:18:46] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:18:46] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:18:48] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:48] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [09:18:49] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:18:51] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:18:54] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:18:55] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [09:18:59] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 39.91, 22.18, 19.27 [09:19:01] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:19:03] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:19:04] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:08] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [09:19:14] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:19:14] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:19:23] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 51.80, 28.47, 19.40 [09:19:24] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:24] PROBLEM - mw182 PowerDNS Recursor on mw182 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:19:25] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 16 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw163 mw173 mw191 mw192 mw193 mw201 mw202 mw203 [09:19:30] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:19:31] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.63, 14.49, 12.00 [09:19:35] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 47.72, 24.53, 17.00 [09:19:39] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:41] PROBLEM - mw203 PowerDNS Recursor on mw203 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:19:43] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:48] PROBLEM - mw192 PowerDNS Recursor on mw192 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:19:50] PROBLEM - mw193 Current Load on mw193 is CRITICAL: LOAD CRITICAL - total load average: 46.39, 30.52, 20.84 [09:19:51] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:19:52] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 47.63, 32.89, 19.70 [09:19:54] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 45.11, 25.16, 17.56 [09:19:57] PROBLEM - mw202 PowerDNS Recursor on mw202 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:20:12] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:19] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.81, 17.60, 13.85 [09:20:22] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:20:28] PROBLEM - mw191 Current Load on mw191 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:34] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.222 second response time [09:20:35] PROBLEM - mw193 conntrack_table_size on mw193 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:37] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:37] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 48.00, 29.39, 20.75 [09:20:40] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 6.124 second response time [09:20:51] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 3.493 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [09:21:00] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.119 second response time [09:21:07] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 57% [09:21:08] PROBLEM - mw201 APT on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:12] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.188 second response time [09:21:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.56, 6.71, 11.55 [09:21:16] PROBLEM - mw162 SSH on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:17] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:18] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:22] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:21:25] PROBLEM - mw182 APT on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:27] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:29] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.911 second response time [09:21:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 36.28, 23.50, 15.87 [09:21:29] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.072 second response time [09:21:30] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 20.40, 14.59 [09:21:34] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.65, 21.92, 16.98 [09:21:35] PROBLEM - mw192 APT on mw192 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:37] PROBLEM - mw193 APT on mw193 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:43] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:44] PROBLEM - mw192 SSH on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:49] PROBLEM - mw203 APT on mw203 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:52] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:53] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.16, 22.40, 17.51 [09:21:57] RECOVERY - mw202 PowerDNS Recursor on mw202 is OK: DNS OK: 0.089 seconds response time. mw202.fsslc.wtnet returns 10.0.20.163 [09:22:16] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.70, 20.20, 15.39 [09:22:17] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.316 second response time [09:22:53] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.687 second response time [09:22:55] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 1.358 second response time [09:22:56] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.115 second response time [09:22:57] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.628 second response time [09:23:05] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 35% [09:23:07] RECOVERY - mw201 APT on mw201 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:23:14] RECOVERY - mw162 SSH on mw162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:23:15] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 16.81, 23.30, 19.57 [09:23:20] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [09:23:25] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 6.696 second response time [09:23:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 16.90, 20.70, 15.74 [09:23:28] RECOVERY - mw182 APT on mw182 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:23:33] RECOVERY - mw182 PowerDNS Recursor on mw182 is OK: DNS OK: 0.062 seconds response time. mw182.fsslc.wtnet returns 10.0.18.105 [09:23:33] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.44, 20.06, 16.86 [09:23:55] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [09:24:37] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 8.042 second response time [09:24:57] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.049 second response time [09:25:06] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 9.875 second response time [09:25:12] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:25:15] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:25:22] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.936 second response time [09:25:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.74, 20.68, 16.26 [09:25:41] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.119 second response time [09:25:43] RECOVERY - mw203 PowerDNS Recursor on mw203 is OK: DNS OK: 0.074 seconds response time. mw203.fsslc.wtnet returns 10.0.20.165 [09:25:46] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:25:52] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.25, 19.23, 17.45 [09:25:52] PROBLEM - mw193 php-fpm on mw193 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:25:56] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:57] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:25:58] PROBLEM - mw192 php-fpm on mw192 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:25:58] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.52, 19.03, 15.16 [09:26:02] PROBLEM - mw191 php-fpm on mw191 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:26:11] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [09:26:17] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.226 second response time [09:26:19] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.33, 22.84, 17.73 [09:27:02] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 4.324 second response time [09:27:10] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.87, 19.89, 18.98 [09:27:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.99, 13.44, 12.82 [09:27:15] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 6.065 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [09:27:18] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 39 minutes ago with 0 failures [09:27:18] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [09:27:18] RECOVERY - mw193 APT on mw193 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:27:19] RECOVERY - mw192 APT on mw192 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:27:19] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.020 second response time [09:27:27] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 31.43, 23.95, 17.99 [09:27:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:27:46] RECOVERY - mw193 php-fpm on mw193 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [09:27:47] RECOVERY - mw192 SSH on mw192 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:27:53] RECOVERY - mw192 php-fpm on mw192 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [09:27:57] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.06, 20.70, 16.20 [09:28:01] RECOVERY - mw191 php-fpm on mw191 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [09:28:01] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 7.969 second response time [09:28:03] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 51.16, 29.01, 19.75 [09:28:17] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.60, 24.21, 18.78 [09:28:22] RECOVERY - mw192 PowerDNS Recursor on mw192 is OK: DNS OK: 4.109 seconds response time. mw192.fsslc.wtnet returns 10.0.19.161 [09:28:23] PROBLEM - mw191 PowerDNS Recursor on mw191 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:28:28] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:28:34] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:28:45] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:28:57] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.989 second response time [09:28:58] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:29:00] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.098 second response time [09:29:12] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.217 second response time [09:29:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 14.89, 20.60, 17.49 [09:29:30] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 25.41, 20.29 [09:29:42] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 16.04, 20.82, 16.39 [09:29:47] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 12.55, 21.51, 21.12 [09:29:50] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.21, 22.59, 19.55 [09:29:56] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.71, 20.10, 16.44 [09:29:59] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 47% [09:30:16] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.67, 21.57, 18.42 [09:30:20] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 16.80, 23.79, 22.23 [09:30:40] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 9.928 second response time [09:30:55] PROBLEM - mw191 APT on mw191 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:31:00] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:31:27] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 33.17, 26.24, 19.94 [09:31:31] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.51, 23.53, 20.15 [09:31:36] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:31:40] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.52, 19.31, 16.41 [09:31:57] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [09:32:00] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:32:01] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.91, 23.87, 19.66 [09:32:16] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 38.20, 23.97, 18.17 [09:32:18] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.23, 22.82, 19.14 [09:32:21] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 28.89, 25.16, 22.90 [09:32:25] PROBLEM - mw191 SSH on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:31] RECOVERY - mw203 APT on mw203 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:32:38] RECOVERY - mw191 PowerDNS Recursor on mw191 is OK: DNS OK: 2.983 seconds response time. mw191.fsslc.wtnet returns 10.0.19.160 [09:32:41] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 8.139 second response time [09:32:43] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:32:51] PROBLEM - mw192 PowerDNS Recursor on mw192 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:32:52] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:54] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 20.74, 18.68, 13.63 [09:32:56] RECOVERY - mw191 APT on mw191 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:32:58] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [09:33:23] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 8.518 second response time [09:33:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.70, 23.80, 19.81 [09:33:30] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 10.52, 18.94, 18.90 [09:33:33] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:33:33] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 0.352 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [09:33:41] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.35, 21.25, 20.93 [09:33:44] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:48] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 8.96, 17.27, 18.27 [09:33:56] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 50% [09:34:01] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.390 second response time [09:34:03] RECOVERY - mw193 conntrack_table_size on mw193 is OK: OK: nf_conntrack is 0 % full [09:34:07] PROBLEM - mw192 SSH on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:16] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.41, 19.95, 18.49 [09:34:44] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:34:45] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.254 second response time [09:34:54] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 13.08, 16.37, 13.39 [09:35:13] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.212 second response time [09:35:27] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.29, 20.37, 19.01 [09:35:38] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.00, 19.55, 20.32 [09:35:38] PROBLEM - mw202 PowerDNS Recursor on mw202 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:35:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:35:55] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:35:59] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.59, 19.55, 18.77 [09:36:02] RECOVERY - mw192 SSH on mw192 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:36:05] PROBLEM - mw192 APT on mw192 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:36:10] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:14] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:36:16] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.19, 23.90, 19.51 [09:36:29] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 42.04, 33.53, 24.89 [09:36:46] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 4.981 second response time [09:36:50] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.068 second response time [09:36:57] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.065 second response time [09:37:04] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.905 second response time [09:37:04] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.030 second response time [09:37:13] PROBLEM - mw191 PowerDNS Recursor on mw191 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:37:37] RECOVERY - mw202 PowerDNS Recursor on mw202 is OK: DNS OK: 0.136 seconds response time. mw202.fsslc.wtnet returns 10.0.20.163 [09:37:57] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 2.335 second response time [09:38:09] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:38:13] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:38:15] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.37, 24.32, 20.12 [09:38:18] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 6.791 second response time [09:38:38] RECOVERY - mw191 SSH on mw191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:39:08] RECOVERY - mw191 PowerDNS Recursor on mw191 is OK: DNS OK: 0.336 seconds response time. mw191.fsslc.wtnet returns 10.0.19.160 [09:39:08] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.070 second response time [09:39:20] RECOVERY - mw193 SSH on mw193 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:39:29] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.89, 21.31, 20.94 [09:39:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:39:50] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 20.72, 19.25 [09:39:58] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 46.40, 29.46, 22.87 [09:40:00] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.367 second response time [09:40:08] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.657 second response time [09:40:13] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.09, 20.93, 19.39 [09:40:19] PROBLEM - mw203 SSH on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:27] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.15, 21.90, 20.50 [09:40:41] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.465 second response time [09:40:52] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 7.926 second response time [09:40:59] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.760 second response time [09:41:05] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:41:18] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:21] RECOVERY - mw192 PowerDNS Recursor on mw192 is OK: DNS OK: 0.103 seconds response time. mw192.fsslc.wtnet returns 10.0.19.161 [09:41:21] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 42 minutes ago with 0 failures [09:41:23] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:25] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 10.92, 17.37, 19.54 [09:41:28] RECOVERY - mw192 APT on mw192 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:41:28] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:41:36] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:49] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.87, 23.25, 20.49 [09:42:32] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 44.18, 28.39, 22.91 [09:43:06] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:24] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.804 second response time [09:43:34] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.064 second response time [09:43:41] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.67, 24.21, 19.88 [09:43:48] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.05, 20.37, 19.82 [09:43:49] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [09:44:02] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.564 second response time [09:44:05] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.861 second response time [09:44:08] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.887 second response time [09:44:12] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.24, 20.07, 19.38 [09:44:15] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.092 second response time [09:44:21] RECOVERY - mw203 SSH on mw203 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:44:36] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.065 second response time [09:44:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758789680000&orgId=1&to=1758793280000 [09:44:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [09:45:01] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.166 second response time [09:45:06] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.062 second response time [09:45:16] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [09:45:31] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.060 second response time [09:45:31] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.171 second response time [09:45:40] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 8.56, 18.06, 18.14 [09:45:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [09:45:48] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 18% [09:45:50] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 6.58, 18.21, 20.30 [09:45:52] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.182 second response time [09:46:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [09:46:30] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 6.55, 18.23, 20.21 [09:46:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [09:49:50] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:49:59] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 mw163 mw183 mw191 mw192 mw193 mw203 [09:50:15] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:20] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:50:21] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:50:22] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:31] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 15.39, 19.44, 23.94 [09:50:32] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [09:50:34] PROBLEM - mw182 SSH on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 10 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 mw153 mw191 mw193 [09:50:42] PROBLEM - mw182 php-fpm on mw182 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:50:44] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:50:55] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:50:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 16 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 mw153 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [09:51:00] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [09:51:00] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 44.86, 30.26, 24.29 [09:51:02] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:51:04] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:51:06] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:11] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:12] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [09:51:14] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 4.238 second response time [09:51:14] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [09:51:28] PROBLEM - mw171 SSH on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:31] PROBLEM - mw171 php-fpm on mw171 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:51:33] PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:51:43] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:51:43] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 58% [09:51:45] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:46] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.02, 22.92, 21.72 [09:51:56] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.38, 21.95, 19.22 [09:52:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 31.56, 32.72, 24.77 [09:52:16] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.197 second response time [09:52:18] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.158 second response time [09:52:29] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:36] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.66, 22.39, 18.62 [09:52:38] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:38] PROBLEM - mw172 APT on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:41] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 71.25, 43.85, 28.75 [09:52:48] RECOVERY - mw182 php-fpm on mw182 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [09:52:58] PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:53:01] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 0.079 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [09:53:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 3.48, 7.14, 11.68 [09:53:22] RECOVERY - mw171 SSH on mw171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:53:27] RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.070 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123 [09:53:28] RECOVERY - mw171 php-fpm on mw171 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [09:53:30] PROBLEM - mw203 APT on mw203 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:53:40] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.189 second response time [09:53:42] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.927 second response time [09:53:42] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [09:54:01] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.945 second response time [09:54:07] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time [09:54:23] PROBLEM - mw201 Current Load on mw201 is CRITICAL: LOAD CRITICAL - total load average: 28.11, 22.89, 24.25 [09:54:34] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.79, 20.32, 18.35 [09:54:41] RECOVERY - mw182 SSH on mw182 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [09:54:56] RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.108 seconds response time. mw152.fsslc.wtnet returns 10.0.15.115 [09:55:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 14.77, 13.89 [09:55:40] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.38, 21.08, 20.93 [09:55:41] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 55% [09:55:56] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:56:24] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:31] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.046 second response time [09:56:39] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:57:03] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 7.297 second response time [09:57:07] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.046 second response time [09:57:09] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [09:57:16] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [09:57:29] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.083 second response time [09:57:35] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [09:57:37] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.33, 17.58, 19.66 [09:57:38] RECOVERY - mw172 APT on mw172 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:57:39] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [09:57:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [09:57:54] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.86, 21.32, 19.88 [09:57:59] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:58:07] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [09:58:26] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:58:36] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.85, 25.06, 20.76 [09:58:39] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 3.952 second response time [09:58:39] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.351 second response time [09:58:45] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 6.393 second response time [09:58:46] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:46] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [09:58:56] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.011 second response time [09:59:09] RECOVERY - mw203 APT on mw203 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [09:59:13] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.216 second response time [09:59:17] PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [09:59:18] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.679 second response time [09:59:18] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:59:31] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.051 second response time [09:59:38] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 47% [09:59:54] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.20, 18.45, 18.98 [10:00:04] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.953 second response time [10:00:25] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.095 second response time [10:00:27] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [10:00:49] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.492 second response time [10:00:50] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:51] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.056 second response time [10:01:09] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:01:19] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 6.213 second response time [10:01:27] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.230 second response time [10:01:35] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [10:01:56] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:02:12] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.090 second response time [10:02:14] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.624 second response time [10:02:18] PROBLEM - mw193 SSH on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:29] PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:02:47] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.770 second response time [10:04:15] PROBLEM - mw182 SSH on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:54] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:05:24] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:26] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:05:29] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:05:31] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 20.67, 20.20 [10:05:34] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [10:05:35] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [10:05:41] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 20.66, 19.74, 16.19 [10:05:41] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:05:44] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:45] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:05:55] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 9.915 second response time [10:05:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.29, 21.77, 20.40 [10:06:01] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:06:02] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:06:06] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.81, 21.02, 19.23 [10:06:21] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:21] RECOVERY - mw193 SSH on mw193 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:06:35] RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 0.067 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164 [10:07:03] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [10:07:17] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:23] RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.243 second response time [10:07:32] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.089 second response time [10:07:35] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.058 second response time [10:07:36] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 11.29, 16.96, 15.61 [10:07:41] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.463 second response time [10:07:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.27, 18.83, 19.48 [10:08:00] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.229 second response time [10:08:05] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.64, 18.93, 18.77 [10:08:19] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.389 second response time [10:08:26] RECOVERY - mw182 SSH on mw182 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:08:26] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:08:26] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 10.42, 16.06, 22.70 [10:08:34] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 12.93, 21.57, 21.97 [10:08:35] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:45] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:09:01] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/secupgrade.sh] [10:09:06] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.733 second response time [10:09:13] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.415 second response time [10:09:17] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.255 second response time [10:09:31] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [10:09:40] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:09:43] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:50] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:09:54] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [10:10:24] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.068 second response time [10:10:28] PROBLEM - mw202 Current Load on mw202 is CRITICAL: LOAD CRITICAL - total load average: 45.96, 26.16, 25.49 [10:10:35] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.36, 22.34, 20.14 [10:10:46] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:10:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758791420000&orgId=1&to=1758795053352 [10:11:11] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:27] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:11:27] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:35] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:11:48] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.168 second response time [10:11:51] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.175 second response time [10:11:51] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:06] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [10:12:15] PROBLEM - mw182 APT on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:16] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:35] PROBLEM - mw202 SSH on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:44] PROBLEM - mw202 PowerDNS Recursor on mw202 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:12:45] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.980 second response time [10:12:48] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.063 second response time [10:12:51] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:03] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 52.23, 30.91, 25.06 [10:13:05] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [10:13:11] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 2.985 second response time [10:13:16] PROBLEM - mw201 PowerDNS Recursor on mw201 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:13:22] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:27] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.13, 21.52, 23.96 [10:13:28] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 38% [10:13:30] RECOVERY - mw153 HTTPS on mw153 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [10:13:35] PROBLEM - mw202 APT on mw202 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:13:44] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.635 second response time [10:13:48] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [10:13:48] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 0.064 second response time [10:13:48] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.061 second response time [10:13:52] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.057 second response time [10:13:52] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.151 second response time [10:14:38] RECOVERY - mw202 SSH on mw202 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:14:40] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 15.43, 20.62, 23.81 [10:14:47] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:14:49] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.157 second response time [10:14:57] PROBLEM - mw203 php-fpm on mw203 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:15:17] RECOVERY - mw201 PowerDNS Recursor on mw201 is OK: DNS OK: 0.181 seconds response time. mw201.fsslc.wtnet returns 10.0.20.162 [10:15:17] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:17] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:15:25] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.630 second response time [10:15:27] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.31, 23.69, 24.45 [10:15:39] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:33] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:16:36] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [10:16:58] PROBLEM - mw182 PowerDNS Recursor on mw182 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:16:59] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:17:03] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:05] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:17:09] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.209 second response time [10:17:23] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:17:24] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:25] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 54% [10:17:27] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:17:35] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 4.187 second response time [10:17:35] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 33 minutes ago with 0 failures [10:17:39] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.085 second response time [10:17:39] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.842 second response time [10:17:39] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.255 second response time [10:17:41] RECOVERY - mw182 APT on mw182 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:17:43] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:18:10] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.219 second response time [10:18:23] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time [10:18:27] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.052 second response time [10:18:27] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [10:18:32] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.180 second response time [10:18:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 12.03, 20.01, 20.30 [10:18:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [10:18:37] RECOVERY - mw202 APT on mw202 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:18:45] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.054 second response time [10:18:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [10:18:57] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.045 second response time [10:18:57] RECOVERY - mw182 PowerDNS Recursor on mw182 is OK: DNS OK: 0.327 seconds response time. mw182.fsslc.wtnet returns 10.0.18.105 [10:19:00] RECOVERY - mw203 php-fpm on mw203 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2' [10:19:02] RECOVERY - mw202 PowerDNS Recursor on mw202 is OK: DNS OK: 0.315 seconds response time. mw202.fsslc.wtnet returns 10.0.20.163 [10:19:02] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.181 second response time [10:19:03] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.047 second response time [10:19:10] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.064 second response time [10:19:11] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [10:19:15] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.053 second response time [10:19:20] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.059 second response time [10:19:20] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.168 second response time [10:19:23] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [10:19:23] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 16% [10:19:25] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.182 second response time [10:19:26] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 10.22, 20.85, 23.49 [10:19:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 11.45, 22.04, 23.53 [10:19:39] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.062 second response time [10:20:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [10:20:36] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 9.03, 18.11, 23.48 [10:20:53] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758791960000&orgId=1&to=1758795560000[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758791420000&orgId=1&to=1758795653354 [10:21:01] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.21, 20.95, 23.17 [10:21:27] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.71, 22.52, 23.40 [10:21:29] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 29.17, 21.92, 23.38 [10:22:35] PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 28.28, 23.85, 24.99 [10:22:44] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 33.50, 22.42, 22.64 [10:22:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 13 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 mw163 mw191 mw192 mw193 mw201 mw202 [10:23:10] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:23:11] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 15 backends are down. mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw183 mw191 mw192 mw201 mw202 mw203 [10:23:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.66, 9.88, 11.65 [10:23:23] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:23:29] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:23:30] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:34] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:23:42] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:23:47] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.056 second response time [10:23:48] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:56] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 15 backends are down. mw151 mw161 mw171 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 [10:24:09] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [10:24:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 12 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw183 mw192 mw193 mw201 mw202 mw203 [10:25:06] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.063 second response time [10:25:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 19.00, 11.93, 12.09 [10:25:18] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [10:25:27] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.09, 23.26, 23.85 [10:25:27] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 14.25, 22.21, 23.51 [10:25:35] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.306 second response time [10:25:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [10:26:09] PROBLEM - mw192 APT on mw192 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:16] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:17] PROBLEM - mw203 PowerDNS Recursor on mw203 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:26:37] PROBLEM - mw192 SSH on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:01] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:27:03] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 22.85, 23.51 [10:27:09] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.13, 11.21, 11.89 [10:27:28] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.065 second response time [10:27:39] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.169 second response time [10:27:41] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 32.14, 24.46, 24.04 [10:28:06] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:28:19] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.920 second response time [10:28:35] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:28:36] PROBLEM - mw201 APT on mw201 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:39] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:28:41] RECOVERY - mw192 SSH on mw192 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:28:53] PROBLEM - mw202 PowerDNS Recursor on mw202 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:29:14] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ferm/functions.conf] [10:29:29] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 32.46, 25.50, 24.35 [10:29:35] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.122 second response time [10:29:38] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.73, 23.55, 23.76 [10:29:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [10:30:07] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.151 second response time [10:30:16] RECOVERY - mw203 PowerDNS Recursor on mw203 is OK: DNS OK: 0.088 seconds response time. mw203.fsslc.wtnet returns 10.0.20.165 [10:30:34] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:30:36] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:43] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:30:43] PROBLEM - mw182 PowerDNS Recursor on mw182 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:30:46] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:31:15] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [10:31:22] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:26] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:31:32] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:35] PROBLEM - mw192 PowerDNS Recursor on mw192 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:31:40] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 42.22, 30.47, 26.23 [10:31:49] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:31:52] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.058 second response time [10:31:58] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:32:12] PROBLEM - mw171 SSH on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:36] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:32:37] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.074 second response time [10:32:39] PROBLEM - mw162 PowerDNS Recursor on mw162 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:32:45] PROBLEM - mw162 SSH on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:47] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.901 second response time [10:32:48] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:02] RECOVERY - mw202 PowerDNS Recursor on mw202 is OK: DNS OK: 0.077 seconds response time. mw202.fsslc.wtnet returns 10.0.20.163 [10:33:04] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:05] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.71, 22.80, 23.45 [10:33:12] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 56% [10:33:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 16.16, 11.95, 11.83 [10:33:25] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.068 second response time [10:33:28] RECOVERY - mw201 APT on mw201 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:33:32] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.697 second response time [10:33:48] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:34:06] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 6.090 second response time [10:34:38] RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.753 seconds response time. mw162.fsslc.wtnet returns 10.0.16.133 [10:34:43] RECOVERY - mw162 SSH on mw162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:34:49] PROBLEM - mw181 PowerDNS Recursor on mw181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:34:51] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.305 second response time [10:35:11] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [10:35:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 5.16, 9.04, 10.77 [10:35:26] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.974 second response time [10:35:27] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.35, 24.83, 22.04 [10:35:28] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 4.518 second response time [10:35:50] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [10:35:51] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:35:55] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [10:35:58] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:10] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:36:13] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:36:16] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:19] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:21] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:36:29] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:34] PROBLEM - mw191 SSH on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:42] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.251 second response time [10:36:43] PROBLEM - mw181 SSH on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:45] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.222 second response time [10:36:54] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:56] RECOVERY - mw192 APT on mw192 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:36:58] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 48.17, 31.57, 23.09 [10:37:00] PROBLEM - mw181 APT on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:03] PROBLEM - mw201 PowerDNS Recursor on mw201 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:37:04] RECOVERY - mw182 PowerDNS Recursor on mw182 is OK: DNS OK: 0.681 seconds response time. mw182.fsslc.wtnet returns 10.0.18.105 [10:37:05] PROBLEM - mw171 PowerDNS Recursor on mw171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:37:06] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:37:08] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 6.844 second response time [10:37:11] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.06, 9.06, 10.48 [10:37:25] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.441 second response time [10:37:52] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.375 second response time [10:37:54] RECOVERY - mw192 PowerDNS Recursor on mw192 is OK: DNS OK: 0.527 seconds response time. mw192.fsslc.wtnet returns 10.0.19.161 [10:37:55] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.424 second response time [10:37:56] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.052 second response time [10:37:57] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 5.514 second response time [10:37:59] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.12, 20.73, 19.84 [10:38:00] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.362 second response time [10:38:05] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 4.818 second response time [10:38:07] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.060 second response time [10:38:09] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.057 second response time [10:38:12] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.260 second response time [10:38:14] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [10:38:15] RECOVERY - mw171 SSH on mw171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:38:15] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.067 second response time [10:38:28] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.206 second response time [10:38:32] RECOVERY - mw191 SSH on mw191 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:38:41] RECOVERY - mw181 SSH on mw181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:38:45] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.446 second response time [10:38:47] RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.064 seconds response time. mw181.fsslc.wtnet returns 10.0.18.104 [10:38:58] RECOVERY - mw181 APT on mw181 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:38:59] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.213 second response time [10:39:08] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 28% [10:39:13] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [10:39:25] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:40:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 33.82, 24.13, 21.06 [10:41:11] PROBLEM - mw202 HTTPS on mw202 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:41:12] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:41:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.18, 9.97, 10.65 [10:41:31] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:42] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.278 second response time [10:41:46] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:59] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:42:04] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.14, 20.07, 19.96 [10:42:10] PROBLEM - mw191 PowerDNS Recursor on mw191 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:42:18] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:42:19] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [10:42:27] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:28] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:42:33] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:42:35] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:36] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:51] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:42:55] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:56] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.95, 22.98, 22.47 [10:43:05] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 58% [10:43:05] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.397 second response time [10:43:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.29, 7.12, 9.52 [10:43:15] RECOVERY - mw171 PowerDNS Recursor on mw171 is OK: DNS OK: 0.148 seconds response time. mw171.fsslc.wtnet returns 10.0.17.122 [10:43:15] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:20] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 39.53, 23.29, 22.02 [10:43:55] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [10:44:05] RECOVERY - mw191 PowerDNS Recursor on mw191 is OK: DNS OK: 0.061 seconds response time. mw191.fsslc.wtnet returns 10.0.19.160 [10:44:07] PROBLEM - mw203 PowerDNS Recursor on mw203 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:44:18] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:21] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.736 second response time [10:44:28] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.233 second response time [10:44:28] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.365 second response time [10:44:29] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.614 second response time [10:44:31] PROBLEM - mw202 APT on mw202 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:32] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.233 second response time [10:44:55] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:44:55] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 22.03, 24.35, 23.14 [10:44:57] PROBLEM - mw172 APT on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:59] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.582 second response time [10:45:00] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.846 second response time [10:45:02] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:45:10] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:11] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 2.304 second response time [10:45:39] RECOVERY - mw201 PowerDNS Recursor on mw201 is OK: DNS OK: 0.575 seconds response time. mw201.fsslc.wtnet returns 10.0.20.162 [10:45:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [10:45:48] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:46:09] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.193 second response time [10:46:37] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.507 second response time [10:46:39] RECOVERY - mw202 APT on mw202 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:46:40] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [10:46:52] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.066 second response time [10:46:54] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.94, 23.13, 22.91 [10:47:08] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.226 second response time [10:47:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 9.61, 12.45, 11.32 [10:47:17] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 11.24, 20.46, 21.59 [10:47:51] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.116 second response time [10:47:59] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 5.563 second response time [10:48:07] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:48:21] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.048 second response time [10:48:56] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:08] PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:49:11] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:25] RECOVERY - mw202 HTTPS on mw202 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.076 second response time [10:49:43] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [10:50:09] RECOVERY - mw203 PowerDNS Recursor on mw203 is OK: DNS OK: 0.344 seconds response time. mw203.fsslc.wtnet returns 10.0.20.165 [10:50:40] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:50:41] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time [10:50:53] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758793850000&orgId=1&to=1758797453359[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758791420000&orgId=1&to=1758797453359 [10:50:55] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 31.62, 24.31, 23.13 [10:51:00] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [10:51:09] PROBLEM - mw171 APT on mw171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:51:31] PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:51:36] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 15.32, 23.24, 23.58 [10:51:39] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:51:45] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:53] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.282 second response time [10:51:54] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:51:58] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.881 second response time [10:52:11] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [10:52:16] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [10:52:16] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [10:52:19] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:52:45] RECOVERY - mw172 APT on mw172 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [10:52:46] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importImages.php --wiki=easonmusicwiki /home/reception/easonmusic --search-recursively --summary=Imported from https://www.easonmusic.com (START) [10:52:56] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:53:10] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.377 second response time [10:53:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 8.79, 10.83, 11.21 [10:53:16] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.98, 20.73, 21.17 [10:53:26] RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.958 second response time [10:53:39] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.82, 25.28, 24.30 [10:53:41] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.435 second response time [10:53:41] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:53:49] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/importImages.php --wiki=easonmusicwiki /home/reception/easonmusic --search-recursively --summary=Imported from https://www.easonmusic.com (END - exit=0) [10:54:06] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 20.43, 21.44, 17.66 [10:54:08] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.068 second response time [10:54:10] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:13] PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:54:14] PROBLEM - mw183 HTTPS on mw183 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [10:54:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.46, 22.23, 22.58 [10:55:14] PROBLEM - mw173 Current Load on mw173 is CRITICAL: LOAD CRITICAL - total load average: 32.72, 25.17, 20.31 [10:55:16] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.28, 20.95, 21.22 [10:55:24] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:55:27] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:44] PROBLEM - mw203 SSH on mw203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:45] PROBLEM - mw161 SSH on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:49] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:55:53] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.271 second response time [10:55:55] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4312 bytes in 9.664 second response time [10:56:05] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 15.70, 18.70, 17.08 [10:56:09] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.470 second response time [10:56:10] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:12] RECOVERY - mw183 HTTPS on mw183 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.217 second response time [10:56:13] PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [10:56:17] RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 9.820 second response time [10:56:52] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.670 second response time [10:56:54] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:55] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42% [10:56:56] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.206 second response time [10:57:05] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.069 second response time [10:57:08] PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 18.42, 21.98, 19.69 [10:57:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 28.43, 14.41, 12.20 [10:57:21] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.787 second response time [10:57:23] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.340 second response time [10:57:42] RECOVERY - mw203 SSH on mw203 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:57:46] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.289 second response time [10:57:50] RECOVERY - mw161 SSH on mw161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [10:57:54] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [10:58:23] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.516 second response time [10:58:25] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [10:58:26] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.547 second response time [10:58:38] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 2.103 second response time [10:59:01] RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 15.26, 19.77, 19.15 [10:59:12] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 44.12, 27.37, 23.33 [10:59:16] RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.066 second response time [10:59:30] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.009 second response time [11:00:20] RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 8.245 second response time [11:00:35] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:43] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:54] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.44, 17.58, 20.29 [11:00:55] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:01:10] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.77, 23.66, 22.45 [11:01:25] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.096 second response time [11:01:25] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 1.987 second response time [11:02:09] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.277 second response time [11:02:12] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [11:02:24] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 3.218 second response time [11:02:28] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 3.185 second response time [11:02:34] PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:02:40] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [11:02:53] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [11:02:58] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:28] PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.111 second response time [11:03:39] PROBLEM - mw191 HTTPS on mw191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:03:51] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 15.76, 21.31, 23.66 [11:04:09] PROBLEM - cp201 Varnish Backends on cp201 is WARNING: No backends detected. If this is an error, see readme.txt [11:04:10] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.066 second response time [11:04:31] RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.208 second response time [11:04:32] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.958 second response time [11:04:36] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [11:04:37] PROBLEM - cp171 Varnish Backends on cp171 is WARNING: No backends detected. If this is an error, see readme.txt [11:04:37] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.195 second response time [11:04:39] PROBLEM - mw181 PowerDNS Recursor on mw181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:04:53] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [11:04:56] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 422s [11:04:56] PROBLEM - cp161 Varnish Backends on cp161 is WARNING: No backends detected. If this is an error, see readme.txt [11:05:05] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 2.91, 14.23, 19.01 [11:05:11] RECOVERY - mw171 APT on mw171 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [11:05:11] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [11:05:17] !log [paladox@mwtask181] starting deploy of {'config': True, 'force': True} to all [11:05:24] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.053 second response time [11:05:29] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 22.05, 26.37, 23.62 [11:05:39] RECOVERY - mw191 HTTPS on mw191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.067 second response time [11:06:02] PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.089 second response time [11:06:32] RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.059 seconds response time. mw181.fsslc.wtnet returns 10.0.18.104 [11:06:40] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.899 second response time [11:06:45] PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 2.26, 13.70, 22.23 [11:06:49] PROBLEM - cp201 health.wikitide.net HTTPS on cp201 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 86 bytes in 1.011 second response time [11:06:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [11:07:04] PROBLEM - cp171 health.wikitide.net HTTPS on cp171 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 86 bytes in 1.012 second response time [11:07:07] PROBLEM - mw183 MediaWiki Rendering on mw183 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.092 second response time [11:07:07] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 1.66, 12.54, 22.17 [11:07:14] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 7.74, 10.55, 11.69 [11:07:21] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.090 second response time [11:07:27] PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 5.866 second response time [11:07:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 5.51, 18.23, 20.95 [11:07:36] PROBLEM - mw152 APT on mw152 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:07:42] PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 6.94, 14.21, 22.69 [11:07:43] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.098 second response time [11:07:49] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 4.69, 11.21, 18.93 [11:07:49] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 7.89, 15.05, 22.64 [11:08:04] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.089 second response time [11:08:06] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 5.41, 14.21, 22.78 [11:08:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw152 [11:08:14] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.060 second response time [11:08:16] RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4279 bytes in 0.072 second response time [11:08:25] PROBLEM - mw202 MediaWiki Rendering on mw202 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.093 second response time [11:08:25] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 7.75, 15.26, 22.66 [11:08:28] PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.097 second response time [11:08:28] PROBLEM - mw191 MediaWiki Rendering on mw191 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.102 second response time [11:08:28] PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.085 second response time [11:08:30] PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:08:37] PROBLEM - mw203 MediaWiki Rendering on mw203 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.096 second response time [11:08:41] PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 8191 bytes in 0.075 second response time [11:08:49] RECOVERY - cp201 health.wikitide.net HTTPS on cp201 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.009 second response time [11:08:56] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4257 bytes in 0.075 second response time [11:09:11] PROBLEM - cp191 Varnish Backends on cp191 is WARNING: No backends detected. If this is an error, see readme.txt [11:09:21] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:09:22] PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 5.43, 14.32, 23.16 [11:09:27] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 6.05, 14.35, 19.20 [11:09:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [11:09:40] RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 3.66, 10.75, 20.40 [11:09:52] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 3.74, 12.91, 22.06 [11:10:09] PROBLEM - cp201 Varnish Backends on cp201 is WARNING: No backends detected. If this is an error, see readme.txt [11:10:11] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 3.06, 13.15, 22.49 [11:10:13] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/mediawiki-firejail-convert] [11:10:16] PROBLEM - mw152 SSH on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:20] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 2.53, 12.98, 23.52 [11:10:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [11:10:37] RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 4.08, 8.96, 18.45 [11:10:40] RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.356 seconds response time. mw152.fsslc.wtnet returns 10.0.15.115 [11:10:53] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758794930000&orgId=1&to=1758798530000[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758791420000&orgId=1&to=1758798653362 [11:11:00] RECOVERY - mw183 MediaWiki Rendering on mw183 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.229 second response time [11:11:00] !log [paladox@mwtask181] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 343s [11:11:00] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 3.95, 8.39, 18.43 [11:11:03] RECOVERY - cp171 health.wikitide.net HTTPS on cp171 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.014 second response time [11:11:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:11:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.02, 6.12, 9.75 [11:11:17] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.214 second response time [11:11:18] RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [11:11:19] RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.215 second response time [11:11:21] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.486 second response time [11:11:26] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.230 second response time [11:11:37] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.198 second response time [11:11:38] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.236 second response time [11:11:44] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 5.03, 9.56, 18.74 [11:11:49] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time [11:11:49] RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.232 second response time [11:11:50] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 6.29, 10.69, 20.16 [11:12:01] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.195 second response time [11:12:04] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.443 second response time [11:12:06] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 5.57, 9.06, 18.67 [11:12:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 5.79, 9.67, 18.69 [11:12:20] RECOVERY - mw202 MediaWiki Rendering on mw202 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.228 second response time [11:12:25] RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time [11:12:26] RECOVERY - mw191 MediaWiki Rendering on mw191 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.223 second response time [11:12:29] RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.431 second response time [11:12:32] RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.234 second response time [11:12:36] RECOVERY - mw203 MediaWiki Rendering on mw203 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.209 second response time [11:12:39] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.200 second response time [11:12:45] PROBLEM - mw152 Current Load on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: Connection refusedconnect to host 10.0.15.115 port 5666: Connection refused [11:12:49] PROBLEM - mw152 conntrack_table_size on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: Connection refusedconnect to host 10.0.15.115 port 5666: Connection refused [11:12:50] PROBLEM - cp201 health.wikitide.net HTTPS on cp201 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 86 bytes in 1.013 second response time [11:13:19] RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 5.59, 9.25, 19.05 [11:13:26] PROBLEM - cp191 health.wikitide.net HTTPS on cp191 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 86 bytes in 1.012 second response time [11:14:09] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 5.53, 9.12, 18.74 [11:14:18] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 5.17, 8.92, 19.45 [11:14:19] RECOVERY - mw152 SSH on mw152 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0) [11:14:23] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4309 bytes in 0.067 second response time [11:14:37] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 1 backends are down. mw152 [11:14:44] RECOVERY - mw152 APT on mw152 is OK: APT OK: 131 packages available for upgrade (0 critical updates). [11:14:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 0.24, 0.07, 0.02 [11:14:49] RECOVERY - mw152 conntrack_table_size on mw152 is OK: OK: nf_conntrack is 0 % full [11:14:49] RECOVERY - cp201 health.wikitide.net HTTPS on cp201 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.010 second response time [11:14:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:14:56] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw152 [11:15:11] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw152 [11:15:25] RECOVERY - cp191 health.wikitide.net HTTPS on cp191 is OK: HTTP OK: HTTP/2 200 - 112 bytes in 0.012 second response time [11:15:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:15:43] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [11:15:53] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758795110000&orgId=1&to=1758798710000[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758795260000&orgId=1&to=1758798860000 [11:16:09] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw152 [11:16:53] RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 14% [11:17:30] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.937 second response time [11:18:07] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [11:18:09] RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy [11:18:32] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4208 bytes in 0.057 second response time [11:18:37] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy [11:18:56] RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy [11:19:11] RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy [11:36:31] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:01:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758798080000&orgId=1&to=1758801713356 [12:12:14] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CommentStreams'} to test151 [12:12:16] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'CommentStreams'} to test151 - SUCCESS in 1s [12:12:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:20:48] miraheze/RottenLinks - translatewiki the build passed. [12:27:58] miraheze/MirahezeMagic - translatewiki the build passed. [12:28:16] !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php purgeParserCache --wiki=monkeisleswiki --age=36000 (END - exit=0) [12:28:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:11:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758802280000&orgId=1&to=1758805880000 [13:27:14] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 36.65, 14.06, 6.39 [13:27:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758803240000&orgId=1&to=1758806873354 [13:29:14] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.47, 9.58, 5.67 [13:47:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758804200000&orgId=1&to=1758807800000 [14:21:45] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.87, 11.73, 6.66 [14:23:41] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.89, 8.22, 5.97 [14:43:08] PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1957MiB (10% inode=89%); [14:58:02] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 5.31, 11.98, 7.47 [14:59:58] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.03, 8.60, 6.76 [15:38:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758811100000&orgId=1&to=1758814733355 [15:58:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758812180000&orgId=1&to=1758815780000 [16:01:08] PROBLEM - matomo151 Disk Space on matomo151 is CRITICAL: DISK CRITICAL - free space: / 1076MiB (5% inode=89%); [16:03:08] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6506MiB (36% inode=89%); [16:26:07] PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27159MiB (5% inode=99%); [16:33:23] PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 27186MiB (5% inode=99%); [17:45:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758818720000&orgId=1&to=1758822353356 [17:50:06] PROBLEM - cp191 Disk Space on cp191 is CRITICAL: DISK CRITICAL - free space: / 27233MiB (5% inode=99%); [18:05:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758819740000&orgId=1&to=1758823340000 [19:11:16] PROBLEM - graylog161 Disk Space on graylog161 is WARNING: DISK WARNING - free space: / 1347MiB (7% inode=93%); [19:13:16] RECOVERY - graylog161 Disk Space on graylog161 is OK: DISK OK - free space: / 2191MiB (12% inode=93%); [19:19:16] PROBLEM - graylog161 Disk Space on graylog161 is CRITICAL: DISK CRITICAL - free space: / 0MiB (0% inode=93%); [19:25:16] PROBLEM - graylog161 Disk Space on graylog161 is WARNING: DISK WARNING - free space: / 1606MiB (8% inode=93%); [19:27:17] RECOVERY - graylog161 Disk Space on graylog161 is OK: DISK OK - free space: / 2861MiB (15% inode=93%); [19:37:50] sigh [20:29:53] [Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758828560000&orgId=1&to=1758832193354 [20:34:53] [Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1758828800000&orgId=1&to=1758832400000 [21:44:50] miraheze/puppet - paladox the build passed. [21:55:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/47eadaea25be3670ba6658ce15d3cdf35710ffe4 [21:55:30] 02ssl/03main 07WikiTideBot 0347eadae Bot: Auto-update domain lists [22:15:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/240617e21fc37c8cbaf37705840e25bc80e8ec1a [22:15:29] 02ssl/03main 07WikiTideBot 03240617e Bot: Auto-update domain lists [22:43:55] <@&1196218038530355290> [22:44:02] aha [22:45:04] hehe [22:45:23] i think you are also literally the only person with that role [22:45:34] can we ping verified wiki users [22:45:54] i am not going to find out [22:46:11] me neither [22:50:49] Skye: smh [22:50:53] Fired [23:15:31] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c3d93a595a4e042d73addef3384890a28d4fdb7d [23:15:31] 02ssl/03main 07WikiTideBot 03c3d93a5 Bot: Auto-update domain lists [23:25:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/52e0c7f546d0a8600f1ab3db78d16a8e571a6936 [23:25:24] 02ssl/03main 07WikiTideBot 0352e0c7f Bot: Auto-update domain lists [23:25:30] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/73812d4d44672db7cbd03b635e7d2e46b2fe328a [23:25:30] 02ssl/03main 07MacFan4000 0373812d4 Update redirects.yaml [23:31:09] PROBLEM - www.dccomicswiki.com - Cloudflare on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.805B29FB26150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [23:35:28] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/1b9c50ef8caf646494e467bbe057f4f807f25c23 [23:35:28] 02ssl/03main 07WikiTideBot 031b9c50e Bot: Auto-update domain lists [23:45:31] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/82f939d113aa84bac3d58ba1ab764ff3bfce67ca [23:45:31] 02ssl/03main 07WikiTideBot 0382f939d Bot: Auto-update domain lists [23:52:18] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/cf63c3b62c2523236584f52af16b2958fa6a5172 [23:52:18] 02ssl/03main 07MacFan4000 03cf63c3b Update redirects.yaml [23:54:10] PROBLEM - www.rippaversewiki.com - Cloudflare on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.804BABD600150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1605:SSL alert number 40 [23:55:29] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/e2646d1db6a3f7b17f8c848b1aaa0f40d1ba6c58 [23:55:29] 02ssl/03main 07WikiTideBot 03e2646d1 Bot: Auto-update domain lists