[00:00:06] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:02:06] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 121.58 ms [00:04:10] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/flake8-comprehensions-3.16.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/a4b1503bf5f7 [00:04:13] [02python-functions] 07dependabot[bot] 03a4b1503 - Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github [00:04:15] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/flake8-comprehensions-3.16.0 - 13https://github.com/miraheze/python-functions [00:04:17] [02python-functions] 07dependabot[bot] labeled pull request 03#89: Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github - 13https://github.com/miraheze/python-functions/pull/89 [00:04:18] [02python-functions] 07dependabot[bot] opened pull request 03#89: Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github - 13https://github.com/miraheze/python-functions/pull/89 [00:04:20] [02python-functions] 07dependabot[bot] labeled pull request 03#89: Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github - 13https://github.com/miraheze/python-functions/pull/89 [00:04:21] [02python-functions] 07coderabbitai[bot] commented on pull request 03#89: Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github - 13https://github.com/miraheze/python-functions/pull/89#issuecomment-2442898775 [00:08:11] [02python-functions] 07github-merge-queue[bot] created branch 03gh-readonly-queue/master/pr-89-7752191f6367a9d0e4c097d5259c848f21106c9c - 13https://github.com/miraheze/python-functions [00:08:14] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03gh-readonly-queue/master/pr-89-7752191f6367a9d0e4c097d5259c848f21106c9c [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/d797d7e4aef1 [00:08:16] [02python-functions] 07dependabot[bot] 03d797d7e - Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github (#89) [00:08:20] miraheze/python-functions - dependabot[bot] the build passed. [00:11:32] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 126.53 ms [00:12:12] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/python-functions/compare/7752191f6367...d797d7e4aef1 [00:12:14] [02python-functions] 07dependabot[bot] 03d797d7e - Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github (#89) [00:12:16] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-89-7752191f6367a9d0e4c097d5259c848f21106c9c [00:12:19] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-89-7752191f6367a9d0e4c097d5259c848f21106c9c - 13https://github.com/miraheze/python-functions [00:12:20] [02python-functions] 07github-actions[bot] closed pull request 03#89: Bump flake8-comprehensions from 3.15.0 to 3.16.0 in /.github - 13https://github.com/miraheze/python-functions/pull/89 [00:12:21] [02python-functions] 07github-actions[bot] deleted branch 03dependabot/pip/dot-github/flake8-comprehensions-3.16.0 - 13https://github.com/miraheze/python-functions [00:12:22] [02python-functions] 07github-actions[bot] deleted branch 03dependabot/pip/dot-github/flake8-comprehensions-3.16.0 [00:16:26] miraheze/python-functions - github-merge-queue[bot] the build passed. [00:19:51] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 127.13 ms [00:26:06] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 125.42 ms [00:34:25] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 123.85 ms [00:43:08] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 127.16 ms [00:45:09] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 128.40 ms [01:36:15] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 128.52 ms [01:40:21] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 127.72 ms [03:01:02] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 31.67 ms [03:03:02] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.64 ms [03:11:44] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.67 ms [03:20:03] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.77 ms [04:07:05] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.69 ms [04:19:36] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms [05:21:45] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 31.64 ms [05:30:04] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.63 ms [05:44:30] PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [05:50:27] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [05:54:34] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.71 ms [06:14:01] RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000. [07:03:07] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.70 ms [07:03:22] Yes icinga we know [07:03:29] Now go away and shut up about it [07:05:08] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.65 ms [07:05:38] !log sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/invalidateUserSessions.php --wiki=loginwiki --user "Kugeln.io" [07:05:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:29] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.70 ms [07:17:48] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.73 ms [07:37:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:57:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:26:38] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:37:07] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.68 ms [09:19:37] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.66 ms [09:21:37] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms [09:26:01] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 31.64 ms [09:26:04] PROBLEM - Host mattermost1 is DOWN: PING CRITICAL - Packet loss = 100% [09:30:32] RECOVERY - Host mattermost1 is UP: PING OK - Packet loss = 0%, RTA = 31.71 ms [09:38:38] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.70 ms [11:11:46] [02CreateWiki] 07BlankEclair reviewed pull request 03#605 commit - 13https://github.com/miraheze/CreateWiki/pull/605#discussion_r1820594945 [11:12:55] [02CreateWiki] 07BlankEclair reviewed pull request 03#605 commit - 13https://github.com/miraheze/CreateWiki/pull/605#discussion_r1820598682 [11:13:56] [02CreateWiki] 07BlankEclair reviewed pull request 03#579 commit - 13https://github.com/miraheze/CreateWiki/pull/579#discussion_r1820600095 [11:22:14] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [11:29:44] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 31.64 ms [11:31:45] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.68 ms [11:48:28] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [11:57:40] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 32.33 ms [11:59:40] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.66 ms [12:04:02] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 31.71 ms [12:04:10] [02dns] 07Reception123 closed pull request 03#641: Rename zones/sudo /root/androidwiki.info to zones/androidwiki.info - 13https://github.com/miraheze/dns/pull/641 [12:04:13] [02dns] 07Reception123 pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/dns/compare/c7cab27c6247...bb75bf1ef229 [12:04:15] [02dns] 07MacFan4000 03bb75bf1 - Rename zones/sudo /root/androidwiki.info to zones/androidwiki.info (#641) [12:05:09] miraheze/dns - Reception123 the build passed. [12:06:02] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.62 ms [12:26:38] [02MirahezeMagic] 07Reception123 created branch 03Reception123-patch-2 - 13https://github.com/miraheze/MirahezeMagic [12:26:39] [02MirahezeMagic] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/36c3d41c6a08 [12:26:41] [02MirahezeMagic] 07Reception123 0336c3d41 - add classification of wiki types field [12:26:43] [02MirahezeMagic] 07Reception123 opened pull request 03#521: add classification of wiki types field - 13https://github.com/miraheze/MirahezeMagic/pull/521 [12:26:50] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#521: add classification of wiki types field - 13https://github.com/miraheze/MirahezeMagic/pull/521#issuecomment-2444071607 [12:28:33] [02MirahezeMagic] 07github-actions[bot] pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/36c3d41c6a08...6de6bfb4b7d7 [12:28:36] [02MirahezeMagic] 07github-actions 036de6bfb - CI: lint code to MediaWiki standards [12:28:38] [02MirahezeMagic] 07github-actions[bot] synchronize pull request 03#521: add classification of wiki types field - 13https://github.com/miraheze/MirahezeMagic/pull/521 [12:30:00] miraheze/MirahezeMagic - Reception123 the build has errored. [12:52:08] @rhinosf1 downtimed the ping check on Icinga [13:08:14] Peace [15:00:42] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.21 ms [15:01:05] PROBLEM - eventgate181 Puppet on eventgate181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:02:54] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:04:39] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms [15:09:28] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:15:04] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:16:39] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.19 ms [15:17:32] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:17:55] uh [15:18:06] is cloud17 okay? [15:19:39] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.19 ms [15:19:53] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:19:57] PROBLEM - mw174 Puppet on mw174 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:20:21] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:20:25] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:20:45] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:21:19] PROBLEM - tssm.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:24] PROBLEM - db181 MariaDB on db181 is UNKNOWN: [15:22:04] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 137.76 ms [15:22:31] PROBLEM - theenglishlakes.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:32] PROBLEM - emmytherobot.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:34] PROBLEM - db182 MariaDB on db182 is UNKNOWN: [15:23:04] PROBLEM - db151 MariaDB on db151 is UNKNOWN: [15:23:05] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:36] PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:23:39] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:02] PROBLEM - changeprop151 Puppet on changeprop151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:07] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.74 ms [15:24:17] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:36] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:25:01] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 1037603 Threads: 6 Questions: 1218258276 Slow queries: 464 Opens: 422896 Open tables: 1222 Queries per second avg: 1174.108 [15:25:01] RECOVERY - db151 MariaDB on db151 is OK: Uptime: 1037634 Threads: 69 Questions: 2542552920 Slow queries: 394 Opens: 625188 Open tables: 210000 Queries per second avg: 2450.336 [15:25:15] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 9 backends are down. mw152 mw162 mw171 mw181 mw153 mw154 mw173 mw174 mw183 [15:25:36] RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [15:25:39] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.21 ms [15:25:54] PROBLEM - db172 MariaDB on db172 is UNKNOWN: [15:25:58] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 8 backends are down. mw151 mw162 mw182 mw153 mw164 mw173 mw183 mw184 [15:25:59] PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [15:25:59] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [15:26:12] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:26:14] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:26:23] PROBLEM - db161 MariaDB on db161 is UNKNOWN: [15:26:43] PROBLEM - db171 MariaDB on db171 is UNKNOWN: [15:26:44] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms [15:27:18] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:27:58] RECOVERY - db151 MariaDB Connections on db151 is OK: OK connection usage: 5.3%Current connections: 53 [15:27:58] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:28:00] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:28:00] PROBLEM - persist.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:00] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:01] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:08] PROBLEM - graphite151 Puppet on graphite151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:28:26] PROBLEM - pandorastale.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:21] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 335051 Threads: 258 Questions: 1165568833 Slow queries: 123 Opens: 292991 Open tables: 210000 Queries per second avg: 3478.780 [15:29:29] PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:29:49] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4021 bytes in 0.061 second response time [15:29:53] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 0.23 ms [15:29:57] PROBLEM - bast181 Puppet on bast181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:29:59] PROBLEM - swiftobject161 Puppet on swiftobject161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:30:08] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:46] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4021 bytes in 0.065 second response time [15:30:51] RECOVERY - db172 MariaDB on db172 is OK: Uptime: 1017883 Threads: 1 Questions: 9424748 Slow queries: 0 Opens: 3960 Open tables: 527 Queries per second avg: 9.259 [15:31:12] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:31:25] PROBLEM - wikigenius.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:28] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 137.70 ms [15:31:32] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:35] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 48% [15:31:51] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:31:52] RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms [15:31:53] PROBLEM - db182 MariaDB on db182 is UNKNOWN: [15:31:56] PROBLEM - db151 MariaDB on db151 is UNKNOWN: [15:31:59] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:32:16] PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:32:18] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 0.16 ms [15:32:42] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:32:52] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 0.22 ms [15:32:58] PROBLEM - db172 Puppet on db172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:33:05] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:33:19] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:33:22] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:33:25] PROBLEM - db161 Puppet on db161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:33:35] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 36% [15:34:00] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:34:06] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 1038162 Threads: 11 Questions: 1218387430 Slow queries: 464 Opens: 422896 Open tables: 1222 Queries per second avg: 1173.600 [15:34:18] RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.17 ms [15:34:38] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:34:55] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 150420 Threads: 37 Questions: 204140304 Slow queries: 74 Opens: 128916 Open tables: 128577 Queries per second avg: 1357.135 [15:35:14] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:28] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:35:29] @agentisai any idea of cause [15:36:35] PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:36:39] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:37:07] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:37:12] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:37:37] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.76 ms [15:37:43] PROBLEM - db172 MariaDB on db172 is UNKNOWN: [15:38:01] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.19 ms [15:38:38] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:38:40] PROBLEM - edapedia.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:40] PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:04] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4021 bytes in 0.055 second response time [15:39:13] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [15:39:13] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [15:39:23] PROBLEM - vesc.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:49] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Warning: mysqli_real_connect(): Error while reading greeting packet. PID=3630184 in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): Error while reading greeting packet. PID=3630184 in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66PHP Fatal error: Uncaught mysqli_sql_exception: MySQL server has gone away in /usr/lib/ [15:39:56] That’s new [15:40:30] @agentisai [15:46:26] PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:46:29] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:30] [Grafana] FIRING: An unusually high number of threats are being reported by CloudFlare! https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: A MediaWiki pool is sick according to CloudFlare https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:46:37] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:46:53] PROBLEM - mw151.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:57] RECOVERY - mattermost1 PowerDNS Recursor on mattermost1 is OK: DNS OK: 0.027 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [15:47:04] PROBLEM - mockgovernments.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:09] PROBLEM - cloud17 Current Load on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:47:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:47:23] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:47:28] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:47:30] PROBLEM - clubpenguinfanon.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:30] PROBLEM - kafka181 Puppet on kafka181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:47:39] PROBLEM - ecole.science - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:41] PROBLEM - poupedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:41] PROBLEM - mw162.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:44] PROBLEM - aryavratpedia.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:50] PROBLEM - osfirsttimer.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:59] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [15:48:22] PROBLEM - johanloopmans.nl - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:22] PROBLEM - rarewarewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:25] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:34] PROBLEM - voecwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:39] RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.17 ms [15:48:57] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:04] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:49:20] PROBLEM - cloud15 conntrack_table_size on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:49:31] PROBLEM - inourownwords.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:34] PROBLEM - darkwaterswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:37] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:37] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 0.23 ms [15:49:55] PROBLEM - db171 MariaDB on db171 is CRITICAL: Lost connection to server at 'handshake: reading initial communication packet', system error: 104 [15:49:57] PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:59] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 50% [15:49:59] PROBLEM - mw182.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:07] PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:12] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:50:21] RECOVERY - mattermost1 Mattermost on mattermost1 is OK: PROCS OK: 1 process with args '/opt/mattermost/bin/mattermost' [15:50:48] PROBLEM - thegreatwar.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:27] PROBLEM - cloud16 APT on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:33] PROBLEM - mw163.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:35] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:39] PROBLEM - gimkit.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:57] PROBLEM - kingdomdeath.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:58] PROBLEM - mattermost1 NTP time on mattermost1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:59] PROBLEM - ping6 on cloud15 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 0.19 ms [15:57:11] PROBLEM - lostmediawiki.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:13] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:57:18] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 1.5%Current connections: 15 [15:57:18] RECOVERY - cloud15 conntrack_table_size on cloud15 is OK: OK: nf_conntrack is 0 % full [15:57:26] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [15:57:26] PROBLEM - db172 MariaDB on db172 is UNKNOWN: [15:57:37] PROBLEM - cloud17 Disk Space on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:37] PROBLEM - cloud17 PowerDNS Recursor on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:44] RECOVERY - cloud17 SSH on cloud17 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [15:57:45] PROBLEM - corru.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:45] PROBLEM - cloud17 ferm_active on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:46] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:48] PROBLEM - cloud17 NTP time on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:54] RECOVERY - cloud15 Current Load on cloud15 is OK: LOAD OK - total load average: 8.57, 7.22, 9.04 [15:57:59] PROBLEM - geeu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:46] RECOVERY - mattermost1 Mattermost on mattermost1 is OK: PROCS OK: 1 process with args '/opt/mattermost/bin/mattermost' [16:03:46] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:46] PROBLEM - volunteerforukraine.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:48] RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 3.569 second response time [16:03:54] RECOVERY - mattermost1 PowerDNS Recursor on mattermost1 is OK: DNS OK: 0.027 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.205 [16:03:59] RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 1.923 second response time [16:03:59] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 9.3%Current connections: 93 [16:04:07] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:12] PROBLEM - cloud16 PowerDNS Recursor on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:18] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:21] PROBLEM - agesofconflict.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:22] PROBLEM - beidipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:32] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:04:33] PROBLEM - terraforming.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:33] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:33] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:33] PROBLEM - mw164.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:33] PROBLEM - fashiondreamer.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:35] PROBLEM - hastursnotebook.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:40] PROBLEM - az-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:40] PROBLEM - mw154 MediaWiki Rendering on mw154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time [16:04:41] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:41] RECOVERY - cloud18 Disk Space on cloud18 is OK: DISK OK - free space: / 76692MiB (84% inode=98%); [16:05:01] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:05:01] PROBLEM - cloud18 IPMI Sensors on cloud18 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:02] PROBLEM - mw154.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:10] PROBLEM - db181 MariaDB on db181 is UNKNOWN: [16:05:13] PROBLEM - mw154 HTTPS on mw154 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:05:21] PROBLEM - cloud15 Current Load on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:26] PROBLEM - rct.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:29] PROBLEM - phighting.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:30] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:05:30] PROBLEM - rdb151 Puppet on rdb151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:05:30] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:51] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.012 second response time [16:05:51] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:05:53] RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 3.763 second response time [16:06:05] RECOVERY - cloud15 PowerDNS Recursor on cloud15 is OK: DNS OK: 0.028 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205 [16:06:05] PROBLEM - ncuwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:05] PROBLEM - mw153 MediaWiki Rendering on mw153 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time [16:06:05] PROBLEM - apeirology.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:05] PROBLEM - landofliberos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:27] RECOVERY - cloud16 conntrack_table_size on cloud16 is OK: OK: nf_conntrack is 0 % full [16:06:32] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:06:33] PROBLEM - mw174.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:38] RECOVERY - cloud16 IPMI Sensors on cloud16 is OK: IPMI Status: OK [16:06:39] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [16:06:41] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [16:06:47] PROBLEM - maxcapacity.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:47] RECOVERY - cloud17 IPMI Sensors on cloud17 is OK: IPMI Status: OK [16:06:51] PROBLEM - psycho.engineering - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:51] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 1.00 ms [16:06:53] RECOVERY - cloud18 Current Load on cloud18 is OK: LOAD OK - total load average: 5.70, 4.32, 4.74 [16:06:58] RECOVERY - cloud18 ferm_active on cloud18 is OK: OK ferm input default policy is set [16:06:58] PROBLEM - holocron.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:02] RECOVERY - cloud18 PowerDNS Recursor on cloud18 is OK: DNS OK: 2.171 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [16:07:07] RECOVERY - cloud18 IPMI Sensors on cloud18 is OK: IPMI Status: OK [16:07:07] PROBLEM - annapolishistorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:08] PROBLEM - worldtriggerwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:15] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.69 ms [16:07:24] RECOVERY - cloud15 ferm_active on cloud15 is OK: OK ferm input default policy is set [16:07:26] PROBLEM - mw153.wikitide.net SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:28] PROBLEM - lostidols.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:28] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001589953899 secs [16:07:33] PROBLEM - kagaga.jp - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:35] PROBLEM - nintendowiki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:38] PROBLEM - portalsofphereon.com - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:44] RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK [16:07:47] PROBLEM - kalons-reverie.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:47] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:07:48] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:57] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:07:59] RECOVERY - cloud16 Current Load on cloud16 is OK: LOAD OK - total load average: 4.38, 2.87, 3.19 [16:08:12] RECOVERY - cloud17 Current Load on cloud17 is OK: LOAD OK - total load average: 4.98, 4.08, 4.79 [16:08:21] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:08:34] PROBLEM - trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:44] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 2.8%Current connections: 28 [16:09:00] PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:02] RECOVERY - vesc.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vesc.wiki' will expire on Thu 26 Dec 2024 02:32:54 PM GMT +0000. [16:09:03] RECOVERY - cloud17 ferm_active on cloud17 is OK: OK ferm input default policy is set [16:09:17] RECOVERY - cloud15 NTP time on cloud15 is OK: NTP OK: Offset -0.0002253055573 secs [16:09:17] PROBLEM - cloud16 ferm_active on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:09:39] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [16:10:00] RECOVERY - cloud16 PowerDNS Recursor on cloud16 is OK: DNS OK: 0.027 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [16:10:07] RECOVERY - mattermost1 conntrack_table_size on mattermost1 is OK: OK: nf_conntrack is 0 % full [16:10:30] PROBLEM - pyramidgames.wiki - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:30] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:08] RECOVERY - mattermost1 NTP time on mattermost1 is OK: NTP OK: Offset -0.0001502633095 secs [16:11:16] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:11:20] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:11:40] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 51% [16:11:46] PROBLEM - caverncrusher.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:55] PROBLEM - looneypyramids.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:56] RECOVERY - mattermost1 APT on mattermost1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:12:20] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4021 bytes in 0.313 second response time [16:12:58] PROBLEM - weavefarers.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:03] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 0.16 ms [16:13:12] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4021 bytes in 0.058 second response time [16:13:17] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:31] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 137.75 ms [16:13:37] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:38] PROBLEM - cloud16 IPMI Sensors on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:14:09] PROBLEM - aryavartpedia.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:12] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:18] RECOVERY - mw153 MediaWiki Rendering on mw153 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.991 second response time [16:14:18] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [16:14:21] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:32] PROBLEM - ns1 NTP time on ns1 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o