[00:00:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.92, 3.94, 3.82 [00:02:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.76, 3.75, 3.79 [00:06:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.38, 3.66, 3.69 [00:08:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.35, 3.05, 3.46 [00:10:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.16, 3.15, 3.42 [00:12:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.72, 3.30, 3.47 [00:14:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.66, 4.27, 3.81 [00:18:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:19:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 21.39, 17.93 [00:21:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.42, 20.69, 18.13 [00:21:22] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:23:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.71, 18.37, 17.58 [00:25:29] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [00:31:36] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:35:13] miraheze/ssl - WikiTideSSLBot the build has errored. [00:36:35] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:38:46] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:42:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.88, 3.43, 3.88 [00:42:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.35, 19.87, 18.32 [00:43:46] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:44:40] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.81, 19.23, 18.31 [00:46:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.49, 3.78, 3.91 [00:48:17] RECOVERY - wiki.taes.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.taes.com.br' will expire on Sat 09 Nov 2024 10:51:01 PM GMT +0000. [00:48:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.07, 3.02, 3.60 [00:52:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.67, 2.84, 3.39 [00:58:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.84, 3.31, 3.43 [01:00:08] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.86, 21.59, 19.34 [01:04:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.98, 2.71, 3.21 [01:05:54] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 22.91, 20.56 [01:07:50] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.39, 21.71, 20.44 [01:09:45] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.33, 18.63, 19.48 [01:12:46] [02RequestSSL] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/commit/38b9d5717648 [01:12:49] [02RequestSSL] 07dependabot[bot] 0338b9d57 - Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 [01:12:50] [02RequestSSL] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 - 13https://github.com/miraheze/RequestSSL [01:12:51] [02RequestSSL] 07dependabot[bot] labeled pull request 03#54: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RequestSSL/pull/54 [01:12:52] [02RequestSSL] 07dependabot[bot] labeled pull request 03#54: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RequestSSL/pull/54 [01:12:53] [02RequestSSL] 07dependabot[bot] opened pull request 03#54: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RequestSSL/pull/54 [01:12:56] [02RequestSSL] 07coderabbitai[bot] commented on pull request 03#54: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RequestSSL/pull/54#issuecomment-2282972346 [01:16:49] miraheze/RequestSSL - dependabot[bot] the build has errored. [02:23:50] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.25, 22.09, 18.55 [02:25:46] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.03, 22.98, 19.34 [02:35:23] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.37, 19.30, 19.46 [02:53:15] PROBLEM - wiki.gestaltonline.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gestaltonline.ca' expires in 15 day(s) (Wed 28 Aug 2024 02:26:53 AM GMT +0000). [02:53:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9aeb3fa6c3a1...e9a67c44c297 [02:53:27] [02ssl] 07WikiTideSSLBot 03e9a67c4 - Bot: Update SSL cert for wiki.gestaltonline.ca [03:00:19] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 18 seconds old and 0 bytes [03:10:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [03:18:04] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.31, 21.88, 19.64 [03:20:00] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.89, 18.79, 18.77 [03:21:59] RECOVERY - wiki.gestaltonline.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gestaltonline.ca' will expire on Sun 10 Nov 2024 01:54:51 AM GMT +0000. [03:27:39] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.52, 2.72 [03:29:35] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.55, 3.55, 2.81 [03:31:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.31, 3.07, 2.73 [03:32:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:39:00] miraheze/ssl - WikiTideSSLBot the build has errored. [03:45:02] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.18, 3.68, 3.22 [03:46:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.38, 4.00, 3.37 [03:48:52] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.93, 3.29, 3.17 [03:57:43] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.35, 5.41, 4.05 [03:59:40] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:10:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.61, 3.52, 3.82 [04:12:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.72, 3.82 [04:18:33] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.22, 3.05, 3.53 [04:24:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.98, 3.55, 3.58 [04:30:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.36, 3.53, 3.64 [04:32:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.39, 4.50, 3.97 [04:33:22] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:34] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:35:25] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:37:33] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:38:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.16, 3.76, 3.89 [04:40:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [04:44:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.08, 2.46, 3.31 [04:50:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.78, 5.24, 4.19 [04:54:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.45, 3.52, 3.76 [04:56:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 4.25, 3.99 [05:06:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.93, 3.51, 4.00 [05:08:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.84, 4.05, 4.15 [05:10:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.13, 3.37, 3.89 [05:12:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 3.48, 3.84 [05:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.92, 3.22, 3.68 [05:15:55] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:18:24] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 21.37, 18.56 [05:18:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 3.81, 3.76 [05:19:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:20:19] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.88, 21.37, 18.89 [05:21:26] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:24:10] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.94, 19.40, 18.65 [05:24:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.53, 3.54, 3.82 [05:28:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 3.73, 3.84 [05:34:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.35, 3.37, 3.77 [05:38:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 4.22, 4.02 [05:39:02] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:06] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [05:44:30] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:46:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.04, 3.53, 3.98 [05:48:33] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 3.69, 3.96 [05:51:01] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:51:19] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:55] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:59:36] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [06:02:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.52, 3.50, 3.90 [06:08:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 3.85, 3.88 [06:10:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.14, 3.28, 3.65 [06:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.29, 3.70, 3.70 [06:16:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.14, 3.85, 3.79 [06:18:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.37, 4.95, 4.18 [06:24:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.73, 3.77, 4.00 [06:28:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.37, 4.12, 4.05 [06:30:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.08, 3.45, 3.82 [06:34:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.16, 3.85, 3.87 [06:36:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.32, 3.58, 3.79 [06:40:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.35, 3.69, 3.76 [06:42:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.52, 3.24, 3.57 [06:46:20] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:47:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:48:34] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.36, 4.55, 3.93 [06:51:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.49, 19.58, 18.00 [06:53:24] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.80, 19.00, 17.97 [06:54:33] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.98, 3.79, 3.81 [06:58:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.54, 4.23, 3.93 [07:00:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.09, 3.37, 3.66 [07:00:42] [02phorge-extensions] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 - 13https://github.com/miraheze/phorge-extensions [07:00:44] [02phorge-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/commit/eea13bf1aed2 [07:00:47] [02phorge-extensions] 07dependabot[bot] 03eea13bf - Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 [07:00:48] [02phorge-extensions] 07dependabot[bot] labeled pull request 03#49: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/phorge-extensions/pull/49 [07:00:50] [02phorge-extensions] 07dependabot[bot] labeled pull request 03#49: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/phorge-extensions/pull/49 [07:00:53] [02phorge-extensions] 07dependabot[bot] opened pull request 03#49: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/phorge-extensions/pull/49 [07:01:39] miraheze/phorge-extensions - dependabot[bot] the build passed. [07:03:55] [02phorge-extensions] 07coderabbitai[bot] commented on pull request 03#49: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/phorge-extensions/pull/49#issuecomment-2283241176 [07:04:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.37, 3.99, 3.79 [07:06:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.16, 3.27, 3.56 [07:12:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.34, 2.89, 3.35 [07:14:30] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:21:23] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.95, 3.43, 3.35 [07:23:18] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.70, 3.99, 3.54 [07:25:13] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.38, 3.39, 3.36 [07:29:04] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.14, 3.63, 3.52 [07:30:59] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.70, 4.55, 3.88 [07:32:56] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.92, 3.75, 3.66 [07:38:42] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.43, 4.07, 3.78 [07:42:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.07, 3.82, 3.78 [07:44:46] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.88, 21.46, 20.15 [07:46:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.20, 4.41, 3.99 [07:48:37] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.08, 20.22, 19.97 [07:50:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:55:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:02:08] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.21, 19.68, 19.44 [08:04:03] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.43, 18.12, 18.88 [08:10:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.66, 3.28, 3.81 [08:16:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.47, 3.93, 3.90 [08:18:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.99, 3.23, 3.63 [08:20:35] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 4.13, 3.91 [08:22:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.02, 3.57, 3.76 [08:24:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.06, 4.46, 4.05 [08:26:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.77, 21.20, 19.37 [08:32:01] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.16, 19.84, 19.50 [08:32:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.94, 3.83, 3.97 [08:38:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.23, 4.20, 3.94 [08:38:47] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:39:07] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:41] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.228 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [08:41:04] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [08:44:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.57, 3.76, 3.97 [08:45:23] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:48:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.87, 4.91, 4.34 [09:03:27] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:08:27] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:08:46] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:44] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003896653652 secs [09:16:18] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:18:20] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o