[00:00:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:01:31] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.17, 4.85, 4.24
[00:10:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:11:31] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.13, 3.60, 3.92
[00:15:29] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.69, 4.07, 3.99
[00:17:28] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.51, 3.47, 3.76
[00:21:27] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.25, 3.95, 3.85
[00:21:45] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:23:26] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.77, 3.47, 3.67
[00:25:26] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.54, 4.66, 4.08
[00:27:40] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[00:29:38] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[00:30:42] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:31:45] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:33:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:36:55] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.781 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:38:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:43:22] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.35, 3.18, 3.93
[00:44:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:47:20] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.48, 4.81, 4.38
[00:49:21] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[00:51:18] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.084 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[00:54:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:57:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:58:01] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 22.37, 20.78
[00:59:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.90, 21.35, 20.59
[01:01:49] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.34, 19.22, 19.90
[01:07:56] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:09:39] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:11:37] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures
[01:12:00] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.059 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:12:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:15:09] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[01:17:11] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[01:21:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:22:54] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:24:54] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.80, 19.86, 19.24
[01:28:26] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:28:59] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[01:30:54] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.12, 19.97, 19.62
[01:34:54] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 22.38, 20.66
[01:36:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:36:54] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.19, 22.54, 20.95
[01:40:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:40:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.49, 25.31, 22.27
[01:40:58] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.208 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[01:43:24] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/396e3ecb6e87...67ea193b7ec4
[01:43:26] <botifico>	 [02ssl] 07WikiTideSSLBot 0367ea193 - Bot: Add SSL cert for clubpenguinfanon.wiki
[01:44:54] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.81, 23.37, 22.29
[01:45:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.30, 3.13, 3.75
[01:45:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:46:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:48:55] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.09, 22.47, 22.05
[01:49:12] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.07, 2.48, 3.39
[01:53:13] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.02, 3.15, 3.53
[01:54:20] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#538: add requested records for phighting.wiki - 13https://github.com/miraheze/dns/pull/538
[01:55:00] <MacFan4000>	 @orduin https://github.com/miraheze/dns/pull/538
[01:55:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.77, 4.02, 3.79
[01:56:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:57:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.73, 3.45, 3.62
[01:58:25] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.04, 19.03, 16.53
[02:00:25] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.90, 17.27, 16.18
[02:00:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:01:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.73, 3.68
[02:03:11] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.54, 3.11, 3.45
[02:05:11] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.69, 2.82, 3.29
[02:05:50] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:07:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:12:49] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[02:13:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.59, 5.03, 3.97
[02:14:02] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:14:46] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.088 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[02:15:12] <botifico>	 [02dns] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e63c35f004af...a31bb03dac99
[02:15:15] <botifico>	 [02dns] 07MacFan4000 03a31bb03 - add requested records for phighting.wiki (#538)
[02:15:18] <botifico>	 [02dns] 07The-Voidwalker closed pull request 03#538: add requested records for phighting.wiki - 13https://github.com/miraheze/dns/pull/538
[02:15:25] <MirahezeRelay>	 <orduin> Done!
[02:16:18] <MirahezeRelay>	 <songngu.xyz> uh and btw, it might not seems neccessary, but is it possible to change my Phorge username to `Pisces`?
[02:16:38] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.74, 21.85, 18.90
[02:17:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:18:01] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[02:18:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.97, 3.86, 3.88
[02:18:37] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 22.70, 19.56
[02:19:59] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.30, 21.37, 17.22
[02:21:56] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.42, 21.56, 17.81
[02:22:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:22:26] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.45, 4.22, 3.98
[02:23:02] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:24:21] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.87, 3.56, 3.79
[02:24:37] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax]
[02:24:58] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.245 second response time
[02:25:52] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.37, 22.59, 18.97
[02:26:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.72, 19.60, 16.43
[02:26:16] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 4.09, 3.96
[02:27:20] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:27:49] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.98, 19.38, 18.24
[02:28:06] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.92, 17.43, 16.01
[02:28:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.08, 3.47, 3.76
[02:28:33] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.09, 22.21, 21.42
[02:32:20] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:32:31] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.99, 17.74, 19.75
[02:35:50] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.69, 2.80, 3.37
[02:37:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:42:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:46:07] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax]
[02:47:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:52:02] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[02:52:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[02:55:34] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[02:56:08] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.13, 4.24, 3.56
[02:57:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[02:58:04] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.88, 3.47, 3.35
[03:00:00] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.58, 2.96, 3.19
[03:02:20] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:03:06] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[03:03:15] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:03:52] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.79, 4.99, 3.97
[03:05:02] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.112 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[03:05:09] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[03:07:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:07:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.45, 3.10, 3.50
[03:09:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.08, 2.08, 3.07
[03:14:25] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[03:45:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[03:47:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[04:29:46] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[04:31:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[05:05:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.88, 23.04, 23.95
[05:07:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 23.41, 23.95
[05:13:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.04, 22.15, 23.55
[05:15:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[05:17:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[05:17:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 24.96, 24.31
[05:21:56] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:23:53] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 4.932284355e-05 secs
[05:29:14] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.50, 20.88, 18.74
[05:31:13] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.93, 19.81, 18.60
[05:45:06] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.00, 21.40, 19.31
[05:50:32] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.60, 19.13, 15.69
[05:51:22] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.12, 20.44, 17.35
[05:52:32] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.94, 19.09, 16.11
[05:53:22] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.47, 20.28, 17.70
[05:59:00] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.19, 22.66, 22.40
[05:59:46] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[06:01:42] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[06:10:55] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.66, 18.16, 20.18
[06:30:15] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[06:49:39] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[06:51:35] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[07:13:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.78, 22.27, 23.77
[07:19:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.11, 23.15, 23.57
[07:23:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.79, 22.14, 23.18
[07:28:57] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[07:35:52] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.49, 17.62, 20.37
[08:27:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:46:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 21.12, 17.65
[08:48:45] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.86, 20.36, 17.78
[09:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:02:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.67, 22.02, 19.32
[09:06:26] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.04, 23.35, 20.57
[09:07:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:09:40] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[09:09:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.05, 4.09, 2.17
[09:10:22] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 24.11, 21.38
[09:11:36] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.081 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[09:12:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:16:40] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:17:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.24, 3.64, 2.82
[09:21:40] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:21:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.95, 3.12
[09:21:50] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:24:09] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.59, 21.48, 23.07
[09:25:09] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[09:25:42] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[09:26:50] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:27:25] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:28:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:29:20] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[09:32:02] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.50, 16.53, 19.98
[09:33:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:33:23] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.063 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[09:35:20] <icinga-miraheze>	 [Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:36:40] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:36:43] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures
[09:38:38] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[09:40:20] <icinga-miraheze>	 [Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:47:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:52:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:56:22] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[09:56:40] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:57:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[09:58:18] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[09:58:34] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[10:01:36] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.46, 21.77, 19.42
[10:03:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[10:03:34] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.78, 19.26, 18.78
[10:03:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.17, 3.29, 3.99
[10:05:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[10:07:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:09:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 3.49, 3.80
[10:11:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.27, 3.39, 3.75
[10:13:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.37, 4.26, 4.03
[10:18:30] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[10:21:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.78, 3.25, 3.76
[10:21:56] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[10:24:13] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:25:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.49, 4.14, 3.96
[10:26:10] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001083999872 secs
[10:27:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.71, 3.52, 3.73
[10:31:44] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.07, 2.78, 3.37
[10:37:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:42:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:44:50] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures
[10:47:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[10:47:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:48:33] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.49, 4.65, 3.92
[10:49:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[10:52:22] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.11, 3.55, 3.61
[10:52:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:56:12] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.53, 2.73, 3.24
[10:57:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:00:02] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.56, 3.33, 3.44
[11:02:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:03:51] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.63, 2.59, 3.19
[11:07:29] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.76, 21.39, 19.50
[11:09:28] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.02, 20.29, 19.33
[11:13:22] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.89, 22.21, 20.29
[11:15:22] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax]
[11:21:15] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.98, 21.26
[11:25:11] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.27, 22.87, 21.66
[11:29:42] <icinga-miraheze>	 PROBLEM - kalons-reverie.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kalons-reverie.com' expires in 15 day(s) (Thu 15 Aug 2024 10:59:32 AM GMT +0000).
[11:29:54] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/67ea193b7ec4...23e205deb8e7
[11:29:56] <botifico>	 [02ssl] 07WikiTideSSLBot 0323e205d - Bot: Update SSL cert for kalons-reverie.com
[11:31:05] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.38, 24.94, 22.74
[11:33:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[11:35:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[11:37:00] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.72, 22.19, 22.53
[11:44:26] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[11:50:47] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.01, 22.60, 22.17
[11:56:27] <icinga-miraheze>	 PROBLEM - www.hebammenwiki.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.hebammenwiki.de' expires in 15 day(s) (Thu 15 Aug 2024 11:31:21 AM GMT +0000).
[11:56:39] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/23e205deb8e7...c0893637ca03
[11:56:41] <botifico>	 [02ssl] 07WikiTideSSLBot 03c089363 - Bot: Update SSL cert for www.hebammenwiki.de
[11:58:40] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.20, 23.43, 23.07
[11:58:54] <icinga-miraheze>	 RECOVERY - kalons-reverie.com - LetsEncrypt on sslhost is OK: OK - Certificate 'kalons-reverie.com' will expire on Mon 28 Oct 2024 10:29:48 AM GMT +0000.
[12:08:30] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.31, 21.07, 21.34
[12:10:28] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.52, 21.17, 21.36
[12:12:27] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.94, 23.55, 22.20
[12:14:41] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 19.72, 15.60
[12:20:38] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.25, 22.78, 18.53
[12:21:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[12:23:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[12:24:37] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.70, 19.38, 18.10
[12:25:29] <icinga-miraheze>	 RECOVERY - www.hebammenwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hebammenwiki.de' will expire on Mon 28 Oct 2024 10:56:33 AM GMT +0000.
[12:36:05] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.54, 22.97, 23.93
[12:38:03] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.22, 25.56, 24.77
[12:40:29] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 20.79, 19.13
[12:41:13] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 384.28 ms
[12:41:44] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 6 backends are down. mw161 mw162 mw171 mw172 mw181 mw182
[12:43:16] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 222.25 ms
[12:43:41] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[12:48:26] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.19, 19.55, 19.81
[12:53:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.89, 19.90, 23.35
[12:59:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.37, 20.66, 22.33
[13:01:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.26, 18.86, 21.48
[13:07:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 22.23, 21.98
[13:09:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.50, 21.04, 21.60
[13:13:52] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.08, 16.56, 19.58
[13:53:56] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:55:53] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0004495084286 secs
[13:56:17] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 21.86, 19.72
[13:58:15] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.63, 19.12, 18.99
[14:05:37] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[14:07:34] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[14:22:38] <MirahezeLSBot>	 !log [salt-user@mwtask181] Renamed federalrepublicoftomninkiawiki to tharkestanwiki using renamewiki.py
[14:22:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:38:56] <MirahezeRelay>	 <originalauthority> salt ftw
[14:39:16] <MirahezeRelay>	 <pixldev> https://tenor.com/view/salty-salt-bae-gif-7529894
[14:47:05] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax]
[14:55:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[14:57:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[14:57:42] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[15:02:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:37:48] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[15:39:45] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[15:39:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 19.92, 15.39
[15:41:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.28, 20.99, 16.38
[15:43:52] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.70, 19.72, 16.45
[15:45:47] <MirahezeRelay>	 <originalauthority, replying to pixldev> spiceh
[15:53:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.27, 21.64, 18.70
[16:00:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.06, 20.90, 15.76
[16:04:32] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.05, 21.40, 15.43
[16:06:15] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 21.47, 16.41
[16:08:55] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.95, 22.48, 16.76
[16:10:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.43, 22.51, 17.21
[16:14:32] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.25, 23.08, 19.72
[16:14:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 16.11, 20.98, 17.92
[16:16:03] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 14.19, 22.21, 20.31
[16:16:32] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.11, 21.17, 19.13
[16:16:32] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.03, 19.04, 18.64
[16:16:45] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 10.62, 17.59, 17.08
[16:18:01] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.07, 19.50, 19.55
[16:18:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.34, 23.96, 23.22
[16:18:26] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.85, 18.15, 18.25
[16:28:05] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.70, 17.57, 19.94
[16:29:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[16:31:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[16:39:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.64, 21.37, 23.89
[16:41:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.90, 23.54, 24.35
[16:52:00] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.63, 21.12, 19.14
[16:55:59] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.27, 20.38, 19.33
[16:59:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.11, 22.53, 20.56
[17:01:55] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.94, 20.06, 19.91
[17:02:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:03:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:05:26] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.233 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:05:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.50, 4.57, 2.00
[17:07:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:09:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.31, 3.51, 2.12
[17:11:32] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[17:11:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.20, 3.04, 2.12
[17:13:28] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.00183147192 secs
[17:15:45] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.78, 3.69, 2.67
[17:17:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.80, 3.19, 2.62
[17:19:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[17:21:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[17:24:54] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:25:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.49, 4.74, 3.43
[17:26:59] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 9.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:29:43] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 20.22, 17.99
[17:29:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.58, 3.68, 3.29
[17:30:15] <icinga-miraheze>	 PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[17:31:42] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.97, 20.60, 18.37
[17:31:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.43, 3.16, 3.17
[17:33:41] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.22, 19.34, 18.19
[17:35:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.45, 3.94, 3.56
[17:37:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.30, 3.20, 3.35
[17:41:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[17:41:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 5.57, 4.25
[17:42:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:43:26] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.059 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:47:34] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.49, 22.62, 19.12
[17:47:38] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:47:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:48:56] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output
[17:49:20] <icinga-miraheze>	 PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: No response from DNS 10.0.17.136
[17:49:33] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.74, 21.22, 19.08
[17:51:33] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.18, 19.04, 18.50
[17:52:06] <botifico>	 [02dns] 07MacFan4000 opened pull request 03#539: fix syntax error - 13https://github.com/miraheze/dns/pull/539
[17:52:42] <botifico>	 [02dns] 07redbluegreenhat closed pull request 03#539: fix syntax error - 13https://github.com/miraheze/dns/pull/539
[17:52:45] <botifico>	 [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a31bb03dac99...595471876d7c
[17:52:46] <botifico>	 [02dns] 07MacFan4000 035954718 - fix syntax error (#539)
[17:52:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:54:56] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:55:20] <icinga-miraheze>	 RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.042 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[17:56:19] <MirahezeRelay>	 <rhinosf1, replying to originalauthority> I need to fix that logging
[17:57:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.53, 3.54, 3.94
[17:57:49] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:57:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:59:28] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.90, 21.84, 19.43
[18:00:23] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:01:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.30, 3.94, 3.95
[18:02:07] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[18:02:20] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003442466259 secs
[18:02:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:03:27] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.51, 23.07, 20.69
[18:03:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.07, 3.54, 3.83
[18:05:26] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.21, 20.40, 20.01
[18:07:32] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sensors failed with return code 1.-> /usr/sbin/ipmi-sensors was executed with the following parameters: [HIDDEN]sudo /usr/sbin/ipmi-sensors --exclude-sensor-types Drive_Slot,Entity_Presence --quiet-cache --sdr-cache-recreate --interpret-oem-data --output-senso
[18:07:33] <icinga-miraheze>	 ignore-not-available-sensors --output-sensor-thresholds
[18:07:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:09:34] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[18:13:22] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.95, 20.89, 20.12
[18:13:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.31, 3.04, 3.40
[18:14:33] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[18:15:21] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.10, 18.98, 19.51
[18:19:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.72, 4.19, 3.64
[18:21:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.65, 3.77, 3.57
[18:25:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.66, 3.00, 3.33
[18:25:51] <MirahezeRelay>	 <originalauthority> Why what's wrong with it
[18:26:11] <RhinosF1>	 It saying salt-user and not the human
[18:26:38] <RhinosF1>	 It's technically true but human in there would be better
[18:27:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:29:13] <MirahezeRelay>	 <originalauthority> i love salt
[18:29:30] <RhinosF1>	 Ye salt is fine
[18:29:38] <RhinosF1>	 It's a rename script problem tbh
[18:29:42] <RhinosF1>	 Salt is great
[18:29:51] <MirahezeRelay>	 <originalauthority> I've just switched from puppet
[18:29:53] <MirahezeRelay>	 <originalauthority> puppet is horrid
[18:30:14] <RhinosF1>	 Puppet is weird
[18:30:26] <botifico>	 [02mediawiki-repos] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/355e5094abba...25ac2e1dc1c0
[18:30:27] <botifico>	 [02mediawiki-repos] 07The-Voidwalker 0325ac2e1 - change CountDownClock to master branch
[18:31:26] <RhinosF1>	 https://bash.toolforge.org/quip/AU_3cMwj1oXzWjit5obk
[18:32:29] <RhinosF1>	 https://bash.toolforge.org/search?p=0&q=Puppet in general
[18:32:34] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.12, 5.04, 4.04
[18:36:23] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.62, 3.95, 3.82
[18:37:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:37:50] <icinga-miraheze>	 PROBLEM - wiki.fediverse.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fediverse.kr could not be found
[18:40:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.64, 3.60, 3.67
[18:41:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.48, 20.12, 23.65
[18:42:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:44:02] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.89, 3.54, 3.68
[18:44:10] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_42 CountDownClock]
[18:45:25] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:45:38] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_42 CountDownClock]
[18:46:11] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[18:47:20] <MirahezeLSBot>	 !log [void@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to all
[18:47:21] <MirahezeLSBot>	 !log [void@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to all - SUCCESS in 1s
[18:47:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:47:38] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:47:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:47:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:47:52] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.78, 4.91, 4.12
[18:47:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.85, 21.71, 23.09
[18:48:22] <MirahezeLSBot>	 !log [void@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to all
[18:48:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:48:35] <MirahezeLSBot>	 !log [void@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to all - SUCCESS in 13s
[18:48:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:51:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.61, 23.19, 23.44
[18:53:14] <botifico>	 [02dns-check-action] 07redbluegreenhat pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns-check-action/commit/f275b5620e7e
[18:53:17] <botifico>	 [02dns-check-action] 07redbluegreenhat 03f275b56 - Push license and README
[18:53:18] <botifico>	 [02dns-check-action] 07redbluegreenhat created branch 03master - 13https://github.com/miraheze/dns-check-action
[18:53:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[18:53:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 23.38, 23.46
[18:55:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[18:56:25] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_42 CountDownClock]
[18:57:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:57:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.40, 23.23, 23.46
[18:59:09] <MirahezeRelay>	 <originalauthority> salt is good
[18:59:13] <MirahezeRelay>	 <originalauthority> salt is easy to read
[19:01:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.79, 3.01, 3.89
[19:02:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:03:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 21.83, 22.64
[19:04:20] <MirahezeRelay>	 <originalauthority> @rhinosf1  why doesn't mwdeploy maintain symlinks?
[19:04:46] <MirahezeRelay>	 <originalauthority> I'm confused because miraheze symlinks /config/LocalSettings.php > /{version}/LocalSettings.php to make MediaWiki happy but mwdeploy doesn't maintain that?
[19:05:43] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.05, 1.40, 3.02
[19:07:11] <MirahezeRelay>	 <rhinosf1, replying to originalauthority> Not sure
[19:07:26] <MirahezeRelay>	 <rhinosf1> I don't think there was ever a design to not
[19:07:29] <MirahezeRelay>	 <rhinosf1> So a bug?
[19:07:46] <MirahezeRelay>	 <originalauthority> maybe
[19:07:51] <MirahezeRelay>	 <originalauthority> I think the symlink is done in puppet
[19:07:58] <MirahezeRelay>	 <originalauthority> so maybe just relying on puppet to ensure its always there?
[19:08:22] <MirahezeRelay>	 <originalauthority> but that confuses me even more because doesn't the rsync delete any files that aren't present in /mediawiki-staging when it sends it over
[19:08:54] <MirahezeRelay>	 <originalauthority> i love python now its wonderful
[19:09:18] <MirahezeRelay>	 <rhinosf1> I can't remember tbh if we have delete set on rsync
[19:14:15] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:27:46] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cloud17
[19:27:59] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on bast161
[19:28:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:28:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:28:13] <MirahezeLSBot>	 !log [void@test151] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to test151
[19:28:14] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, bind9-libs, and dnsutils on eventgate181
[19:28:15] <MirahezeLSBot>	 !log [void@test151] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'CountDownClock'} to test151 - SUCCESS in 1s
[19:28:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:28:25] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[19:28:27] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cp36
[19:28:27] <icinga-miraheze>	 RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 101 packages available for upgrade (0 critical updates).
[19:28:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:28:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:28:41] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on bots171
[19:28:48] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 20.35, 17.29
[19:28:54] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cloud15
[19:28:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:28:57] <icinga-miraheze>	 RECOVERY - bast161 APT on bast161 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:28:58] <icinga-miraheze>	 RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 109 packages available for upgrade (0 critical updates).
[19:29:04] <icinga-miraheze>	 RECOVERY - bots171 APT on bots171 is OK: APT OK: 63 packages available for upgrade (0 critical updates).
[19:29:06] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on jobchron171
[19:29:16] <icinga-miraheze>	 RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 62 packages available for upgrade (0 critical updates).
[19:29:18] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cloud16
[19:29:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:29:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:29:32] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on db151
[19:29:38] <icinga-miraheze>	 RECOVERY - db151 APT on db151 is OK: APT OK: 60 packages available for upgrade (0 critical updates).
[19:29:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:29:44] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on db161
[19:29:46] <icinga-miraheze>	 RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 45 packages available for upgrade (0 critical updates).
[19:29:57] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on matomo151
[19:29:57] <icinga-miraheze>	 RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 101 packages available for upgrade (0 critical updates).
[19:30:11] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on db171
[19:30:11] <icinga-miraheze>	 RECOVERY - cp36 APT on cp36 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:30:14] <icinga-miraheze>	 RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 63 packages available for upgrade (0 critical updates).
[19:30:23] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on graylog161
[19:30:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:30:36] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, bind9-libs, and dnsutils on graphite151
[19:30:47] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.01, 19.53, 17.36
[19:30:48] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cp37
[19:31:00] <icinga-miraheze>	 RECOVERY - db161 APT on db161 is OK: APT OK: 59 packages available for upgrade (0 critical updates).
[19:31:03] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on bast181
[19:31:05] <icinga-miraheze>	 RECOVERY - db171 APT on db171 is OK: APT OK: 59 packages available for upgrade (0 critical updates).
[19:31:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:31:17] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on db181
[19:31:28] <icinga-miraheze>	 RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 56 packages available for upgrade (0 critical updates).
[19:31:32] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cloud18
[19:31:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:31:46] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on db182
[19:32:01] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, bind9-libs, and dnsutils on kafka181
[19:32:09] <icinga-miraheze>	 RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 40 packages available for upgrade (0 critical updates).
[19:32:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:32:19] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cp26
[19:32:26] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[19:32:26] <icinga-miraheze>	 RECOVERY - db181 APT on db181 is OK: APT OK: 60 packages available for upgrade (0 critical updates).
[19:32:29] <icinga-miraheze>	 RECOVERY - bast181 APT on bast181 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[19:32:31] <icinga-miraheze>	 RECOVERY - cp26 APT on cp26 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:32:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:32:35] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on ldap171
[19:32:45] <icinga-miraheze>	 RECOVERY - cp37 APT on cp37 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:32:47] <icinga-miraheze>	 RECOVERY - db182 APT on db182 is OK: APT OK: 60 packages available for upgrade (0 critical updates).
[19:32:53] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cp27
[19:33:05] <icinga-miraheze>	 RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 102 packages available for upgrade (0 critical updates).
[19:33:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:33:08] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mem161
[19:33:22] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mem151
[19:33:22] <icinga-miraheze>	 RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 38 packages available for upgrade (0 critical updates).
[19:33:37] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mon181
[19:33:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:33:52] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cp41
[19:34:05] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on os151
[19:34:09] <icinga-miraheze>	 RECOVERY - mem161 APT on mem161 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:34:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:34:19] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on os162
[19:34:19] <icinga-miraheze>	 RECOVERY - cp27 APT on cp27 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:34:23] <icinga-miraheze>	 RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:34:29] <icinga-miraheze>	 RECOVERY - mon181 APT on mon181 is OK: APT OK: 58 packages available for upgrade (0 critical updates).
[19:34:32] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 20.54, 16.75
[19:34:33] <icinga-miraheze>	 RECOVERY - os162 APT on os162 is OK: APT OK: 54 packages available for upgrade (0 critical updates).
[19:34:34] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mwtask181
[19:34:44] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.23, 24.82, 20.02
[19:34:47] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on os161
[19:34:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:35:02] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw161
[19:35:12] <icinga-miraheze>	 RECOVERY - mem151 APT on mem151 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:35:19] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw172
[19:35:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:35:26] <icinga-miraheze>	 RECOVERY - os151 APT on os151 is OK: APT OK: 54 packages available for upgrade (0 critical updates).
[19:35:29] <icinga-miraheze>	 RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 63 packages available for upgrade (0 critical updates).
[19:35:32] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on prometheus151
[19:35:36] <icinga-miraheze>	 RECOVERY - cp41 APT on cp41 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:35:51] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on phorge171
[19:35:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:36:05] <icinga-miraheze>	 RECOVERY - os161 APT on os161 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[19:36:06] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mwtask171
[19:36:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 19.38, 15.43
[19:36:16] <icinga-miraheze>	 RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 63 packages available for upgrade (0 critical updates).
[19:36:24] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181
[19:36:24] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on ns1
[19:36:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:36:39] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw162
[19:36:47] <icinga-miraheze>	 RECOVERY - mw161 APT on mw161 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[19:36:47] <icinga-miraheze>	 RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:36:54] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw151
[19:37:00] <icinga-miraheze>	 RECOVERY - mw172 APT on mw172 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[19:37:09] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw152
[19:37:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:37:16] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.15, 20.07, 15.51
[19:37:22] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 20.98, 16.78
[19:37:23] <icinga-miraheze>	 RECOVERY - mw151 APT on mw151 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[19:37:24] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw171
[19:37:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:37:41] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on cp51
[19:37:51] <icinga-miraheze>	 RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 54 packages available for upgrade (0 critical updates).
[19:37:57] <icinga-miraheze>	 RECOVERY - cp51 APT on cp51 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:37:57] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw182
[19:37:58] <icinga-miraheze>	 RECOVERY - mw162 APT on mw162 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[19:38:06] <icinga-miraheze>	 RECOVERY - mw182 APT on mw182 is OK: APT OK: 66 packages available for upgrade (0 critical updates).
[19:38:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.30, 20.23, 16.23
[19:38:09] <icinga-miraheze>	 RECOVERY - mw152 APT on mw152 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[19:38:10] <icinga-miraheze>	 RECOVERY - ns1 APT on ns1 is OK: APT OK: 51 packages available for upgrade (0 critical updates).
[19:38:10] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on rdb151
[19:38:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:38:20] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[19:38:28] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on mw181
[19:38:42] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on reports171
[19:38:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 21.61, 16.72
[19:38:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:38:56] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on puppet181
[19:39:10] <icinga-miraheze>	 RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:39:11] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on test151
[19:39:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:39:23] <icinga-miraheze>	 RECOVERY - mw171 APT on mw171 is OK: APT OK: 65 packages available for upgrade (0 critical updates).
[19:39:23] <icinga-miraheze>	 RECOVERY - mw181 APT on mw181 is OK: APT OK: 66 packages available for upgrade (0 critical updates).
[19:39:25] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftobject151
[19:39:27] <icinga-miraheze>	 RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:39:28] <icinga-miraheze>	 RECOVERY - reports171 APT on reports171 is OK: APT OK: 64 packages available for upgrade (0 critical updates).
[19:39:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:39:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:39:40] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftobject161
[19:39:52] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 75 packages available for upgrade (0 critical updates).
[19:39:57] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftac171
[19:40:06] <icinga-miraheze>	 RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 61 packages available for upgrade (0 critical updates).
[19:40:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.81, 22.96, 17.72
[19:40:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:40:13] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftobject171
[19:40:15] <icinga-miraheze>	 RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 41 packages available for upgrade (0 critical updates).
[19:40:27] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftproxy161
[19:40:34] <icinga-miraheze>	 RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 41 packages available for upgrade (0 critical updates).
[19:40:45] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftobject181
[19:40:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:40:58] <icinga-miraheze>	 RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 53 packages available for upgrade (0 critical updates).
[19:41:01] <MirahezeLSBot>	 !log [void@puppet181] Upgraded packages bind9-host, bind9-dnsutils, and bind9-libs on swiftproxy171
[19:41:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:41:38] <icinga-miraheze>	 RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:42:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.85, 23.56, 18.63
[19:42:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:42:12] <icinga-miraheze>	 RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 52 packages available for upgrade (0 critical updates).
[19:42:22] <icinga-miraheze>	 RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 55 packages available for upgrade (0 critical updates).
[19:42:45] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 15.81, 20.64, 17.55
[19:42:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:43:16] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.80, 20.38, 17.61
[19:43:22] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.04, 19.68, 17.97
[19:43:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:43:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:44:06] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.80, 19.61, 17.79
[19:44:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:44:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:44:32] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 9.93, 20.17, 19.77
[19:44:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:44:45] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 11.54, 17.08, 16.60
[19:44:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:45:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:45:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:45:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:45:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:39] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 11.06, 19.81, 21.45
[19:46:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:46:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:47:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:47:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:47:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:47:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:47:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:48:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:48:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:50:38] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.93, 16.99, 19.92
[19:57:49] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:00:26] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[20:02:49] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:31:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.59, 21.60, 23.79
[20:35:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[20:35:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.64, 24.31, 24.33
[20:37:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[21:05:43] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[21:07:39] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002765357494 secs
[21:13:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.77, 22.46, 23.80
[21:17:35] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:17:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 23.78, 24.07
[21:35:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.40, 22.70, 23.90
[21:37:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 24.34, 24.40
[21:39:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.69, 21.46, 23.27
[21:43:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 22.09, 23.05
[21:44:53] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:45:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.10, 21.58, 22.74
[21:49:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 22.55, 22.79
[21:50:23] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:51:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.18, 20.96, 22.19
[21:52:19] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008991360664 secs
[21:53:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 22.81, 22.67
[21:55:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.58, 22.24, 22.49
[21:59:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.67, 23.96, 23.07
[22:01:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.45, 23.59, 23.09
[22:03:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 24.23, 23.41
[22:09:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.72, 22.45, 23.04
[22:11:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 23.31, 23.26
[22:13:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.58, 22.46, 22.94
[22:15:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 22.57, 22.89
[22:19:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[22:21:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[22:27:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.57, 23.93, 23.93
[22:35:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 22.70, 23.07
[23:02:05] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.25, 20.48, 17.57
[23:02:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:07:21] <icinga-miraheze>	 PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:07:30] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[23:07:44] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.87, 4.22, 2.02
[23:09:43] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.63, 3.58, 2.03
[23:11:19] <icinga-miraheze>	 RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[23:11:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[23:11:35] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.752 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[23:11:44] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.36, 3.23, 2.11
[23:12:05] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.30, 19.39, 18.92
[23:12:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:13:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]
[23:22:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:27:25] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.33, 4.01, 3.11
[23:27:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:31:15] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.50, 3.51, 3.13
[23:32:25] <botifico>	 [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/mypy-1.11.1 - 13https://github.com/miraheze/python-functions
[23:32:26] <botifico>	 [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/mypy-1.11.1 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/2d15c8e8a20a
[23:32:28] <botifico>	 [02python-functions] 07dependabot[bot] 032d15c8e - Bump mypy from 1.10.1 to 1.11.1 in /.github
[23:32:31] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#51: Bump mypy from 1.10.1 to 1.11.1 in /.github - 13https://github.com/miraheze/python-functions/pull/51
[23:32:34] <botifico>	 [02python-functions] 07dependabot[bot] labeled pull request 03#51: Bump mypy from 1.10.1 to 1.11.1 in /.github - 13https://github.com/miraheze/python-functions/pull/51
[23:32:35] <botifico>	 [02python-functions] 07dependabot[bot] opened pull request 03#51: Bump mypy from 1.10.1 to 1.11.1 in /.github - 13https://github.com/miraheze/python-functions/pull/51
[23:32:37] <botifico>	 [02python-functions] 07dependabot[bot] commented on pull request 03#45: Bump mypy from 1.10.1 to 1.11.0 in /.github - 13https://github.com/miraheze/python-functions/pull/45#issuecomment-2259367810
[23:32:40] <botifico>	 [02python-functions] 07dependabot[bot] closed pull request 03#45: Bump mypy from 1.10.1 to 1.11.0 in /.github - 13https://github.com/miraheze/python-functions/pull/45
[23:32:42] <botifico>	 [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/mypy-1.11.0 - 13https://github.com/miraheze/python-functions
[23:32:44] <botifico>	 [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/mypy-1.11.0
[23:32:46] <botifico>	 [02python-functions] 07coderabbitai[bot] commented on pull request 03#51: Bump mypy from 1.10.1 to 1.11.1 in /.github - 13https://github.com/miraheze/python-functions/pull/51#issuecomment-2259367854
[23:32:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:33:10] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.92, 5.12, 3.78
[23:35:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.34, 22.52, 23.94
[23:36:59] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.40, 3.47, 3.42
[23:37:07] <miraheze-github>	 miraheze/python-functions - dependabot[bot] the build passed.
[23:37:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:37:52] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.65, 24.29, 24.40
[23:40:54] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.58, 3.58, 3.46
[23:42:50] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.33, 2.95, 3.23
[23:47:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:48:36] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.44, 3.89, 3.58
[23:52:25] <icinga-miraheze>	 RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.09, 3.16, 3.33
[23:52:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:57:31] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all
[23:57:49] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:58:12] <icinga-miraheze>	 PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.83, 4.41, 3.80
[23:59:33] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [441 system event log (SEL) entries present]