[00:00:00] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[00:00:01] <MirahezeLSBot>	 !log [www-data@test151] Began automatic backing up
[00:00:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:00:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:07:27] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 56957MiB (12% inode=99%);
[00:07:33] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57221MiB (12% inode=99%);
[00:08:13] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 56374MiB (12% inode=99%);
[00:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756685457037[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756685457037
[00:27:42] <icinga-miraheze>	 PROBLEM - test151 Current Load on test151 is CRITICAL: LOAD CRITICAL - total load average: 25.27, 21.33, 13.82
[00:51:12] <MirahezeRelay>	 <orduin> I've done a bit of digging into the dns issue, seems to be caused by powerdns throttling some requests seemingly due to timeouts
[00:53:25] <BlankEclair>	 job queue bigger than my depression smh
[01:10:45] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.01, 20.51, 19.83
[01:12:04] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:12:57] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:13:01] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:13:30] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:13:52] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[01:14:28] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[01:14:28] <icinga-miraheze>	 PROBLEM - test151 Redis Process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:14:35] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:15:26] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[01:15:50] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 0.079 second response time
[01:16:05] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.073 second response time
[01:16:26] <icinga-miraheze>	 RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server'
[01:16:31] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.0003753602505 secs
[01:17:03] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[01:17:08] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.087 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[01:18:41] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.96, 19.83, 19.89
[01:20:12] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[01:20:26] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:21:40] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:22:20] <icinga-miraheze>	 PROBLEM - test151 APT on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:23:26] <MirahezeRelay>	 <orduin> Heck is going on on test151
[01:23:40] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.894 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[01:24:48] <BlankEclair>	 everybody do the flop
[01:25:46] <MirahezeRelay>	 <orduin> Whole thing has frozen up, might have to force a reboot
[01:28:15] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[01:28:35] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 4.157 second response time
[01:28:37] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.228 second response time
[01:30:32] <icinga-miraheze>	 PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:31:00] <BlankEclair>	 it's not fully frozen, just... extremely slow
[01:31:30] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[01:31:56] <icinga-miraheze>	 PROBLEM - test151 poolcounter process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:32:08] <MirahezeRelay>	 <orduin> The load average is 116 and appears to be climbing
[01:32:51] <BlankEclair>	 i got close with debug1: Offering public key: id_miraheze.claire@diamondprism ED25519-SK SHA256:yWZoqCzMn4/9F4UrMh8+ilscSP3zFiJ17UlIvPLZk3g explicit authenticator agent
[01:32:56] <BlankEclair>	 but connecttimeout got me in the end
[01:32:57] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[01:32:58] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:33:07] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:33:30] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:33:30] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:33:31] <icinga-miraheze>	 PROBLEM - test151 Redis Process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:34:33] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:35:50] <icinga-miraheze>	 PROBLEM - test151 php-fpm on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:36:15] <icinga-miraheze>	 RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd'
[01:36:44] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[01:37:06] <BlankEclair>	 ughhhhhhhhhhhhhhhhh i need a better ssh agent
[01:37:17] <MirahezeRelay>	 <orduin> Huh, there seems to be 63 running dumpBackup scripts on test151
[01:37:29] <BlankEclair>	 that'd do it
[01:37:56] <BlankEclair>	 who'd win: ssh or the race against the bell
[01:41:11] <icinga-miraheze>	 PROBLEM - test151 poolcounter process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[01:42:12] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:42:15] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: NTP CRITICAL: No response from NTP server
[01:43:16] <icinga-miraheze>	 PROBLEM - test151 ferm_active on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:43:16] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:43:41] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:44:31] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[01:44:33] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter
[01:44:34] <icinga-miraheze>	 RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2'
[01:44:34] <icinga-miraheze>	 RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server'
[01:45:38] <icinga-miraheze>	 RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd'
[01:45:39] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[01:45:41] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 9.680 second response time
[01:45:41] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[01:45:41] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[01:45:41] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[01:47:32] <BlankEclair>	 bleh i gtg
[01:47:42] <MirahezeRelay>	 <orduin> I'm going to reboot, not much use for us as is
[01:47:51] <icinga-miraheze>	 RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full
[01:48:18] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[01:48:23] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[01:48:36] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures
[01:48:42] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -8.165836334e-06 secs
[01:49:45] <icinga-miraheze>	 RECOVERY - test151 Current Load on test151 is OK: LOAD OK - total load average: 1.07, 0.32, 0.11
[01:49:53] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 62 packages available for upgrade (0 critical updates).
[01:50:11] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.036 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[01:51:25] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.659 second response time
[01:55:11] <MirahezeLSBot>	 !log [void@test151] reboot after server got locked up from starting 63 backup scripts for some reason
[01:55:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:56:20] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.24, 20.43, 20.29
[01:58:19] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.57, 19.89, 20.11
[02:04:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.58, 20.93, 20.47
[02:07:22] <MirahezeRelay>	 <orduin> [1/2] The cause of the spotty DNS stuff is probably <https://github.com/miraheze/puppet/commit/63dd30230ee362660e6a8eb6f0caea0dea9e46bb>, since powerDNS attempts to resolve using IPv4 first and most of our servers can't connect to IPv4, the requests time out, causing powerDNS to throttle the request. Reversing it isn't really an option, but we can probably just forward all internal DNS to o
[02:07:22] <MacFanBot>	 [GitHub] [miraheze/puppet] BlankEclair: dns: allow powerdns to use ipv4 to resolve domains (#4478)… | 2 changes in 1 file | Authored 9 days, 13 hours ago | Committed 9 days, 13 hours ago
[02:07:22] <MirahezeRelay>	 <orduin> [2/2] ur dns servers to resolve since they have IPv4 connectivity.
[02:10:37] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Testing pdns solutions --Void, last run 7 minutes ago with 0 failures
[02:11:42] <icinga-miraheze>	 PROBLEM - test151 Current Load on test151 is WARNING: LOAD WARNING - total load average: 23.21, 20.15, 12.56
[02:13:42] <icinga-miraheze>	 PROBLEM - test151 Current Load on test151 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 21.49, 13.98
[02:14:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.66, 20.16, 20.35
[02:18:37] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[02:20:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.11, 20.95, 20.60
[02:20:44] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.0001040101051 secs
[02:26:27] <MirahezeRelay>	 <orduin> Of course gdns doesn't do recursive lookups
[02:28:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.45, 19.99, 20.30
[02:31:22] <MirahezeRelay>	 <orduin> Also test151 seems to have done the way too many backups at once thing again
[02:31:24] <icinga-miraheze>	 PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 21.08, 17.67, 15.32
[02:31:41] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 20.57, 20.18, 19.32
[02:32:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.89, 20.47, 20.43
[02:35:41] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.52, 20.21, 19.57
[02:41:24] <icinga-miraheze>	 RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 14.05, 19.04, 17.74
[02:42:36] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 19.62, 20.41, 19.93
[02:42:41] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[02:44:35] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 19.51, 20.10, 19.87
[03:11:32] <icinga-miraheze>	 PROBLEM - farthestfrontier.wiki - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:20:04] <MirahezeRelay>	 <blankeclair, replying to orduin> fml
[03:20:25] <MirahezeRelay>	 <blankeclair> wait, shouldn't it insta fail- oh wait, dns is udp...
[03:20:37] <MirahezeRelay>	 <blankeclair> fml², do we need to dynamic select ipv4?
[03:24:57] <MirahezeRelay>	 <orduin> IKR, big headache. I'm thinking of turning bast* into resolvers for everything in fsslc.wtnet
[03:30:27] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[03:32:22] <MirahezeRelay>	 <orduin> BTW, anyone know where the /etc/resolv.conf comes from for ns*? I don't think it's in puppet.
[03:37:32] <icinga-miraheze>	 PROBLEM - test151 memcached on test151 is CRITICAL: connect to address 10.0.15.118 and port 11211: Connection refused
[03:41:22] <icinga-miraheze>	 PROBLEM - farthestfrontier.wiki - Cloudflare on sslhost is WARNING: WARNING - Certificate 'farthestfrontier.wiki' expires in 22 day(s) (Tue 23 Sep 2025 04:23:30 PM GMT +0000).
[04:05:51] <icinga-miraheze>	 PROBLEM - test151 APT on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:08:24] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 62 packages available for upgrade (0 critical updates).
[04:10:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.64, 20.01, 20.37
[04:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756699857070[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756699857070
[04:11:46] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 21073MiB (9% inode=93%);
[04:14:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.90, 20.30, 20.39
[04:16:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.45, 20.09, 20.31
[04:16:59] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:16:59] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:17:06] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received
[04:17:07] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:17:15] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:17:24] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:17:31] <icinga-miraheze>	 PROBLEM - test151 APT on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:19:28] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[04:19:51] <icinga-miraheze>	 PROBLEM - test151 poolcounter process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:22:14] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:24:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.89, 20.54, 20.49
[04:27:21] <icinga-miraheze>	 PROBLEM - test151 php-fpm on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:27:25] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:27:51] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:27:54] <icinga-miraheze>	 PROBLEM - test151 Redis Process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:28:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.97, 20.19, 20.38
[04:28:49] <icinga-miraheze>	 PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:37] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:42] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:47] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:49] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:50] <icinga-miraheze>	 PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:50] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:52] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:52] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:53] <icinga-miraheze>	 PROBLEM - test151 ferm_active on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:53] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:29:55] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:30:02] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:30:53] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:30:53] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:30:54] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:31:09] <icinga-miraheze>	 RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full
[04:32:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify
[04:32:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics
[04:32:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks
[04:32:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table
[04:32:26] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[04:32:27] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash
[04:32:27] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[04:32:28] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[04:32:28] <icinga-miraheze>	 RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server'
[04:32:28] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps
[04:32:29] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[04:32:29] <icinga-miraheze>	 RECOVERY - test151 Check unit status of prometheus_jobqueue_stats on test151 is OK: OK: Status of the systemd unit prometheus_jobqueue_stats
[04:32:29] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.0009434819221 secs
[04:32:29] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[04:32:31] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[04:32:36] <icinga-miraheze>	 RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd'
[04:33:13] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[04:33:13] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter
[04:33:13] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[04:34:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.35, 20.44, 20.36
[04:34:53] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.502 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[04:36:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.19, 20.11, 20.24
[04:42:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.93, 20.31, 20.26
[04:43:29] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:43:46] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 58731MiB (26% inode=93%);
[04:44:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.78, 19.76, 20.07
[04:44:30] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:45:39] <icinga-miraheze>	 PROBLEM - test151 poolcounter process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:45:39] <icinga-miraheze>	 PROBLEM - test151 Redis Process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:45:40] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.002428203821 secs
[04:45:41] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:46:33] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:46:37] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:46:39] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:46:47] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:46:56] <icinga-miraheze>	 PROBLEM - test151 ferm_active on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:03] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:08] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:21] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:32] <icinga-miraheze>	 PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:46] <icinga-miraheze>	 PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:50] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:51] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:51] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:53] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:47:56] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:48:09] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[04:49:25] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics
[04:49:26] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[04:50:58] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[04:51:00] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify
[04:52:24] <icinga-miraheze>	 RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 2 % full
[04:52:51] <icinga-miraheze>	 RECOVERY - test151 Check unit status of prometheus_jobqueue_stats on test151 is OK: OK: Status of the systemd unit prometheus_jobqueue_stats
[04:52:51] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[04:52:51] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[04:52:51] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps
[04:52:52] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table
[04:52:52] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter
[04:52:52] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash
[04:52:53] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 8.099 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[04:53:03] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 0.439 second response time
[04:57:15] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:57:17] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:57:25] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[04:57:30] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:58:20] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[04:58:29] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[04:58:40] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:58:53] <icinga-miraheze>	 PROBLEM - test151 ferm_active on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[04:59:19] <icinga-miraheze>	 PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:00:00] <icinga-miraheze>	 PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:00:00] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:00:04] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:00:14] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:00:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.17, 20.63, 20.40
[05:00:24] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:00:26] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:01:19] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:01:20] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:01:27] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:01:27] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:01:33] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:05:33] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps
[05:05:36] <icinga-miraheze>	 RECOVERY - test151 Check unit status of prometheus_jobqueue_stats on test151 is OK: OK: Status of the systemd unit prometheus_jobqueue_stats
[05:07:15] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[05:07:18] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[05:07:41] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[05:08:33] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[05:08:33] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash
[05:08:33] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table
[05:08:33] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter
[05:08:34] <icinga-miraheze>	 RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd'
[05:09:30] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics
[05:09:30] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[05:09:30] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks
[05:09:31] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[05:10:00] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[05:11:48] <icinga-miraheze>	 RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server'
[05:11:48] <icinga-miraheze>	 RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2'
[05:11:49] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[05:11:49] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 1.785 second response time. test151.fsslc.wtnet returns 10.0.15.118
[05:13:05] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[05:14:19] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 4.608 second response time
[05:14:46] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.0001631975174 secs
[05:16:54] <icinga-miraheze>	 PROBLEM - test151 poolcounter process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:16:59] <icinga-miraheze>	 PROBLEM - test151 Redis Process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:17:39] <MirahezeRelay>	 <cosmicalpha> @orduin cant we just add hieradata to make adding ipv4 connectivity only via bast rather than all other servers that dont have it so pdns only connects to ipv4 when it can?
[05:17:53] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:17:54] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:18:25] <MirahezeRelay>	 <orduin, replying to cosmicalpha> That's what I'm setting up now
[05:18:38] <MirahezeRelay>	 <cosmicalpha> oh nice thanks!
[05:18:41] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[05:18:58] <MirahezeRelay>	 <cosmicalpha> also what is happening on test151...
[05:19:22] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:19:52] <icinga-miraheze>	 PROBLEM - test151 php-fpm on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:20:01] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: NTP CRITICAL: No response from NTP server
[05:20:02] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[05:21:27] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:37] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:44] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:44] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:49] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:50] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:50] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:51] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:21:57] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:22:07] <MirahezeRelay>	 <orduin> I'm pretty sure the IA backups systemd unit bugged out and decided to start 60+ backup scripts (at least that's what happened earlier)
[05:22:12] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:26:18] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[05:26:43] <MirahezeRelay>	 <orduin> Created <https://github.com/miraheze/puppet/pull/4479>, should also fix powerdns on os202
[05:26:43] <MacFanBot>	 [GitHub] [miraheze/puppet #4479] open PR by The-Voidwalker, created 1 minute, 55 seconds ago: powerdns: pull query-local-address and forward-zones from hiera | (empty comment)
[05:27:02] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics
[05:27:02] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[05:27:03] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter
[05:27:03] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify
[05:27:03] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[05:27:03] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 2.484 second response time
[05:27:03] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[05:27:04] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps
[05:27:04] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[05:27:05] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[05:27:39] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[05:27:52] <icinga-miraheze>	 RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full
[05:28:31] <MirahezeRelay>	 <orduin> Not able to test/deploy myself, as it's late, and I need to get to bed
[05:31:24] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[05:31:27] <icinga-miraheze>	 RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2'
[05:31:31] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.003634393215 secs
[05:31:34] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[05:31:50] <icinga-miraheze>	 RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server'
[05:32:04] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:34:57] <icinga-miraheze>	 PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:36:32] <icinga-miraheze>	 PROBLEM - test151 JobRunner Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:36:35] <icinga-miraheze>	 PROBLEM - test151 Redis Process on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:36:37] <icinga-miraheze>	 PROBLEM - test151 JobChron Service on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:36:47] <icinga-miraheze>	 PROBLEM - test151 php-fpm on test151 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds
[05:37:53] <icinga-miraheze>	 PROBLEM - test151 ferm_active on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:19] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:19] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:20] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:22] <icinga-miraheze>	 PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:28] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:37] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:50] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:52] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:58] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:38:58] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:39:03] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:39:04] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:40:30] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> Might be worth just rebooting it
[05:41:27] <MirahezeRelay>	 <orduin> I did earlier, and the script still wound up starting up again
[05:42:27] <MirahezeRelay>	 <cosmicalpha> That script probably shouldnt even run on test151
[05:42:52] <MirahezeRelay>	 <orduin> I've downtimed the server in icinga web for 10 hours so the notifications aren't disruptive until someone gets around to fixing it. If it's fixed before then, be sure to clear the downtime.
[05:42:56] <MirahezeRelay>	 <orduin> I'm off for bed
[05:43:14] <MirahezeRelay>	 <cosmicalpha> thanks and night. Im not able to fix right now myself.
[06:42:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.70, 19.55, 20.37
[06:54:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.74, 20.24, 20.25
[06:58:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.04, 20.20, 20.24
[07:04:17] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.06, 20.56, 20.37
[07:12:17] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.24, 20.13, 20.31
[07:30:08] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.55, 20.51, 20.46
[08:00:13] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49842MiB (10% inode=99%);
[08:00:27] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1
[08:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756714257103[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756714257103
[08:43:34] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.70, 19.69, 20.31
[08:55:27] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.51, 20.11, 20.04
[08:59:26] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.16, 20.33, 20.16
[09:01:33] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49932MiB (10% inode=99%);
[09:05:22] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.54, 20.94, 20.48
[09:39:06] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.14, 20.08, 20.40
[09:43:03] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.42, 20.22, 20.39
[09:45:02] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.99, 20.07, 20.31
[09:53:27] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49863MiB (10% inode=99%);
[10:12:48] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.61, 20.15, 20.02
[10:14:47] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.56, 20.01, 20.00
[12:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756728657135[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756728657135
[12:25:41] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.23, 19.33, 17.62
[12:25:42] <MacFanBot>	 [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/0ac297947985a4a4f2235950683ebe9fbc934d6f
[12:25:42] <MacFanBot>	 02ImportDump/03main 07translatewiki.net 030ac2979 Localisation updates from https://translatewiki.net.
[12:25:43] <MacFanBot>	 [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/8a0c5f692b007166d562073a64895121a7c4e289
[12:25:43] <MacFanBot>	 02MatomoAnalytics/03main 07translatewiki.net 038a0c5f6 Localisation updates from https://translatewiki.net.
[12:25:45] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/c66f46b14923d6845739dfd56ab55905bbff6729
[12:25:47] <MacFanBot>	 [02RequestCustomDomain] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RequestCustomDomain/commit/dafb8bfb179468dd5517dc9fcedb8232e510e7c9
[12:25:49] <MacFanBot>	 [02RottenLinks] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/2e989355adb8944c3658f706d5d5826c9ff2eab7
[12:25:51] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 03c66f46b Localisation updates from https://translatewiki.net.
[12:25:52] <MacFanBot>	 02RequestCustomDomain/03main 07translatewiki.net 03dafb8bf Localisation updates from https://translatewiki.net.
[12:25:54] <MacFanBot>	 02RottenLinks/03main 07translatewiki.net 032e98935 Localisation updates from https://translatewiki.net.
[12:27:41] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 21.23, 18.52
[12:29:41] <icinga-miraheze>	 PROBLEM - mwtask171 Current Load on mwtask171 is WARNING: LOAD WARNING - total load average: 22.73, 21.69, 19.03
[12:33:07] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 14.68, 8.41
[12:33:41] <icinga-miraheze>	 RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 17.04, 20.26, 19.15
[12:34:43] <miraheze-github>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[12:35:04] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 4.67, 10.42, 7.59
[12:37:01] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.59, 7.43, 6.82
[12:37:39] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:38:29] <miraheze-github>	 miraheze/RottenLinks - translatewiki the build passed.
[12:40:27] <icinga-miraheze>	 PROBLEM - changeprop201 Disk Space on changeprop201 is WARNING: DISK WARNING - free space: / 2069MiB (10% inode=92%);
[12:46:22] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:52:05] <miraheze-github>	 miraheze/RequestCustomDomain - translatewiki the build passed.
[14:40:27] <icinga-miraheze>	 PROBLEM - changeprop201 Disk Space on changeprop201 is CRITICAL: DISK CRITICAL - free space: / 1130MiB (5% inode=92%);
[14:49:32] <MacFan4000>	 mmm, changeprop running out of spavce
[15:37:49] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:38:24] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[15:41:32] <icinga-miraheze>	 PROBLEM - farthestfrontier.wiki - Cloudflare on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[15:42:43] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:45:29] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.002626389265 secs
[15:49:15] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[15:49:16] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify
[15:49:20] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[15:54:31] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[15:56:37] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:56:41] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[15:56:57] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:00:27] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:05:41] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:08:35] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:10:13] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash
[16:10:15] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify
[16:10:18] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics
[16:10:35] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks
[16:10:45] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[16:10:46] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.001249760389 secs
[16:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756743057168[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756743057168
[16:11:23] <icinga-miraheze>	 PROBLEM - farthestfrontier.wiki - Cloudflare on sslhost is WARNING: WARNING - Certificate 'farthestfrontier.wiki' expires in 22 day(s) (Tue 23 Sep 2025 04:23:30 PM GMT +0000).
[16:16:09] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:17:54] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[16:18:15] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[16:18:28] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: NTP CRITICAL: No response from NTP server
[16:18:43] <icinga-miraheze>	 PROBLEM - test151 ferm_active on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:19:27] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table
[16:19:29] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[16:21:07] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:23:45] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[16:23:49] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -2.527236938e-05 secs
[16:25:08] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-statistics on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:25:29] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:25:36] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:26:38] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:27:01] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:28:54] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 17.25, 13.36, 6.96
[16:29:49] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:31:14] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:31:45] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:31:52] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:36:41] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:36:51] <icinga-miraheze>	 RECOVERY - test151 Check unit status of prometheus_jobqueue_stats on test151 is OK: OK: Status of the systemd unit prometheus_jobqueue_stats
[16:42:32] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 2.68, 11.40, 10.67
[16:44:23] <icinga-miraheze>	 PROBLEM - test151 Check unit status of prometheus_jobqueue_stats on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:44:29] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.93, 7.83, 9.45
[16:46:09] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:48:05] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[16:48:46] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[16:48:46] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[16:48:47] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.0001174211502 secs
[16:48:49] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash
[16:49:27] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps
[16:49:28] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks
[16:49:30] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[16:49:58] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[16:52:29] <icinga-miraheze>	 PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:54:25] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[16:55:16] <MacFanBot>	 [02puppet] 07Universal-Omega created 03backupwikis-no-beta (+1 new commit) 13https://github.com/miraheze/puppet/commit/f2e94c81d7eb
[16:55:16] <MacFanBot>	 02puppet/03backupwikis-no-beta 07CosmicAlpha 03f2e94c8 Don't run backupwikis on beta
[16:55:23] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4480: Don't run backupwikis on beta (03main...03backupwikis-no-beta) 13https://github.com/miraheze/puppet/pull/4480
[16:55:29] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4480: <details> […] 13https://github.com/miraheze/puppet/pull/4480#issuecomment-3242926154
[16:56:06] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:56:12] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:56:19] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:56:35] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:56:52] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:57:11] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[16:57:45] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_refreshlinks on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:00:09] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0afc0b30c5775a7e8628df1443ed548d302960cf
[17:00:10] <MacFanBot>	 02puppet/03main 07CosmicAlpha 030afc0b3 Don't run backupwikis on beta
[17:00:10] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4480: Don't run backupwikis on beta (03main...03backupwikis-no-beta) 13https://github.com/miraheze/puppet/pull/4480
[17:00:10] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03backupwikis-no-beta at 03f2e94c8 13https://api.github.com/repos/miraheze/puppet/commit/f2e94c8
[17:00:54] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:03:35] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[17:09:38] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:09:40] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_manage-inactive-wikis on test151 is OK: OK: Status of the systemd unit mediawiki_job_manage-inactive-wikis
[17:09:42] <icinga-miraheze>	 RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd'
[17:10:00] <icinga-miraheze>	 RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full
[17:10:06] <icinga-miraheze>	 RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[17:10:15] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-loginnotify on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-loginnotify
[17:10:17] <icinga-miraheze>	 RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server'
[17:10:39] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is OK: OK: Status of the systemd unit mediawiki_job_backup-all-wikis-ia
[17:10:47] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-wikibase-sites-table on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-wikibase-sites-table
[17:10:57] <icinga-miraheze>	 RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[17:11:02] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-abusefilter on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-abusefilter
[17:11:18] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_cleanup-upload-stash on test151 is OK: OK: Status of the systemd unit mediawiki_job_cleanup-upload-stash
[17:11:18] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-expired-blocks on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-expired-blocks
[17:11:27] <icinga-miraheze>	 RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[17:11:28] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-checkuser on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-checkuser
[17:11:33] <icinga-miraheze>	 RECOVERY - test151 memcached on test151 is OK: TCP OK - 0.000 second response time on 10.0.15.118 port 11211
[17:11:35] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 62 packages available for upgrade (0 critical updates).
[17:11:36] <icinga-miraheze>	 RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.2'
[17:11:37] <icinga-miraheze>	 RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set
[17:11:38] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -1.320242882e-05 secs
[17:11:42] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_update-statistics on test151 is OK: OK: Status of the systemd unit mediawiki_job_update-statistics
[17:11:45] <icinga-miraheze>	 RECOVERY - test151 Current Load on test151 is OK: LOAD OK - total load average: 1.59, 0.49, 0.17
[17:11:46] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_purge-parsercache on test151 is OK: OK: Status of the systemd unit mediawiki_job_purge-parsercache
[17:11:47] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.294 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[17:11:52] <icinga-miraheze>	 RECOVERY - test151 Check unit status of prometheus_jobqueue_stats on test151 is OK: OK: Status of the systemd unit prometheus_jobqueue_stats
[17:11:57] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_generate-sitemaps on test151 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemaps
[17:12:04] <icinga-miraheze>	 RECOVERY - test151 Check unit status of mediawiki_job_refreshlinks on test151 is OK: OK: Status of the systemd unit mediawiki_job_refreshlinks
[17:13:24] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.202 second response time
[17:13:32] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 0.066 second response time
[17:14:12] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[17:16:31] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is UNKNOWN: NRPE: Command 'check_Check unit status of mediawiki_job_backup-all-wikis-ia' not defined
[17:17:59] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: Connection refusedconnect to host 10.0.15.118 port 5666: Connection refused
[17:18:00] <icinga-miraheze>	 PROBLEM - test151 Check unit status of mediawiki_job_backup-all-wikis-ia on test151 is UNKNOWN: NRPE: Command 'check_Check unit status of mediawiki_job_backup-all-wikis-ia' not defined
[17:19:06] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/aa7fe5657d0721c5b443b7d774b84e0a3e6f5105
[17:19:06] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03aa7fe56 Remove absented
[17:19:48] <icinga-miraheze>	 PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.fsslc.wtnet port 443 after 0 ms: Couldn't connect to server
[17:21:09] <icinga-miraheze>	 RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4114 bytes in 0.069 second response time
[17:21:32] <MirahezeLSBot>	 !log [universalomega@test151] reboot
[17:21:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:21:55] <MirahezeLSBot>	 !log [universalomega@test151] purge mediawiki_job_backup-all-wikis-ia
[17:21:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:24:47] <MacFanBot>	 [02puppet] 07Universal-Omega merged 07The-Voidwalker's pull request #4479: powerdns: pull query-local-address and forward-zones from hiera (03main...03patch-dnsntp) 13https://github.com/miraheze/puppet/pull/4479
[17:24:47] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9fcc616ad57b886b7200afebd706b45521709739
[17:24:49] <MacFanBot>	 02puppet/03main 07The-Voidwalker 039fcc616 powerdns: pull query-local-address and forward-zones from hiera (#4479)
[17:24:51] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03patch-dnsntp at 038271ad3 13https://api.github.com/repos/miraheze/puppet/commit/8271ad3
[17:25:05] <icinga-miraheze>	 PROBLEM - cp161 Puppet on cp161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures
[17:25:11] <icinga-miraheze>	 PROBLEM - kafka181 Puppet on kafka181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures
[17:25:14] <icinga-miraheze>	 PROBLEM - mattermost1 Puppet on mattermost1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures
[17:25:14] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures
[17:25:16] <icinga-miraheze>	 PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[17:25:18] <icinga-miraheze>	 PROBLEM - mem201 Puppet on mem201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures
[17:25:18] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 9 minutes ago with 0 failures
[17:25:20] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[17:25:23] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 19 minutes ago with 0 failures
[17:25:24] <icinga-miraheze>	 PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures
[17:25:24] <icinga-miraheze>	 PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[17:25:27] <icinga-miraheze>	 PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures
[17:25:30] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures
[17:25:33] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures
[17:25:34] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[17:25:35] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 0 failures
[17:25:35] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 24 minutes ago with 0 failures
[17:25:38] <icinga-miraheze>	 PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[17:25:38] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 13 minutes ago with 0 failures
[17:25:39] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[17:25:49] <icinga-miraheze>	 PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[17:25:51] <icinga-miraheze>	 PROBLEM - cp191 Puppet on cp191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures
[17:25:53] <icinga-miraheze>	 PROBLEM - swiftobject201 Puppet on swiftobject201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures
[17:25:54] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[17:25:54] <icinga-miraheze>	 PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[17:25:57] <icinga-miraheze>	 PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[17:26:00] <icinga-miraheze>	 PROBLEM - os202 Puppet on os202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures
[17:26:01] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[17:26:02] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[17:26:05] <icinga-miraheze>	 PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures
[17:26:05] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures
[17:26:06] <icinga-miraheze>	 PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 31 minutes ago with 0 failures
[17:26:06] <icinga-miraheze>	 PROBLEM - cp201 Puppet on cp201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures
[17:26:06] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[17:26:06] <icinga-miraheze>	 PROBLEM - db172 Puppet on db172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures
[17:26:08] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures
[17:26:10] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[17:26:11] <icinga-miraheze>	 PROBLEM - swiftobject191 Puppet on swiftobject191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[17:26:11] <icinga-miraheze>	 PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures
[17:26:15] <icinga-miraheze>	 PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 0 failures
[17:26:18] <icinga-miraheze>	 PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 24 minutes ago with 0 failures
[17:26:19] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[17:26:23] <icinga-miraheze>	 PROBLEM - cloud20 Puppet on cloud20 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 24 minutes ago with 0 failures
[17:26:23] <icinga-miraheze>	 PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[17:26:24] <icinga-miraheze>	 PROBLEM - cp171 Puppet on cp171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[17:26:26] <icinga-miraheze>	 PROBLEM - os191 Puppet on os191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures
[17:26:27] <icinga-miraheze>	 PROBLEM - mem191 Puppet on mem191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures
[17:26:29] <icinga-miraheze>	 PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 24 minutes ago with 0 failures
[17:26:30] <icinga-miraheze>	 PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures
[17:26:31] <icinga-miraheze>	 PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[17:26:33] <icinga-miraheze>	 PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures
[17:26:34] <icinga-miraheze>	 PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures
[17:26:35] <icinga-miraheze>	 PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures
[17:26:35] <icinga-miraheze>	 PROBLEM - eventgate181 Puppet on eventgate181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[17:26:35] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures
[17:26:35] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[17:26:38] <icinga-miraheze>	 PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures
[17:26:38] <icinga-miraheze>	 PROBLEM - os201 Puppet on os201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[17:26:41] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures
[17:26:43] <icinga-miraheze>	 PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[17:26:43] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[17:26:46] <icinga-miraheze>	 PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures
[17:26:48] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[17:26:49] <icinga-miraheze>	 PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures
[17:26:59] <icinga-miraheze>	 PROBLEM - cloud19 Puppet on cloud19 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 5 minutes ago with 0 failures
[17:27:02] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[17:27:05] <icinga-miraheze>	 PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 13 minutes ago with 0 failures
[17:27:06] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 13 minutes ago with 0 failures
[17:27:06] <icinga-miraheze>	 PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures
[17:27:07] <icinga-miraheze>	 PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[17:27:11] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[17:27:11] <icinga-miraheze>	 PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 31 minutes ago with 0 failures
[17:27:17] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures
[17:27:20] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 0 failures
[17:27:23] <icinga-miraheze>	 PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures
[17:30:10] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[17:31:54] <icinga-miraheze>	 RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[17:34:03] <icinga-miraheze>	 PROBLEM - os202 Puppet on os202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:34:24] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/c8ac97b8b0ab052fa317b6756ce26c0177b89d8b
[17:34:24] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03c8ac97b Fix
[17:36:05] <icinga-miraheze>	 RECOVERY - os202 Puppet on os202 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[17:36:10] <icinga-miraheze>	 RECOVERY - os202 PowerDNS Recursor on os202 is OK: DNS OK: 0.136 seconds response time. os202.fsslc.wtnet returns 10.0.20.167
[17:38:33] <icinga-miraheze>	 RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[17:39:38] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures
[17:39:51] <icinga-miraheze>	 RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:00] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:05] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:06] <icinga-miraheze>	 RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:06] <icinga-miraheze>	 RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:07] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:08] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:22] <icinga-miraheze>	 RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:23] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:32] <icinga-miraheze>	 RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:35] <icinga-miraheze>	 RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:36] <icinga-miraheze>	 RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:40] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:40:41] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[17:40:45] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures
[17:40:48] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:03] <icinga-miraheze>	 RECOVERY - cloud19 Puppet on cloud19 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[17:41:05] <icinga-miraheze>	 RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[17:41:07] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:08] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:11] <icinga-miraheze>	 RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[17:41:14] <icinga-miraheze>	 RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:14] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[17:41:15] <icinga-miraheze>	 RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:18] <icinga-miraheze>	 RECOVERY - mem201 Puppet on mem201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:41:20] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[17:41:22] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:24] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:24] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[17:41:27] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:28] <icinga-miraheze>	 RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[17:41:33] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:33] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:35] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:36] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:53] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:57] <icinga-miraheze>	 RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:41:58] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[17:41:58] <icinga-miraheze>	 RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:05] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:06] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:11] <icinga-miraheze>	 RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:15] <icinga-miraheze>	 RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[17:42:20] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:42:27] <icinga-miraheze>	 RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:27] <icinga-miraheze>	 RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[17:42:28] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:31] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:35] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[17:42:35] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[17:42:35] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[17:42:36] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[17:42:41] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:41] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:42:49] <icinga-miraheze>	 RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:50] <icinga-miraheze>	 RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:42:53] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:43:05] <icinga-miraheze>	 RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:43:06] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[17:43:10] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[17:43:11] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[17:43:25] <icinga-miraheze>	 RECOVERY - swiftobject201 Puppet on swiftobject201 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[17:43:27] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[17:43:36] <icinga-miraheze>	 RECOVERY - swiftobject191 Puppet on swiftobject191 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[17:43:39] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[17:43:43] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[17:43:54] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:44:07] <icinga-miraheze>	 RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:44:11] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[17:45:06] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[17:45:35] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[17:46:00] <icinga-miraheze>	 PROBLEM - swiftproxy161 Current Load on swiftproxy161 is WARNING: LOAD WARNING - total load average: 15.61, 8.04, 3.69
[17:46:07] <icinga-miraheze>	 RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 13.59, 7.86, 3.67
[17:51:01] <icinga-miraheze>	 RECOVERY - changeprop201 Disk Space on changeprop201 is OK: DISK OK - free space: / 12978MiB (68% inode=92%);
[17:57:09] <MirahezeRelay>	 <cosmicalpha, replying to MacFan4000> Fixed (also it was running out of space it was completely out)
[17:57:55] <MirahezeRelay>	 <cosmicalpha, replying to orduin> I've removed iabackup from beta and also deployed your pdns patch
[18:01:10] <MacFan4000>	 @cosmicalpha mediawiki_job_generate-sitemap-index also failed it seems
[18:01:32] <MirahezeRelay>	 <cosmicalpha> yeah not worried about it right this moment I'll look later today.
[19:03:38] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/f8505b39751c
[19:03:38] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03f8505b3 changeprop201: cut LocalGlobalUserPageCacheUpdateJob in half to try and speed it up
[19:03:56] <MacFanBot>	 [02puppet] 07paladox opened pull request #4481: changeprop201: cut LocalGlobalUserPageCacheUpdateJob in half to try and speed it up (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4481
[19:04:02] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4481: <details> […] 13https://github.com/miraheze/puppet/pull/4481#issuecomment-3243112435
[19:22:05] <MirahezeRelay>	 <orduin> Sitemap was failing due to pdns, if there's no longer any problems, it should work fine now
[19:23:14] <MacFan4000>	 just needs to be restarted
[19:23:38] <MirahezeRelay>	 <orduin> Yeah
[19:23:55] <MirahezeRelay>	 <orduin> I can do that in a few minutes
[19:34:59] <MacFanBot>	 [02puppet] 07Universal-Omega created 03ntp-internal (+1 new commit) 13https://github.com/miraheze/puppet/commit/ad4bc387185e
[19:34:59] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 03ad4bc38 Allow ntp to connect from internal network
[19:36:24] <icinga-miraheze>	 PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures
[19:37:21] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4482: Allow ntp to connect from internal network (03main...03ntp-internal) 13https://github.com/miraheze/puppet/pull/4482
[19:37:27] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4482: <details> […] 13https://github.com/miraheze/puppet/pull/4482#issuecomment-3243153440
[19:38:07] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 10.99, 12.51, 7.54
[19:40:07] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.14, 8.66, 6.73
[19:41:20] <icinga-miraheze>	 RECOVERY - mwtask181 Check unit status of mediawiki_job_generate-sitemap-index on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_generate-sitemap-index
[19:43:08] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/7f5d3e6e8b14ee54db4db28aea48fed07eb11b11
[19:43:08] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 037f5d3e6 Add
[19:44:37] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/76272e0db7c8da4c6dca1c700fbf02fe69d252a9
[19:44:37] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 0376272e0 Update
[19:46:11] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/bdcff452e99116e1153e9c615aa1ed190a160cb4
[19:46:11] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 03bdcff45 Update
[19:46:57] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/90a1d20efaa92ee490146c719f84ad3616fd89b5
[19:46:57] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 0390a1d20 Update
[19:47:31] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/f5c0efdde98062414a2ff307652cb1cd2459bf1c
[19:47:31] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 03f5c0efd Update
[19:47:50] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/bf8dcb36063583e157c8314f34f0b4ac3f72f93d
[19:47:50] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 03bf8dcb3 Update
[19:48:11] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/7a9f254583cf62c04ffeafc2352a3e5cc43fac5d
[19:48:11] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 037a9f254 Update
[19:48:30] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/eccf3672b991683a936eaf019317776e64d5aab0
[19:48:30] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 03eccf367 Update
[19:52:24] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03ntp-internal 13https://github.com/miraheze/puppet/commit/17baa0056e6ad77e93b7105ca8e00688df637272
[19:52:25] <MacFanBot>	 02puppet/03ntp-internal 07CosmicAlpha 0317baa00 Update
[19:53:20] <icinga-miraheze>	 PROBLEM - matomo151 Disk Space on matomo151 is WARNING: DISK WARNING - free space: / 1969MiB (10% inode=89%);
[19:55:31] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/9e3432fb49a0ace859084b5e0b19148dba5789af
[19:55:31] <MacFanBot>	 02ssl/03main 07WikiTideBot 039e3432f Bot: Auto-update domain lists
[19:57:10] <MacFanBot>	 [02puppet] 07Universal-Omega retitled PR #4482: "Allow ntp to connect from internal network" ➜ "T14165: allow ntp to connect from internal network" 13https://github.com/miraheze/puppet/pull/4482
[20:10:57] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 100000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756532210000&orgId=1&to=1756757457199[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1756652780000&orgId=1&to=1756757457199
[20:15:15] <icinga-miraheze>	 PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures
[20:15:20] <icinga-miraheze>	 PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures
[20:15:21] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[20:15:21] <icinga-miraheze>	 PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures
[20:15:22] <icinga-miraheze>	 PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[20:15:25] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 9 minutes ago with 0 failures
[20:15:29] <icinga-miraheze>	 PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[20:15:31] <icinga-miraheze>	 PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[20:15:32] <icinga-miraheze>	 PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[20:15:33] <icinga-miraheze>	 PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 5 minutes ago with 0 failures
[20:15:35] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[20:15:35] <icinga-miraheze>	 PROBLEM - cp171 Puppet on cp171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[20:15:36] <icinga-miraheze>	 PROBLEM - os191 Puppet on os191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures
[20:15:36] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[20:15:37] <icinga-miraheze>	 PROBLEM - mattermost1 Puppet on mattermost1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[20:15:39] <icinga-miraheze>	 PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures
[20:15:39] <icinga-miraheze>	 PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[20:15:40] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures
[20:15:44] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures
[20:15:46] <icinga-miraheze>	 PROBLEM - cp201 Puppet on cp201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures
[20:15:48] <icinga-miraheze>	 PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 19 minutes ago with 0 failures
[20:15:49] <icinga-miraheze>	 PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[20:15:50] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[20:15:52] <icinga-miraheze>	 PROBLEM - cloud20 Puppet on cloud20 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures
[20:15:53] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 12 minutes ago with 0 failures
[20:15:55] <icinga-miraheze>	 PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 17 minutes ago with 0 failures
[20:15:57] <icinga-miraheze>	 PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[20:16:01] <icinga-miraheze>	 PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures
[20:16:01] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures
[20:16:02] <icinga-miraheze>	 PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures
[20:16:05] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures
[20:16:05] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures
[20:16:09] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures
[20:16:10] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 2 minutes ago with 0 failures
[20:16:11] <icinga-miraheze>	 PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures
[20:16:14] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures
[20:16:14] <icinga-miraheze>	 PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[20:16:14] <icinga-miraheze>	 PROBLEM - os201 Puppet on os201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 31 minutes ago with 0 failures
[20:16:15] <icinga-miraheze>	 PROBLEM - os202 Puppet on os202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures
[20:16:20] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures
[20:16:21] <icinga-miraheze>	 PROBLEM - mw183 Puppet on mw183 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures
[20:16:21] <icinga-miraheze>	 PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[20:16:21] <icinga-miraheze>	 PROBLEM - cp161 Puppet on cp161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures
[20:16:24] <icinga-miraheze>	 PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 32 minutes ago with 0 failures
[20:16:24] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[20:16:26] <icinga-miraheze>	 PROBLEM - mem191 Puppet on mem191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures
[20:16:27] <icinga-miraheze>	 PROBLEM - eventgate181 Puppet on eventgate181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures
[20:16:27] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures
[20:16:27] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures
[20:16:28] <icinga-miraheze>	 PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 2 minutes ago with 0 failures
[20:16:29] <icinga-miraheze>	 PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 16 minutes ago with 0 failures
[20:16:29] <icinga-miraheze>	 PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures
[20:16:34] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures
[20:16:34] <icinga-miraheze>	 PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[20:16:35] <icinga-miraheze>	 PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures
[20:16:35] <icinga-miraheze>	 PROBLEM - db172 Puppet on db172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures
[20:16:43] <icinga-miraheze>	 PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 19 minutes ago with 0 failures
[20:16:45] <icinga-miraheze>	 PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures
[20:16:45] <icinga-miraheze>	 PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures
[20:16:46] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures
[20:16:50] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[20:16:51] <icinga-miraheze>	 PROBLEM - cp191 Puppet on cp191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures
[20:16:51] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 29 minutes ago with 0 failures
[20:16:54] <icinga-miraheze>	 PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 5 minutes ago with 0 failures
[20:16:59] <icinga-miraheze>	 PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 23 minutes ago with 0 failures
[20:17:00] <icinga-miraheze>	 PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 7 minutes ago with 0 failures
[20:17:08] <icinga-miraheze>	 PROBLEM - swiftobject201 Puppet on swiftobject201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 5 minutes ago with 0 failures
[20:17:09] <icinga-miraheze>	 PROBLEM - mem201 Puppet on mem201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures
[20:17:09] <icinga-miraheze>	 PROBLEM - cloud19 Puppet on cloud19 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 25 minutes ago with 0 failures
[20:17:10] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures
[20:17:12] <icinga-miraheze>	 PROBLEM - kafka181 Puppet on kafka181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures
[20:17:16] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[20:17:17] <icinga-miraheze>	 PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures
[20:17:21] <icinga-miraheze>	 PROBLEM - swiftobject191 Puppet on swiftobject191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 6 minutes ago with 0 failures
[20:17:34] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[20:19:06] <icinga-miraheze>	 PROBLEM - os202 NTP time on os202 is CRITICAL: connect to address 10.0.20.167 port 5666: Connection refusedconnect to host 10.0.20.167 port 5666: Connection refused
[20:19:13] <icinga-miraheze>	 RECOVERY - os202 NTP time on os202 is OK: NTP OK: Offset -0.0006413459778 secs
[20:19:59] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4482: T14165: allow ntp to connect from internal network (03main...03ntp-internal) 13https://github.com/miraheze/puppet/pull/4482
[20:20:01] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1746b9dc17d28546ec20b5ef54d8426d1752ed1f
[20:20:03] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03ntp-internal at 0317baa00 13https://api.github.com/repos/miraheze/puppet/commit/17baa00
[20:20:05] <MacFanBot>	 02puppet/03main 07CosmicAlpha 031746b9d T14165: allow ntp to connect from internal network…
[20:20:22] <icinga-miraheze>	 RECOVERY - os202 Puppet on os202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:20:50] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:20:51] <icinga-miraheze>	 RECOVERY - cp191 Puppet on cp191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:20:58] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:09] <icinga-miraheze>	 RECOVERY - cloud19 Puppet on cloud19 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:10] <icinga-miraheze>	 RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:15] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:23] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:33] <icinga-miraheze>	 RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:35] <icinga-miraheze>	 RECOVERY - cp171 Puppet on cp171 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[20:21:37] <icinga-miraheze>	 RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:39] <icinga-miraheze>	 RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[20:21:45] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:21:47] <icinga-miraheze>	 RECOVERY - cp201 Puppet on cp201 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[20:21:48] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[20:21:51] <icinga-miraheze>	 RECOVERY - cloud20 Puppet on cloud20 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[20:21:53] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[20:21:54] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:21:59] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[20:22:02] <icinga-miraheze>	 RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[20:22:05] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[20:22:09] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[20:22:10] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:14] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:20] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[20:22:21] <icinga-miraheze>	 RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:23] <icinga-miraheze>	 RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[20:22:23] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:24] <icinga-miraheze>	 RECOVERY - cp161 Puppet on cp161 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[20:22:26] <icinga-miraheze>	 RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:27] <icinga-miraheze>	 RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:22:27] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:27] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:22:28] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:22:33] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:34] <icinga-miraheze>	 RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:22:34] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:37] <icinga-miraheze>	 RECOVERY - db172 Puppet on db172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:22:45] <icinga-miraheze>	 RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:45] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:51] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:22:52] <icinga-miraheze>	 RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[20:23:00] <icinga-miraheze>	 RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[20:23:08] <icinga-miraheze>	 RECOVERY - swiftobject201 Puppet on swiftobject201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:09] <icinga-miraheze>	 RECOVERY - mem201 Puppet on mem201 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[20:23:09] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures
[20:23:16] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:18] <icinga-miraheze>	 RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:20] <icinga-miraheze>	 RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[20:23:21] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures
[20:23:21] <icinga-miraheze>	 RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[20:23:21] <icinga-miraheze>	 RECOVERY - swiftobject191 Puppet on swiftobject191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:22] <icinga-miraheze>	 RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[20:23:28] <icinga-miraheze>	 RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:32] <icinga-miraheze>	 RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[20:23:32] <icinga-miraheze>	 RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:34] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:36] <icinga-miraheze>	 RECOVERY - os191 Puppet on os191 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:39] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[20:23:41] <icinga-miraheze>	 PROBLEM - os161 Puppet on os161 is CRITICAL: connect to address 10.0.16.129 port 5666: Connection refusedconnect to host 10.0.16.129 port 5666: Connection refused
[20:23:50] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:23:52] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[20:23:54] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[20:23:57] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[20:23:58] <icinga-miraheze>	 PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: Connection refusedconnect to host 10.0.17.108 port 5666: Connection refused
[20:24:02] <icinga-miraheze>	 RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures
[20:24:04] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:24:11] <icinga-miraheze>	 RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[20:24:14] <icinga-miraheze>	 RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[20:24:18] <icinga-miraheze>	 RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[20:24:18] <icinga-miraheze>	 RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[20:24:19] <icinga-miraheze>	 RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[20:24:28] <icinga-miraheze>	 RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[20:24:28] <icinga-miraheze>	 RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[20:24:45] <icinga-miraheze>	 RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[20:24:51] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:29:02] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a4d94d0ee98be289a3bff2078aef0ca51bcfab5a
[20:29:02] <MacFanBot>	 02ssl/03main 07WikiTideBot 03a4d94d0 Bot: Update SSL cert for farthestfrontier.wiki
[20:32:34] <MacFanBot>	 [02puppet] 07Universal-Omega created 03os202 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e592a17e4691
[20:32:34] <MacFanBot>	 02puppet/03os202 07CosmicAlpha 03e592a17 os202: not a master
[20:32:43] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4483: os202: not a master (03main...03os202) 13https://github.com/miraheze/puppet/pull/4483
[20:32:50] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4483: <details> […] 13https://github.com/miraheze/puppet/pull/4483#issuecomment-3243218308
[20:34:35] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4483: os202: not a master (03main...03os202) 13https://github.com/miraheze/puppet/pull/4483
[20:34:35] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e436fce59d614235359b355810edb25c0f6b25dd
[20:34:36] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03e436fce os202: not a master
[20:34:38] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03os202 at 03e592a17 13https://api.github.com/repos/miraheze/puppet/commit/e592a17
[20:38:12] <MacFanBot>	 [02puppet] 07Universal-Omega created 03os202-pool (+1 new commit) 13https://github.com/miraheze/puppet/commit/62809af40799
[20:38:12] <MacFanBot>	 02puppet/03os202-pool 07CosmicAlpha 0362809af Pool in os202
[20:38:16] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4484: Pool in os202 (03main...03os202-pool) 13https://github.com/miraheze/puppet/pull/4484
[20:38:23] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4484: <details> […] 13https://github.com/miraheze/puppet/pull/4484#issuecomment-3243224492
[20:45:50] <icinga-miraheze>	 PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures
[20:47:04] <MacFanBot>	 [02puppet] 07Universal-Omega created 03privatens (+1 new commit) 13https://github.com/miraheze/puppet/commit/ead9983fdab4
[20:47:05] <MacFanBot>	 02puppet/03privatens 07CosmicAlpha 03ead9983 resolver: only use private ns if forward_use_internal
[20:47:08] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4485: resolver: only use private ns if forward_use_internal (03main...03privatens) 13https://github.com/miraheze/puppet/pull/4485
[20:47:15] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4485: <details> […] 13https://github.com/miraheze/puppet/pull/4485#issuecomment-3243234389
[20:50:37] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03privatens 13https://github.com/miraheze/puppet/commit/7ed79392581e90080586122033e93390f69d5e75
[20:50:37] <MacFanBot>	 02puppet/03privatens 07CosmicAlpha 037ed7939 Update
[20:57:53] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4485: resolver: only use private ns if forward_use_internal (03main...03privatens) 13https://github.com/miraheze/puppet/pull/4485
[20:57:53] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0de82598812fafefbbe17535ef82319ed348c852
[20:57:55] <MacFanBot>	 02puppet/03main 07CosmicAlpha 030de8259 resolver: only use private ns if forward_use_internal
[20:57:57] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03privatens at 037ed7939 13https://api.github.com/repos/miraheze/puppet/commit/7ed7939
[21:00:40] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[21:02:34] <icinga-miraheze>	 PROBLEM - mattermost1 HTTPS on mattermost1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds
[21:02:35] <icinga-miraheze>	 PROBLEM - www.mwcosmos.com - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[21:02:35] <icinga-miraheze>	 PROBLEM - ns2 SSH on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:02:35] <icinga-miraheze>	 PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089
[21:02:35] <icinga-miraheze>	 PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[21:02:55] <icinga-miraheze>	 RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 04 Nov 2025 06:36:56 AM GMT +0000.
[21:03:04] <icinga-miraheze>	 RECOVERY - mattermost1 HTTPS on mattermost1 is OK: HTTP OK: HTTP/2 200 - 505 bytes in 0.479 second response time
[21:03:08] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.305 seconds response time. ns2.wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66
[21:03:12] <icinga-miraheze>	 RECOVERY - ns2 SSH on ns2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[21:03:19] <icinga-miraheze>	 RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 6438MiB (35% inode=90%);
[21:03:31] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.43],Exec[git_pull_femiwiki-deploy-1.44],Exec[git_pull_3d2png]
[21:03:48] <icinga-miraheze>	 RECOVERY - www.mwcosmos.com - Cloudflare on sslhost is OK: OK - Certificate 'www.mwcosmos.com' will expire on Mon 13 Oct 2025 02:50:05 AM GMT +0000.
[21:04:10] <icinga-miraheze>	 PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/openvox-keyring.gpg],File[/usr/local/bin/puppet-enabled]
[21:08:15] <icinga-miraheze>	 RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[21:09:01] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[21:10:26] <icinga-miraheze>	 PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:10:47] <icinga-miraheze>	 RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.918 second response time
[21:10:53] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw161
[21:11:22] <icinga-miraheze>	 RECOVERY - farthestfrontier.wiki - Cloudflare on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Sun 30 Nov 2025 07:30:26 PM GMT +0000.
[21:11:23] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[21:12:35] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[21:15:15] <icinga-miraheze>	 PROBLEM - ping6 on mattermost1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.05 ms
[21:21:14] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw181
[21:23:13] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[21:32:29] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4484: Pool in os202 (03main...03os202-pool) 13https://github.com/miraheze/puppet/pull/4484
[21:32:29] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1b8b7d46520de041a713185b66aa8170334ab735
[21:32:29] <MacFanBot>	 02puppet/03main 07CosmicAlpha 031b8b7d4 Pool in os202
[21:32:30] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03os202-pool at 0362809af 13https://api.github.com/repos/miraheze/puppet/commit/62809af
[21:33:07] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.76, 5.77, 5.03
[21:33:10] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 5.86, 5.07
[21:33:34] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.77, 5.98, 5.13
[21:34:53] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.12, 6.45, 5.36
[21:39:10] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.58, 7.40, 6.06
[21:43:49] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.91, 6.76, 6.20
[22:21:49] <icinga-miraheze>	 PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.04, 5.77, 4.78
[22:21:52] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.41, 5.60, 5.07
[22:22:46] <icinga-miraheze>	 RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.30, 5.80, 4.85
[22:22:52] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.97, 5.52, 5.08
[22:25:22] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.53, 6.37, 5.47
[22:25:36] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 6.76, 6.25, 5.44
[22:25:42] <icinga-miraheze>	 PROBLEM - os202 Current Load on os202 is WARNING: LOAD WARNING - total load average: 7.03, 6.33, 5.22
[22:25:54] <icinga-miraheze>	 RECOVERY - os202 Current Load on os202 is OK: LOAD OK - total load average: 6.41, 6.24, 5.21
[22:44:16] <MacFanBot>	 [02puppet] 07Universal-Omega created 03recursor (+1 new commit) 13https://github.com/miraheze/puppet/commit/b794176dd30c
[22:44:16] <MacFanBot>	 02puppet/03recursor 07CosmicAlpha 03b794176 Fix recursor if forward_use_internal
[22:44:21] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4486: Fix recursor if forward_use_internal (03main...03recursor) 13https://github.com/miraheze/puppet/pull/4486
[22:44:26] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4486: <details> […] 13https://github.com/miraheze/puppet/pull/4486#issuecomment-3243344550
[22:45:04] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 7104MiB (3% inode=93%);
[22:45:06] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03recursor 13https://github.com/miraheze/puppet/commit/cf2da5d300376461a364f37551ecee0f6bc43d8d
[22:45:06] <MacFanBot>	 02puppet/03recursor 07CosmicAlpha 03cf2da5d Update
[22:51:06] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4486: Fix recursor if forward_use_internal (03main...03recursor) 13https://github.com/miraheze/puppet/pull/4486
[22:51:07] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/095b9453cd15c09b0e4a2a96abad80aa01e0c25d
[22:51:08] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03095b945 Fix recursor if forward_use_internal…
[22:51:10] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03recursor at 03cf2da5d 13https://api.github.com/repos/miraheze/puppet/commit/cf2da5d
[22:56:49] <MacFanBot>	 [02puppet] 07Universal-Omega created 03mon (+1 new commit) 13https://github.com/miraheze/puppet/commit/c1e3f3b80bfd
[22:56:49] <MacFanBot>	 02puppet/03mon 07CosmicAlpha 03c1e3f3b Fix monitoring for forward_use_internal
[22:56:53] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #4487: Fix monitoring for forward_use_internal (03main...03mon) 13https://github.com/miraheze/puppet/pull/4487
[22:56:59] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4487: <details> […] 13https://github.com/miraheze/puppet/pull/4487#issuecomment-3243354767
[22:57:48] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-6 (+1 new commit) 13https://github.com/miraheze/puppet/commit/995d7b36e358
[22:57:48] <MacFanBot>	 02puppet/03paladox-patch-6 07paladox 03995d7b3 changeprop201: Lower concurrency for LocalGlobalUserPageCacheUpdateJob to 25…
[22:57:58] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mon 13https://github.com/miraheze/puppet/commit/8d83e97fad07e16e056cee99d7d5a0947c35b7bc
[22:57:58] <MacFanBot>	 02puppet/03mon 07CosmicAlpha 038d83e97 Update
[22:58:03] <MacFanBot>	 [02puppet] 07paladox opened pull request #4488: changeprop201: Lower concurrency for LocalGlobalUserPageCacheUpdateJob to 25 (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4488
[22:58:08] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4488: <details> […] 13https://github.com/miraheze/puppet/pull/4488#issuecomment-3243355669
[22:58:32] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4487: Fix monitoring for forward_use_internal (03main...03mon) 13https://github.com/miraheze/puppet/pull/4487
[22:58:33] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f809c2da3a8a57c0e708bc864e906f512cea820a
[22:58:34] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03f809c2d Fix monitoring for forward_use_internal
[22:58:36] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03mon at 038d83e97 13https://api.github.com/repos/miraheze/puppet/commit/8d83e97
[23:09:16] <MacFanBot>	 [02puppet] 07paladox merged pull request #4488: changeprop201: Lower concurrency for LocalGlobalUserPageCacheUpdateJob to 25 (03main...03paladox-patch-6) 13https://github.com/miraheze/puppet/pull/4488
[23:09:16] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1fd4944bf3afb085524c52c5d734e9f5ab5be10e
[23:09:16] <MacFanBot>	 02puppet/03main 07paladox 031fd4944 changeprop201: Lower concurrency for LocalGlobalUserPageCacheUpdateJob to 25 (#4488)…
[23:09:17] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-6 at 03995d7b3 13https://api.github.com/repos/miraheze/puppet/commit/995d7b3
[23:09:32] <MacFanBot>	 [02puppet] 07paladox closed pull request #4481: changeprop201: cut LocalGlobalUserPageCacheUpdateJob in half to try and speed it up (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4481
[23:09:37] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03f8505b3 13https://api.github.com/repos/miraheze/puppet/commit/f8505b3
[23:09:53] <MacFanBot>	 [02puppet] 07paladox commented on pull request #4472: @dependabot rebase 13https://github.com/miraheze/puppet/pull/4472#issuecomment-3243365558
[23:13:29] <MacFanBot>	 [02puppet] 07dependabot[bot] 04force-pushed 03dependabot/github_actions/actions/checkout-5 from 0384a61de to 035c58947 13https://api.github.com/repos/miraheze/puppet/commits/dependabot/github_actions/actions/checkout-5
[23:13:31] <MacFanBot>	 02puppet/03dependabot/github_actions/actions/checkout-5 07dependabot[bot] 035c58947 build(deps): bump actions/checkout from 4 to 5…
[23:14:42] <miraheze-github>	 miraheze/puppet - dependabot[bot] the build has errored.
[23:21:32] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.54, 21.02, 20.32
[23:22:30] <MirahezeRelay>	 <abaddriverlol, replying to MacFanBot> isn't it slightly suboptimal if GUP jobs take 4 times longer to clear?
[23:27:32] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 21.99, 20.84
[23:29:32] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.88, 21.93, 20.97
[23:31:47] <icinga-miraheze>	 RECOVERY - changeprop201 Puppet on changeprop201 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[23:37:32] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.23, 18.87, 20.10
[23:55:04] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 56645MiB (25% inode=93%);