[00:01:54] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[00:06:07] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57449MiB (12% inode=99%);
[00:06:49] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57933MiB (12% inode=99%);
[00:07:06] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57189MiB (12% inode=99%);
[01:01:09] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all
[01:01:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:02:00] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 51s
[01:02:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:04:44] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[01:09:39] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[01:12:44] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[01:17:23] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[01:17:45] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[01:17:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:18:08] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s
[01:18:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:20:27] <MirahezeRelay>	 <cosmicalpha> !log ALTER TABLE requestssl_requests DROP COLUMN request_private; on metawikibeta
[01:20:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:21:31] <MirahezeRelay>	 <cosmicalpha> !log ALTER TABLE requestssl_requests DROP COLUMN request_private; on metawiki
[01:21:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:25:07] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:ResetWikiCaches --wiki=metawiki (END - exit=0)
[01:25:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:42:36] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build has errored.
[01:53:29] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all
[01:53:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:54:13] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 44s
[01:54:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:55:17] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all
[01:55:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:56:07] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 50s
[01:56:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:01:33] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[02:12:18] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=mockupediawiki dump.xml --no-updates (START)
[02:12:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:13:53] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[03:00:13] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/importDump.php --wiki=mockupediawiki dump.xml --no-updates (END - exit=0)
[03:00:14] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php /srv/mediawiki/1.43/maintenance/rebuildall.php --wiki=mockupediawiki (START)
[03:00:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:00:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:18:16] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build has errored.
[03:34:57] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[04:22:09] <icinga-miraheze>	 PROBLEM - mw153 PowerDNS Recursor on mw153 is CRITICAL: Domain 'mw153.fsslc.wtnet' was not found by the server
[04:22:15] <icinga-miraheze>	 PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 100%
[04:22:18] <icinga-miraheze>	 PROBLEM - mw153 NTP time on mw153 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:22:21] <icinga-miraheze>	 PROBLEM - swiftobject161 PowerDNS Recursor on swiftobject161 is CRITICAL: No response from DNS ::1
[04:22:23] <icinga-miraheze>	 PROBLEM - mw202 PowerDNS Recursor on mw202 is CRITICAL: Domain 'mw202.fsslc.wtnet' was not found by the server
[04:22:24] <icinga-miraheze>	 PROBLEM - mw201 NTP time on mw201 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:22:27] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable
[04:22:28] <icinga-miraheze>	 PROBLEM - mem161 PowerDNS Recursor on mem161 is CRITICAL: Domain 'mem161.fsslc.wtnet' was not found by the server
[04:22:29] <icinga-miraheze>	 PROBLEM - mem191 PowerDNS Recursor on mem191 is CRITICAL: Domain 'mem191.fsslc.wtnet' was not found by the server
[04:22:30] <icinga-miraheze>	 PROBLEM - cloud17 SSH on cloud17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:22:30] <icinga-miraheze>	 PROBLEM - reports171 PowerDNS Recursor on reports171 is CRITICAL: No response from DNS ::1
[04:22:32] <icinga-miraheze>	 PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: No response from DNS ::1
[04:22:32] <icinga-miraheze>	 PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: No response from DNS ::1
[04:22:34] <icinga-miraheze>	 PROBLEM - mw171 PowerDNS Recursor on mw171 is CRITICAL: Domain 'mw171.fsslc.wtnet' was not found by the server
[04:22:38] <icinga-miraheze>	 PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: Domain 'db181.fsslc.wtnet' was not found by the server
[04:22:40] <icinga-miraheze>	 PROBLEM - os201 PowerDNS Recursor on os201 is CRITICAL: Domain 'os201.fsslc.wtnet' was not found by the server
[04:22:41] <icinga-miraheze>	 PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: No response from DNS ::1
[04:22:42] <icinga-miraheze>	 PROBLEM - eventgate181 NTP time on eventgate181 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:22:42] <icinga-miraheze>	 PROBLEM - db172 PowerDNS Recursor on db172 is CRITICAL: Domain 'db172.fsslc.wtnet' was not found by the server
[04:22:44] <icinga-miraheze>	 PROBLEM - mw163 PowerDNS Recursor on mw163 is CRITICAL: No response from DNS ::1
[04:22:46] <icinga-miraheze>	 PROBLEM - mem151 PowerDNS Recursor on mem151 is CRITICAL: Domain 'mem151.fsslc.wtnet' was not found by the server
[04:22:50] <icinga-miraheze>	 PROBLEM - os161 PowerDNS Recursor on os161 is CRITICAL: Domain 'os161.fsslc.wtnet' was not found by the server
[04:22:51] <icinga-miraheze>	 PROBLEM - mw202 NTP time on mw202 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:22:55] <icinga-miraheze>	 PROBLEM - cloud16 SSH on cloud16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:22:59] <icinga-miraheze>	 PROBLEM - mw203 PowerDNS Recursor on mw203 is CRITICAL: No response from DNS ::1
[04:22:59] <icinga-miraheze>	 PROBLEM - os191 PowerDNS Recursor on os191 is CRITICAL: Domain 'os191.fsslc.wtnet' was not found by the server
[04:22:59] <icinga-miraheze>	 PROBLEM - mw172 PowerDNS Recursor on mw172 is CRITICAL: Domain 'mw172.fsslc.wtnet' was not found by the server
[04:23:00] <icinga-miraheze>	 PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable
[04:23:01] <icinga-miraheze>	 PROBLEM - mw193 PowerDNS Recursor on mw193 is CRITICAL: Domain 'mw193.fsslc.wtnet' was not found by the server
[04:23:02] <icinga-miraheze>	 PROBLEM - mem161 NTP time on mem161 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:03] <icinga-miraheze>	 PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: Domain 'db151.fsslc.wtnet' was not found by the server
[04:23:04] <icinga-miraheze>	 PROBLEM - ldap171 NTP time on ldap171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:05] <icinga-miraheze>	 PROBLEM - eventgate181 PowerDNS Recursor on eventgate181 is CRITICAL: No response from DNS ::1
[04:23:07] <icinga-miraheze>	 PROBLEM - mw181 PowerDNS Recursor on mw181 is CRITICAL: Domain 'mw181.fsslc.wtnet' was not found by the server
[04:23:08] <icinga-miraheze>	 PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: Domain 'mw152.fsslc.wtnet' was not found by the server
[04:23:09] <icinga-miraheze>	 PROBLEM - db171 PowerDNS Recursor on db171 is CRITICAL: Domain 'db171.fsslc.wtnet' was not found by the server
[04:23:10] <icinga-miraheze>	 PROBLEM - os162 PowerDNS Recursor on os162 is CRITICAL: Domain 'os162.fsslc.wtnet' was not found by the server
[04:23:11] <icinga-miraheze>	 PROBLEM - mw201 PowerDNS Recursor on mw201 is CRITICAL: Domain 'mw201.fsslc.wtnet' was not found by the server
[04:23:11] <icinga-miraheze>	 PROBLEM - bots171 PowerDNS Recursor on bots171 is CRITICAL: Domain 'bots171.fsslc.wtnet' was not found by the server
[04:23:12] <icinga-miraheze>	 PROBLEM - mem201 PowerDNS Recursor on mem201 is CRITICAL: Domain 'mem201.fsslc.wtnet' was not found by the server
[04:23:12] <icinga-miraheze>	 PROBLEM - os161 NTP time on os161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:16] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:16] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift-container-stats_mw-media
[04:23:17] <icinga-miraheze>	 PROBLEM - mw192 PowerDNS Recursor on mw192 is CRITICAL: Domain 'mw192.fsslc.wtnet' was not found by the server
[04:23:18] <icinga-miraheze>	 PROBLEM - mw163 NTP time on mw163 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:18] <icinga-miraheze>	 PROBLEM - mem151 NTP time on mem151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:19] <icinga-miraheze>	 PROBLEM - mwtask161 PowerDNS Recursor on mwtask161 is CRITICAL: Domain 'mwtask161.fsslc.wtnet' was not found by the server
[04:23:19] <icinga-miraheze>	 PROBLEM - test151 NTP time on test151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:23] <icinga-miraheze>	 PROBLEM - ldap171 PowerDNS Recursor on ldap171 is CRITICAL: Domain 'ldap171.fsslc.wtnet' was not found by the server
[04:23:30] <icinga-miraheze>	 PROBLEM - prometheus151 NTP time on prometheus151 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:36] <icinga-miraheze>	 PROBLEM - mw173 PowerDNS Recursor on mw173 is CRITICAL: Domain 'mw173.fsslc.wtnet' was not found by the server
[04:23:37] <icinga-miraheze>	 PROBLEM - db171 NTP time on db171 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:38] <icinga-miraheze>	 PROBLEM - mwtask151 PowerDNS Recursor on mwtask151 is CRITICAL: No response from DNS ::1
[04:23:41] <icinga-miraheze>	 PROBLEM - mw181 NTP time on mw181 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:43] <icinga-miraheze>	 PROBLEM - os162 NTP time on os162 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:43] <icinga-miraheze>	 PROBLEM - os191 NTP time on os191 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:48] <icinga-miraheze>	 PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100%
[04:23:53] <icinga-miraheze>	 PROBLEM - matomo151 PowerDNS Recursor on matomo151 is CRITICAL: Domain 'matomo151.fsslc.wtnet' was not found by the server
[04:23:54] <icinga-miraheze>	 PROBLEM - Host cloud16 is DOWN: PING CRITICAL - Packet loss = 100%
[04:23:54] <icinga-miraheze>	 PROBLEM - mwtask151 NTP time on mwtask151 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:23:55] <icinga-miraheze>	 PROBLEM - mw173 NTP time on mw173 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:23:57] <icinga-miraheze>	 PROBLEM - mem201 NTP time on mem201 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:00] <icinga-miraheze>	 PROBLEM - mw172 NTP time on mw172 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:24:03] <icinga-miraheze>	 PROBLEM - cloud19 Puppet on cloud19 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:24:04] <icinga-miraheze>	 PROBLEM - os151 PowerDNS Recursor on os151 is CRITICAL: No response from DNS ::1
[04:24:05] <icinga-miraheze>	 PROBLEM - db181 NTP time on db181 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:24:05] <icinga-miraheze>	 PROBLEM - db182 PowerDNS Recursor on db182 is CRITICAL: Domain 'db182.fsslc.wtnet' was not found by the server
[04:24:07] <icinga-miraheze>	 PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: Domain 'prometheus151.fsslc.wtnet' was not found by the server
[04:24:08] <icinga-miraheze>	 PROBLEM - os201 NTP time on os201 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:10] <icinga-miraheze>	 PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: Domain 'puppet181.wikitide.net' was not found by the server
[04:24:14] <icinga-miraheze>	 PROBLEM - rdb151 PowerDNS Recursor on rdb151 is CRITICAL: No response from DNS ::1
[04:24:14] <icinga-miraheze>	 PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100%
[04:24:15] <icinga-miraheze>	 PROBLEM - mw203 NTP time on mw203 is CRITICAL: NTP CRITICAL: No response from NTP server
[04:24:16] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:24:18] <icinga-miraheze>	 RECOVERY - swiftobject161 PowerDNS Recursor on swiftobject161 is OK: DNS OK: 0.392 seconds response time. swiftobject161.fsslc.wtnet returns 10.0.16.134
[04:24:18] <icinga-miraheze>	 PROBLEM - db161 NTP time on db161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:21] <icinga-miraheze>	 PROBLEM - reports171 NTP time on reports171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:22] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.012 second response time
[04:24:26] <icinga-miraheze>	 PROBLEM - mw153 NTP time on mw153 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:27] <icinga-miraheze>	 PROBLEM - os151 NTP time on os151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:28] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.279 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[04:24:29] <icinga-miraheze>	 RECOVERY - mw171 PowerDNS Recursor on mw171 is OK: DNS OK: 0.283 seconds response time. mw171.fsslc.wtnet returns 10.0.17.122
[04:24:31] <icinga-miraheze>	 RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.274 seconds response time. mwtask181.fsslc.wtnet returns 10.0.18.106
[04:24:31] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:24:33] <icinga-miraheze>	 PROBLEM - rdb151 NTP time on rdb151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:38] <icinga-miraheze>	 PROBLEM - mw192 NTP time on mw192 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:40] <icinga-miraheze>	 RECOVERY - os201 PowerDNS Recursor on os201 is OK: DNS OK: 0.051 seconds response time. os201.fsslc.wtnet returns 10.0.20.156
[04:24:42] <icinga-miraheze>	 PROBLEM - mwtask161 NTP time on mwtask161 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:46] <icinga-miraheze>	 PROBLEM - mem191 NTP time on mem191 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:46] <icinga-miraheze>	 PROBLEM - db182 NTP time on db182 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:51] <icinga-miraheze>	 PROBLEM - db172 NTP time on db172 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:54] <icinga-miraheze>	 PROBLEM - eventgate181 NTP time on eventgate181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:24:55] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.027 second response time
[04:24:59] <icinga-miraheze>	 RECOVERY - mw193 PowerDNS Recursor on mw193 is OK: DNS OK: 0.034 seconds response time. mw193.fsslc.wtnet returns 10.0.19.164
[04:25:01] <icinga-miraheze>	 RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.028 seconds response time. db151.fsslc.wtnet returns 10.0.15.110
[04:25:02] <icinga-miraheze>	 PROBLEM - mw202 NTP time on mw202 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:25:03] <icinga-miraheze>	 RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.041 seconds response time. mw152.fsslc.wtnet returns 10.0.15.115
[04:25:10] <icinga-miraheze>	 PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:25:13] <icinga-miraheze>	 RECOVERY - mem161 NTP time on mem161 is OK: NTP OK: Offset -0.0004764199257 secs
[04:25:15] <icinga-miraheze>	 RECOVERY - os161 NTP time on os161 is OK: NTP OK: Offset -1.427531242e-05 secs
[04:25:21] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.0007871985435 secs
[04:25:25] <icinga-miraheze>	 PROBLEM - bots171 NTP time on bots171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:25:34] <icinga-miraheze>	 RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 0.283 seconds response time. mwtask151.fsslc.wtnet returns 10.0.15.150
[04:25:35] <icinga-miraheze>	 RECOVERY - prometheus151 NTP time on prometheus151 is OK: NTP OK: Offset -0.0007048249245 secs
[04:25:45] <icinga-miraheze>	 PROBLEM - db171 NTP time on db171 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:25:47] <icinga-miraheze>	 RECOVERY - os162 NTP time on os162 is OK: NTP OK: Offset -0.0006412565708 secs
[04:25:48] <icinga-miraheze>	 PROBLEM - mw181 NTP time on mw181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:25:50] <icinga-miraheze>	 RECOVERY - matomo151 PowerDNS Recursor on matomo151 is OK: DNS OK: 0.148 seconds response time. matomo151.fsslc.wtnet returns 10.0.15.112
[04:25:55] <icinga-miraheze>	 RECOVERY - mwtask151 NTP time on mwtask151 is OK: NTP OK: Offset -0.0004941523075 secs
[04:25:56] <icinga-miraheze>	 RECOVERY - Host cloud16 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms
[04:26:00] <icinga-miraheze>	 RECOVERY - os151 PowerDNS Recursor on os151 is OK: DNS OK: 0.405 seconds response time. os151.fsslc.wtnet returns 10.0.15.111
[04:26:02] <icinga-miraheze>	 RECOVERY - mw153 PowerDNS Recursor on mw153 is OK: DNS OK: 0.034 seconds response time. mw153.fsslc.wtnet returns 10.0.15.140
[04:26:05] <icinga-miraheze>	 RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.279 seconds response time. prometheus151.fsslc.wtnet returns 10.0.15.116
[04:26:10] <icinga-miraheze>	 RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 0.160 seconds response time. puppet181.wikitide.net returns 10.0.18.100
[04:26:10] <icinga-miraheze>	 PROBLEM - mw172 NTP time on mw172 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:26:10] <icinga-miraheze>	 RECOVERY - rdb151 PowerDNS Recursor on rdb151 is OK: DNS OK: 0.414 seconds response time. rdb151.fsslc.wtnet returns 10.0.15.142
[04:26:12] <icinga-miraheze>	 RECOVERY - os201 NTP time on os201 is OK: NTP OK: Offset -0.0008900165558 secs
[04:26:12] <icinga-miraheze>	 PROBLEM - db181 NTP time on db181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[04:26:17] <icinga-miraheze>	 RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
[04:26:20] <icinga-miraheze>	 RECOVERY - mw203 NTP time on mw203 is OK: NTP OK: Offset -0.0005360245705 secs
[04:26:22] <icinga-miraheze>	 RECOVERY - mem161 PowerDNS Recursor on mem161 is OK: DNS OK: 0.393 seconds response time. mem161.fsslc.wtnet returns 10.0.16.131
[04:26:26] <icinga-miraheze>	 RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.19 ms
[04:26:27] <icinga-miraheze>	 RECOVERY - mw153 NTP time on mw153 is OK: NTP OK: Offset -0.001061469316 secs
[04:26:29] <icinga-miraheze>	 RECOVERY - os151 NTP time on os151 is OK: NTP OK: Offset -0.0005521774292 secs
[04:26:38] <icinga-miraheze>	 RECOVERY - cloud17 SSH on cloud17 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[04:26:38] <icinga-miraheze>	 RECOVERY - rdb151 NTP time on rdb151 is OK: NTP OK: Offset -0.0002561509609 secs
[04:26:40] <icinga-miraheze>	 RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.040 seconds response time. os161.fsslc.wtnet returns 10.0.16.129
[04:26:52] <icinga-miraheze>	 RECOVERY - mw203 PowerDNS Recursor on mw203 is OK: DNS OK: 0.272 seconds response time. mw203.fsslc.wtnet returns 10.0.20.165
[04:26:55] <icinga-miraheze>	 RECOVERY - db172 NTP time on db172 is OK: NTP OK: Offset -2.473592758e-05 secs
[04:26:55] <icinga-miraheze>	 RECOVERY - cloud16 SSH on cloud16 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[04:26:56] <icinga-miraheze>	 RECOVERY - mwtask161 NTP time on mwtask161 is OK: NTP OK: Offset -0.0003541707993 secs
[04:26:59] <icinga-miraheze>	 RECOVERY - eventgate181 NTP time on eventgate181 is OK: NTP OK: Offset -0.001016646624 secs
[04:27:02] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[04:27:02] <icinga-miraheze>	 RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.034 seconds response time. os162.fsslc.wtnet returns 10.0.16.141
[04:27:03] <icinga-miraheze>	 RECOVERY - eventgate181 PowerDNS Recursor on eventgate181 is OK: DNS OK: 0.276 seconds response time. eventgate181.fsslc.wtnet returns 10.0.18.147
[04:27:30] <icinga-miraheze>	 PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:27:32] <icinga-miraheze>	 RECOVERY - mw163 NTP time on mw163 is OK: NTP OK: Offset -0.0002889633179 secs
[04:27:33] <icinga-miraheze>	 RECOVERY - mem151 NTP time on mem151 is OK: NTP OK: Offset -0.0004307925701 secs
[04:27:45] <icinga-miraheze>	 RECOVERY - db171 NTP time on db171 is OK: NTP OK: Offset -0.0002034902573 secs
[04:27:50] <icinga-miraheze>	 RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms
[04:27:53] <icinga-miraheze>	 RECOVERY - os191 NTP time on os191 is OK: NTP OK: Offset -0.0005336403847 secs
[04:27:56] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:28:09] <icinga-miraheze>	 RECOVERY - mw173 NTP time on mw173 is OK: NTP OK: Offset -0.000409245491 secs
[04:28:12] <icinga-miraheze>	 RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset -0.001099377871 secs
[04:28:34] <icinga-miraheze>	 RECOVERY - db172 PowerDNS Recursor on db172 is OK: DNS OK: 0.023 seconds response time. db172.fsslc.wtnet returns 10.0.17.158
[04:28:36] <icinga-miraheze>	 RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.041 seconds response time. db181.fsslc.wtnet returns 10.0.18.102
[04:28:42] <icinga-miraheze>	 RECOVERY - mw163 PowerDNS Recursor on mw163 is OK: DNS OK: 0.284 seconds response time. mw163.fsslc.wtnet returns 10.0.16.151
[04:28:45] <icinga-miraheze>	 RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 0.394 seconds response time. mem151.fsslc.wtnet returns 10.0.15.113
[04:28:55] <icinga-miraheze>	 RECOVERY - os191 PowerDNS Recursor on os191 is OK: DNS OK: 0.043 seconds response time. os191.fsslc.wtnet returns 10.0.19.152
[04:28:57] <icinga-miraheze>	 RECOVERY - mem191 NTP time on mem191 is OK: NTP OK: Offset -0.0002059936523 secs
[04:29:01] <icinga-miraheze>	 RECOVERY - db171 PowerDNS Recursor on db171 is OK: DNS OK: 0.030 seconds response time. db171.fsslc.wtnet returns 10.0.17.119
[04:29:02] <icinga-miraheze>	 RECOVERY - bots171 PowerDNS Recursor on bots171 is OK: DNS OK: 0.289 seconds response time. bots171.fsslc.wtnet returns 10.0.17.143
[04:29:02] <icinga-miraheze>	 RECOVERY - mwtask161 PowerDNS Recursor on mwtask161 is OK: DNS OK: 0.156 seconds response time. mwtask161.fsslc.wtnet returns 10.0.16.157
[04:29:29] <icinga-miraheze>	 RECOVERY - mw173 PowerDNS Recursor on mw173 is OK: DNS OK: 0.042 seconds response time. mw173.fsslc.wtnet returns 10.0.17.153
[04:29:33] <icinga-miraheze>	 RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset -0.0003616809845 secs
[04:29:34] <icinga-miraheze>	 RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset -0.0002645254135 secs
[04:29:58] <icinga-miraheze>	 PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:29:59] <icinga-miraheze>	 PROBLEM - mem191 Puppet on mem191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:30:10] <icinga-miraheze>	 RECOVERY - mem191 PowerDNS Recursor on mem191 is OK: DNS OK: 0.040 seconds response time. mem191.fsslc.wtnet returns 10.0.19.154
[04:30:14] <icinga-miraheze>	 PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:30:27] <icinga-miraheze>	 RECOVERY - mem201 NTP time on mem201 is OK: NTP OK: Offset -0.0006836950779 secs
[04:30:34] <icinga-miraheze>	 RECOVERY - mw172 NTP time on mw172 is OK: NTP OK: Offset -0.0001189410686 secs
[04:30:47] <icinga-miraheze>	 RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.270 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123
[04:30:54] <icinga-miraheze>	 RECOVERY - mw201 NTP time on mw201 is OK: NTP OK: Offset -0.0006066560745 secs
[04:30:55] <icinga-miraheze>	 PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:30:56] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media
[04:30:57] <icinga-miraheze>	 RECOVERY - mw192 NTP time on mw192 is OK: NTP OK: Offset -0.0005653500557 secs
[04:31:03] <icinga-miraheze>	 RECOVERY - mem201 PowerDNS Recursor on mem201 is OK: DNS OK: 0.275 seconds response time. mem201.fsslc.wtnet returns 10.0.20.148
[04:31:04] <icinga-miraheze>	 RECOVERY - db182 NTP time on db182 is OK: NTP OK: Offset -0.0005622804165 secs
[04:31:09] <icinga-miraheze>	 RECOVERY - ldap171 PowerDNS Recursor on ldap171 is OK: DNS OK: 0.398 seconds response time. ldap171.fsslc.wtnet returns 10.0.17.121
[04:31:09] <icinga-miraheze>	 RECOVERY - mw192 PowerDNS Recursor on mw192 is OK: DNS OK: 0.293 seconds response time. mw192.fsslc.wtnet returns 10.0.19.161
[04:32:01] <icinga-miraheze>	 RECOVERY - mw181 NTP time on mw181 is OK: NTP OK: Offset -0.0007241070271 secs
[04:32:03] <icinga-miraheze>	 RECOVERY - db182 PowerDNS Recursor on db182 is OK: DNS OK: 0.026 seconds response time. db182.fsslc.wtnet returns 10.0.18.103
[04:32:43] <icinga-miraheze>	 RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.038 seconds response time. mw181.fsslc.wtnet returns 10.0.18.104
[04:32:49] <icinga-miraheze>	 RECOVERY - mw201 PowerDNS Recursor on mw201 is OK: DNS OK: 0.280 seconds response time. mw201.fsslc.wtnet returns 10.0.20.162
[04:32:54] <icinga-miraheze>	 RECOVERY - db161 NTP time on db161 is OK: NTP OK: Offset -0.0003216564655 secs
[04:33:37] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:34:46] <icinga-miraheze>	 RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.277 seconds response time. db161.fsslc.wtnet returns 10.0.16.128
[04:36:39] <icinga-miraheze>	 PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:43:57] <icinga-miraheze>	 RECOVERY - reports171 NTP time on reports171 is OK: NTP OK: Offset -0.0002949237823 secs
[04:44:02] <icinga-miraheze>	 RECOVERY - reports171 PowerDNS Recursor on reports171 is OK: DNS OK: 0.157 seconds response time. reports171.fsslc.wtnet returns 10.0.17.139
[04:44:24] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:44:38] <icinga-miraheze>	 RECOVERY - mw202 NTP time on mw202 is OK: NTP OK: Offset -0.001499801874 secs
[04:45:56] <icinga-miraheze>	 RECOVERY - mw202 PowerDNS Recursor on mw202 is OK: DNS OK: 0.281 seconds response time. mw202.fsslc.wtnet returns 10.0.20.163
[04:51:10] <icinga-miraheze>	 RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:51:52] <icinga-miraheze>	 RECOVERY - cloud19 Puppet on cloud19 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[04:52:00] <MirahezeRelay>	 <orduin> Weird, looks kind of like we just had a fairly major hiccup. Not sure what exactly happened though, given the first notifications have been recovery (an nobody appears to have complained about an outage).
[04:52:08] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:52:23] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:53:21] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:53:45] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[04:55:47] <icinga-miraheze>	 RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[04:58:14] <icinga-miraheze>	 RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:58:48] <icinga-miraheze>	 RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[04:59:53] <icinga-miraheze>	 RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:00:39] <icinga-miraheze>	 RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[05:01:54] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[05:10:22] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:04:31] <icinga-miraheze>	 RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset -6.049871445e-05 secs
[06:04:58] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -4.479289055e-05 secs
[06:05:01] <icinga-miraheze>	 RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.274 seconds response time. mw152.fsslc.wtnet returns 10.0.15.115
[06:05:27] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:05:38] <icinga-miraheze>	 PROBLEM - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is CRITICAL: CRITICAL: Status of the systemd unit swift_dispersion_stats
[06:05:42] <icinga-miraheze>	 RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.034 seconds response time. db151.fsslc.wtnet returns 10.0.15.110
[06:06:02] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:06:24] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time
[06:06:34] <icinga-miraheze>	 RECOVERY - swiftac171 NTP time on swiftac171 is OK: NTP OK: Offset 3.823637962e-05 secs
[06:06:35] <icinga-miraheze>	 RECOVERY - swiftac171 PowerDNS Recursor on swiftac171 is OK: DNS OK: 0.161 seconds response time. swiftac171.fsslc.wtnet returns 10.0.17.125
[06:06:40] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.153 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[06:06:53] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.021 second response time
[06:06:55] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.022 second response time
[06:07:06] <icinga-miraheze>	 RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time
[06:07:10] <icinga-miraheze>	 RECOVERY - ldap171 PowerDNS Recursor on ldap171 is OK: DNS OK: 1.052 second response time. ldap171.fsslc.wtnet returns 10.0.17.121
[06:07:23] <icinga-miraheze>	 RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset 0.0002961754799 secs
[06:09:12] <icinga-miraheze>	 PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[06:10:55] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift-container-stats_mw-media on swiftproxy171 is OK: OK: Status of the systemd unit swift-container-stats_mw-media
[06:12:05] <MirahezeRelay>	 <cosmicalpha> !log restart all swift* stack servers (swiftobject, swiftac, swiftproxy)
[06:12:49] <MirahezeRelay>	 <cosmicalpha> MacFan4000: around?
[06:12:56] <MirahezeRelay>	 <cosmicalpha> is logbot off IRC side?
[06:13:41] <MacFan4000>	 It is
[06:14:16] <MirahezeRelay>	 <cosmicalpha> Can you give a timestamp when it went offline?
[06:14:40] <MirahezeRelay>	 <cosmicalpha> I am starting to think this was a DNS blip
[06:15:18] <MirahezeRelay>	 <cosmicalpha> Which might explain PowerDNS warnings a bit ago too.
[06:16:55] <MacFan4000>	 00:24 ⇐ MirahezeLogbot quit (~MirahezeL@miraheze/bots) Ping timeout: 260 seconds
[06:17:27] <icinga-miraheze>	 RECOVERY - swiftproxy171 Check unit status of swift_dispersion_stats on swiftproxy171 is OK: OK: Status of the systemd unit swift_dispersion_stats
[06:18:47] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:18:52] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[06:19:51] <MirahezeRelay>	 <cosmicalpha> Is it back MacFan4000?
[06:20:05] <MacFan4000>	 Yes
[06:20:12] <MirahezeRelay>	 <cosmicalpha> Any others down?
[06:20:27] <MirahezeRelay>	 <cosmicalpha> !log restart logbot
[06:20:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:20:45] <icinga-miraheze>	 RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:21:18] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[06:21:21] <MacFan4000>	 The rest seem to be working
[06:21:56] <MirahezeRelay>	 <cosmicalpha> What exactly happened. Seems we had 2 sorta DNS outages over the past few hours.
[06:22:09] <MirahezeRelay>	 <cosmicalpha> Very odd that first notifications are recoveries...
[06:22:37] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:24:13] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[06:24:29] <MirahezeRelay>	 <cosmicalpha> I have no idea what just happened...
[06:24:56] <MirahezeRelay>	 <cosmicalpha> How do you start by notifying of recoveries...
[06:26:01] <icinga-miraheze>	 RECOVERY - mem191 Puppet on mem191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:27:10] <icinga-miraheze>	 RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:30:35] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[06:32:36] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> well, the bot is also hosted by us, on the same server as logbot. kinda hard to report your own outage when ur like
[06:32:37] <MirahezeRelay>	 <pixldev> dead
[06:34:20] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:34:59] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:35:24] <icinga-miraheze>	 RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:36:57] <MirahezeRelay>	 <cosmicalpha, replying to pixldev> Its a different server lol
[06:37:16] <MirahezeRelay>	 <cosmicalpha> hosted by us but a different server than other bots
[06:37:27] <MirahezeRelay>	 <pixldev> oh
[06:37:27] <MirahezeRelay>	 <pixldev> well
[06:37:46] <MirahezeRelay>	 <cosmicalpha> Yeah its very odd anyway.
[06:38:37] <MirahezeRelay>	 <cosmicalpha> It seems if we still did our own DNA entirely we'd have been down completely. Because it seems the outage was a DNA outage.
[06:38:43] <MirahezeRelay>	 <cosmicalpha> ugh DNS
[06:38:46] <MirahezeRelay>	 <cosmicalpha> lol
[06:38:49] <MirahezeRelay>	 <pixldev> you said DNA twice
[06:38:59] <MirahezeRelay>	 <cosmicalpha> Autocorrect x2 lol
[06:39:34] <MirahezeRelay>	 <pixldev> correct sure
[06:40:43] <MirahezeRelay>	 <cosmicalpha> my autocorrect is bizarre
[06:40:56] <MirahezeRelay>	 <cosmicalpha> And ive just been to lazy to turn it off apparently lol
[06:41:36] <MirahezeRelay>	 <cosmicalpha> Anyway I hope nothing explodes. Im off for the nigjt.
[06:43:11] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> cheers
[06:43:18] <MirahezeRelay>	 <pixldev> i should be too its almost 3am
[06:43:49] <MirahezeRelay>	 <cosmicalpha> I was up till 6AM yesterday and the 2 days before that so a bit tired tonight lol
[06:44:04] <MirahezeRelay>	 <pixldev> and here i was gonna say im starting to seem like you
[06:44:18] <MirahezeRelay>	 <pixldev> night and good Cosmic
[09:39:57] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'AutoCreatePage'} to test151
[09:39:58] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'AutoCreatePage'} to test151 - SUCCESS in 1s
[09:40:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:40:03] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'AutoCreatePage'} to all
[09:40:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:40:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:40:28] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'AutoCreatePage'} to all - SUCCESS in 24s
[09:40:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:48:07] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49869MiB (10% inode=99%);
[09:55:06] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49845MiB (10% inode=99%);
[10:07:44] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to test151
[10:07:45] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[10:07:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:07:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:09:23] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to test151
[10:09:24] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to test151 - SUCCESS in 0s
[10:09:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:09:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:09:31] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to test151
[10:09:32] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to test151 - SUCCESS in 0s
[10:09:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:09:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:12:29] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to all
[10:12:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:12:40] <MirahezeRelay>	 <abaddriverlol, replying to MirahezeLSBot> (force upgrade because the maintainers force pushed to master)
[10:12:55] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'EmbedVideo'} to all - SUCCESS in 25s
[10:12:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:24:13] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:ToggleExtension --wiki=exttestwikibeta --name=dynamicsidebar --disable (END - exit=0)
[10:24:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:24:49] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49935MiB (10% inode=99%);
[12:27:36] <miraheze-github>	 miraheze/mw-config - pskyechology the build passed.
[12:28:37] <miraheze-github>	 miraheze/mw-config - pskyechology the build passed.
[12:28:50] <MirahezeLSBot>	 !log [skye@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[12:28:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:29:19] <MirahezeLSBot>	 !log [skye@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s
[12:29:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:27:49] <MirahezeLSBot>	 !log [skye@mwtask181] Starting import for danmachienwiki (XML: dungeonnideaiomotomeru_pages_full.xml; Images: None) (START)
[14:27:50] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=danmachienwiki --no-updates --username-prefix=wikia:danmachi -- dungeonnideaiomotomeru_pages_full.xml (START)
[14:27:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:27:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:26:28] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=basramowiki --new=baslamowiki --user=Skye (END - exit=0)
[16:26:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:28:08] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=baslamowiki (END - exit=0)
[16:28:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:08:24] <MirahezeRelay>	 <cosmicalpha> !log RENAME TABLE requestssl_requests TO customdomain_requests, requestssl_request_comments TO customdomain_request_comments; on metawikibeta
[17:08:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:08:57] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all
[17:09:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:09:46] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 49s
[17:09:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:10:19] <MirahezeRelay>	 <cosmicalpha> !log RENAME TABLE requestssl_requests TO customdomain_requests, requestssl_request_comments TO customdomain_request_comments; on metawiki
[17:10:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:11:04] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[17:24:52] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[17:45:35] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[18:14:51] <icinga-miraheze>	 PROBLEM - os191 NTP time on os191 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:17:31] <icinga-miraheze>	 PROBLEM - mw182 NTP time on mw182 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:17:31] <icinga-miraheze>	 PROBLEM - db151 NTP time on db151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:17:31] <icinga-miraheze>	 PROBLEM - mw152 NTP time on mw152 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:17:31] <icinga-miraheze>	 PROBLEM - cp171 PowerDNS Recursor on cp171 is CRITICAL: No response from DNS ::1
[18:17:31] <icinga-miraheze>	 PROBLEM - cp201 PowerDNS Recursor on cp201 is CRITICAL: No response from DNS ::1
[18:18:02] <icinga-miraheze>	 RECOVERY - db172 PowerDNS Recursor on db172 is OK: DNS OK: 0.036 seconds response time. db172.fsslc.wtnet returns 10.0.17.158
[18:18:07] <icinga-miraheze>	 RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 0.402 seconds response time. mem151.fsslc.wtnet returns 10.0.15.113
[18:18:14] <icinga-miraheze>	 RECOVERY - cloud16 SSH on cloud16 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[18:18:16] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.282 seconds response time. ns2.wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66
[18:18:16] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:19] <icinga-miraheze>	 PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:20] <icinga-miraheze>	 PROBLEM - cloud18 ferm_active on cloud18 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:22] <icinga-miraheze>	 PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:18:22] <icinga-miraheze>	 PROBLEM - cloud18 IPMI Sensors on cloud18 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:23] <icinga-miraheze>	 PROBLEM - cloud15 Disk Space on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:24] <icinga-miraheze>	 PROBLEM - cloud18 APT on cloud18 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:27] <icinga-miraheze>	 RECOVERY - ping6 on cloud19 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[18:18:30] <icinga-miraheze>	 PROBLEM - Host cloud18 is DOWN: PING CRITICAL - Packet loss = 100%
[18:18:31] <icinga-miraheze>	 PROBLEM - cloud15 conntrack_table_size on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:36] <icinga-miraheze>	 RECOVERY - mw172 PowerDNS Recursor on mw172 is OK: DNS OK: 0.033 seconds response time. mw172.fsslc.wtnet returns 10.0.17.123
[18:18:37] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTPS on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.030 second response time
[18:18:40] <RhinosF1>	 Fuck
[18:18:42] <icinga-miraheze>	 RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.273 seconds response time. mw152.fsslc.wtnet returns 10.0.15.115
[18:18:43] <icinga-miraheze>	 PROBLEM - cloud15 ferm_active on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[18:18:49] <icinga-miraheze>	 RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Tue 04 Nov 2025 06:36:56 AM GMT +0000.
[18:18:50] <icinga-miraheze>	 RECOVERY - bots171 PowerDNS Recursor on bots171 is OK: DNS OK: 0.045 seconds response time. bots171.fsslc.wtnet returns 10.0.17.143
[18:18:58] <icinga-miraheze>	 PROBLEM - Host cloud15 is DOWN: PING CRITICAL - Packet loss = 100%
[18:19:01] <icinga-miraheze>	 RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset 0.0003624260426 secs
[18:19:09] <icinga-miraheze>	 RECOVERY - ns2 SSH on ns2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[18:19:14] <icinga-miraheze>	 RECOVERY - mw151 NTP time on mw151 is OK: NTP OK: Offset 0.001781493425 secs
[18:19:16] <icinga-miraheze>	 RECOVERY - mw151 PowerDNS Recursor on mw151 is OK: DNS OK: 0.281 seconds response time. mw151.fsslc.wtnet returns 10.0.15.114
[18:19:22] <icinga-miraheze>	 RECOVERY - kafka181 PowerDNS Recursor on kafka181 is OK: DNS OK: 0.407 seconds response time. kafka181.fsslc.wtnet returns 10.0.18.159
[18:19:31] <icinga-miraheze>	 RECOVERY - cp171 PowerDNS Recursor on cp171 is OK: DNS OK: 0.277 seconds response time. cp171.wikitide.net returns 2602:294:0:b23::112,38.46.223.206
[18:19:31] <icinga-miraheze>	 PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:19:32] <MirahezeRelay>	 <pskyechology> who ate dns
[18:19:35] <icinga-miraheze>	 RECOVERY - swiftproxy171 HTTP on swiftproxy171 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time
[18:19:36] <icinga-miraheze>	 RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset 0.001080065966 secs
[18:19:36] <icinga-miraheze>	 RECOVERY - mwtask151 NTP time on mwtask151 is OK: NTP OK: Offset 0.001052767038 secs
[18:19:36] <icinga-miraheze>	 RECOVERY - kafka181 NTP time on kafka181 is OK: NTP OK: Offset 0.001448214054 secs
[18:19:38] <icinga-miraheze>	 PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:19:39] <icinga-miraheze>	 RECOVERY - os191 NTP time on os191 is OK: NTP OK: Offset 0.0009786784649 secs
[18:19:47] <icinga-miraheze>	 RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.394 seconds response time. test151.fsslc.wtnet returns 10.0.15.118
[18:19:54] <icinga-miraheze>	 RECOVERY - cp171 NTP time on cp171 is OK: NTP OK: Offset 0.0003311634064 secs
[18:19:54] <icinga-miraheze>	 RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.043 seconds response time. db181.fsslc.wtnet returns 10.0.18.102
[18:19:58] <icinga-miraheze>	 RECOVERY - eventgate181 NTP time on eventgate181 is OK: NTP OK: Offset 0.00115853548 secs
[18:20:27] <icinga-miraheze>	 RECOVERY - Host cloud18 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[18:20:37] <icinga-miraheze>	 RECOVERY - os191 PowerDNS Recursor on os191 is OK: DNS OK: 0.399 seconds response time. os191.fsslc.wtnet returns 10.0.19.152
[18:20:40] <MirahezeRelay>	 <rhinosf1, replying to pskyechology> No idea
[18:20:40] <icinga-miraheze>	 RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.419 seconds response time. mw181.fsslc.wtnet returns 10.0.18.104
[18:20:41] <icinga-miraheze>	 RECOVERY - eventgate181 PowerDNS Recursor on eventgate181 is OK: DNS OK: 0.037 seconds response time. eventgate181.fsslc.wtnet returns 10.0.18.147
[18:20:42] <icinga-miraheze>	 RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset 0.001229614019 secs
[18:20:46] <MirahezeRelay>	 <rhinosf1> But reassuringly nothing is down
[18:21:11] <icinga-miraheze>	 RECOVERY - mw181 NTP time on mw181 is OK: NTP OK: Offset 0.001006126404 secs
[18:21:16] <icinga-miraheze>	 PROBLEM - os201 Puppet on os201 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:21:18] <icinga-miraheze>	 RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 0.279 seconds response time. mwtask151.fsslc.wtnet returns 10.0.15.150
[18:21:23] <icinga-miraheze>	 PROBLEM - db171 Puppet on db171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:21:28] <MirahezeRelay>	 <pskyechology> i would cry if my import died
[18:21:31] <icinga-miraheze>	 RECOVERY - cloud18 SSH on cloud18 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[18:21:52] <icinga-miraheze>	 RECOVERY - mw162 NTP time on mw162 is OK: NTP OK: Offset 0.0002483427525 secs
[18:22:01] <icinga-miraheze>	 RECOVERY - ping6 on cloud18 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms
[18:22:16] <icinga-miraheze>	 RECOVERY - cloud18 ferm_active on cloud18 is OK: OK ferm input default policy is set
[18:22:21] <icinga-miraheze>	 RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 138 packages available for upgrade (0 critical updates).
[18:22:21] <icinga-miraheze>	 RECOVERY - cloud18 IPMI Sensors on cloud18 is OK: IPMI Status: OK
[18:23:15] <icinga-miraheze>	 PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:23:21] <icinga-miraheze>	 RECOVERY - mw162 PowerDNS Recursor on mw162 is OK: DNS OK: 0.041 seconds response time. mw162.fsslc.wtnet returns 10.0.16.133
[18:23:53] <icinga-miraheze>	 RECOVERY - mw182 PowerDNS Recursor on mw182 is OK: DNS OK: 0.035 seconds response time. mw182.fsslc.wtnet returns 10.0.18.105
[18:23:57] <icinga-miraheze>	 RECOVERY - mw182 NTP time on mw182 is OK: NTP OK: Offset 0.001238554716 secs
[18:24:12] <icinga-miraheze>	 RECOVERY - mw191 NTP time on mw191 is OK: NTP OK: Offset 0.001259446144 secs
[18:24:43] <icinga-miraheze>	 RECOVERY - mw183 NTP time on mw183 is OK: NTP OK: Offset 0.001604944468 secs
[18:25:12] <icinga-miraheze>	 PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:25:18] <icinga-miraheze>	 RECOVERY - mwtask161 NTP time on mwtask161 is OK: NTP OK: Offset 0.0004163086414 secs
[18:25:29] <icinga-miraheze>	 PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:25:44] <icinga-miraheze>	 RECOVERY - mw191 PowerDNS Recursor on mw191 is OK: DNS OK: 0.279 seconds response time. mw191.fsslc.wtnet returns 10.0.19.160
[18:25:44] <icinga-miraheze>	 RECOVERY - mw183 PowerDNS Recursor on mw183 is OK: DNS OK: 0.041 seconds response time. mw183.fsslc.wtnet returns 10.0.18.155
[18:25:53] <icinga-miraheze>	 RECOVERY - Host cloud15 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms
[18:25:55] <icinga-miraheze>	 RECOVERY - cloud15 SSH on cloud15 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u7 (protocol 2.0)
[18:26:06] <icinga-miraheze>	 RECOVERY - cloud15 ferm_active on cloud15 is OK: OK ferm input default policy is set
[18:26:17] <icinga-miraheze>	 RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK
[18:26:19] <icinga-miraheze>	 RECOVERY - cloud15 conntrack_table_size on cloud15 is OK: OK: nf_conntrack is 0 % full
[18:26:19] <icinga-miraheze>	 RECOVERY - cloud15 Disk Space on cloud15 is OK: DISK OK - free space: / 84101MiB (92% inode=98%);
[18:26:35] <icinga-miraheze>	 RECOVERY - os162 NTP time on os162 is OK: NTP OK: Offset 0.000265955925 secs
[18:26:41] <icinga-miraheze>	 RECOVERY - ping6 on cloud15 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms
[18:26:58] <icinga-miraheze>	 RECOVERY - mwtask161 PowerDNS Recursor on mwtask161 is OK: DNS OK: 0.039 seconds response time. mwtask161.fsslc.wtnet returns 10.0.16.157
[18:28:26] <icinga-miraheze>	 RECOVERY - db171 NTP time on db171 is OK: NTP OK: Offset 0.000263094902 secs
[18:28:33] <icinga-miraheze>	 RECOVERY - os201 NTP time on os201 is OK: NTP OK: Offset 0.001416653395 secs
[18:28:37] <icinga-miraheze>	 RECOVERY - db171 PowerDNS Recursor on db171 is OK: DNS OK: 0.145 seconds response time. db171.fsslc.wtnet returns 10.0.17.119
[18:28:43] <icinga-miraheze>	 RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.055 seconds response time. os162.fsslc.wtnet returns 10.0.16.141
[18:29:13] <icinga-miraheze>	 PROBLEM - rdb151 Puppet on rdb151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:29:37] <icinga-miraheze>	 PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:30:06] <icinga-miraheze>	 RECOVERY - os201 PowerDNS Recursor on os201 is OK: DNS OK: 0.033 seconds response time. os201.fsslc.wtnet returns 10.0.20.156
[18:31:35] <icinga-miraheze>	 PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:32:31] <icinga-miraheze>	 RECOVERY - os161 NTP time on os161 is OK: NTP OK: Offset 0.0003805160522 secs
[18:32:52] <icinga-miraheze>	 RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset 0.0003158450127 secs
[18:33:00] <icinga-miraheze>	 RECOVERY - ldap171 PowerDNS Recursor on ldap171 is OK: DNS OK: 0.052 seconds response time. ldap171.fsslc.wtnet returns 10.0.17.121
[18:33:40] <icinga-miraheze>	 RECOVERY - os161 PowerDNS Recursor on os161 is OK: DNS OK: 0.044 seconds response time. os161.fsslc.wtnet returns 10.0.16.129
[18:34:46] <icinga-miraheze>	 PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:38:51] <icinga-miraheze>	 RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 2.032 seconds response time. db151.fsslc.wtnet returns 10.0.15.110
[18:39:27] <icinga-miraheze>	 RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset 0.001864433289 secs
[18:41:07] <icinga-miraheze>	 RECOVERY - db182 NTP time on db182 is OK: NTP OK: Offset 0.001830041409 secs
[18:41:26] <icinga-miraheze>	 RECOVERY - db182 PowerDNS Recursor on db182 is OK: DNS OK: 0.162 seconds response time. db182.fsslc.wtnet returns 10.0.18.103
[18:41:39] <icinga-miraheze>	 RECOVERY - phorge171 NTP time on phorge171 is OK: NTP OK: Offset 0.000337511301 secs
[18:42:19] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:43:09] <icinga-miraheze>	 RECOVERY - phorge171 PowerDNS Recursor on phorge171 is OK: DNS OK: 0.276 seconds response time. phorge171.fsslc.wtnet returns 10.0.17.124
[18:43:12] <icinga-miraheze>	 PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:43:38] <icinga-miraheze>	 RECOVERY - db161 NTP time on db161 is OK: NTP OK: Offset 0.0003730356693 secs
[18:43:42] <icinga-miraheze>	 RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[18:43:42] <icinga-miraheze>	 RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[18:44:49] <icinga-miraheze>	 RECOVERY - mw202 PowerDNS Recursor on mw202 is OK: DNS OK: 2.041 seconds response time. mw202.fsslc.wtnet returns 10.0.20.163
[18:45:00] <icinga-miraheze>	 RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:45:10] <icinga-miraheze>	 RECOVERY - mw202 NTP time on mw202 is OK: NTP OK: Offset 0.001558691263 secs
[18:45:36] <icinga-miraheze>	 RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 28 Oct 2025 01:56:02 AM GMT +0000.
[18:45:38] <icinga-miraheze>	 RECOVERY - www.nandopedia.it - CloudFlare on sslhost is OK: OK - Certificate 'www.nandopedia.it' will expire on Sun 28 Sep 2025 01:33:25 AM GMT +0000.
[18:45:55] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[18:46:11] <icinga-miraheze>	 RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.145 seconds response time. db161.fsslc.wtnet returns 10.0.16.128
[18:46:13] <icinga-miraheze>	 RECOVERY - rdb151 NTP time on rdb151 is OK: NTP OK: Offset 0.001351803541 secs
[18:47:11] <icinga-miraheze>	 RECOVERY - rdb151 PowerDNS Recursor on rdb151 is OK: DNS OK: 0.029 seconds response time. rdb151.fsslc.wtnet returns 10.0.15.142
[18:47:12] <icinga-miraheze>	 RECOVERY - os201 Puppet on os201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:47:24] <icinga-miraheze>	 RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[18:49:35] <icinga-miraheze>	 RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:51:27] <icinga-miraheze>	 RECOVERY - matomo151 PowerDNS Recursor on matomo151 is OK: DNS OK: 0.028 seconds response time. matomo151.fsslc.wtnet returns 10.0.15.112
[18:51:30] <icinga-miraheze>	 RECOVERY - matomo151 NTP time on matomo151 is OK: NTP OK: Offset 0.001532435417 secs
[18:52:13] <icinga-miraheze>	 RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:52:34] <icinga-miraheze>	 RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:54:49] <icinga-miraheze>	 RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:56:44] <icinga-miraheze>	 RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[19:01:10] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures
[19:02:05] <icinga-miraheze>	 RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[19:06:28] <icinga-miraheze>	 RECOVERY - mw153 NTP time on mw153 is OK: NTP OK: Offset 0.002088993788 secs
[19:07:44] <icinga-miraheze>	 RECOVERY - mw153 PowerDNS Recursor on mw153 is OK: DNS OK: 0.281 seconds response time. mw153.fsslc.wtnet returns 10.0.15.140
[19:07:55] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:10:45] <icinga-miraheze>	 RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:15:11] <icinga-miraheze>	 RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:22:55] <icinga-miraheze>	 RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:39:42] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[19:47:24] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[19:47:50] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s
[19:58:37] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=danmachiwiki ./danmachifiles/ --comment='Importing images from danmachi.fandom.com ([[phorge:T14028|T14028]])' --sleep=1 (START)
[20:09:33] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all
[20:10:19] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 46s
[20:13:36] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:14:12] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importDump --wiki=danmachienwiki --no-updates --username-prefix=wikia:danmachi -- dungeonnideaiomotomeru_pages_full.xml (END - exit=0)
[20:14:13] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=danmachienwiki (START)
[20:14:20] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[20:14:43] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s
[20:27:04] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[20:31:59] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[21:41:31] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build has errored.
[21:54:23] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build has errored.
[22:13:11] <miraheze-github>	 miraheze/RequestSSL - Universal-Omega the build passed.
[22:19:59] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php importImages --wiki=danmachiwiki ./danmachifiles/ --comment='Importing images from danmachi.fandom.com ([[phorge:T14028|T14028]])' --sleep=1 (END - exit=0)
[22:25:07] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php rebuildall --wiki=danmachienwiki (END - exit=0)
[22:25:09] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initEditCount --wiki=danmachienwiki (START)
[22:25:10] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initEditCount --wiki=danmachienwiki (END - exit=0)
[22:25:11] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=danmachienwiki --update (START)
[22:25:12] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php initSiteStats --wiki=danmachienwiki --update (END - exit=0)
[22:25:13] <MirahezeLSBot>	 !log [skye@mwtask181] Finished import for danmachienwiki (XML: dungeonnideaiomotomeru_pages_full.xml; Images: None) (END - exit=0)