[00:09:39] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[01:09:38] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[01:39:39] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[02:39:37] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca
[03:09:39] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[03:19:32] <icinga-wikitide>	 PROBLEM - matomo21 HTTPS on matomo21 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.007 second response time
[03:22:51] <icinga-wikitide>	 RECOVERY - db21 Puppet on db21 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[03:23:23] <icinga-wikitide>	 RECOVERY - matomo21 HTTPS on matomo21 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.402 second response time
[03:24:58] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[03:26:44] <icinga-wikitide>	 PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[03:26:44] <icinga-wikitide>	 PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[03:27:42] <icinga-wikitide>	 PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.67, 6.98, 6.00
[03:31:37] <icinga-wikitide>	 RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.45, 6.40, 6.00
[03:35:30] <icinga-wikitide>	 PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.32, 6.81, 6.28
[03:37:28] <icinga-wikitide>	 RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.54, 5.99, 6.04
[03:48:44] <icinga-wikitide>	 PROBLEM - phorge21 Backups Phorge Static on phorge21 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/phorge-backup.log
[03:49:08] <icinga-wikitide>	 PROBLEM - phorge21 phorge-static.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error
[03:49:09] <icinga-wikitide>	 PROBLEM - phorge21 Puppet on phorge21 is UNKNOWN: NRPE: Unable to read output
[03:49:31] <icinga-wikitide>	 PROBLEM - phorge21 HTTPS on phorge21 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 666 bytes in 2.032 second response time
[03:49:51] <icinga-wikitide>	 PROBLEM - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5584 bytes in 2.036 second response time
[03:51:49] <icinga-wikitide>	 RECOVERY - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is OK: HTTP OK: HTTP/1.1 200 OK - 104780 bytes in 0.124 second response time
[03:53:12] <icinga-wikitide>	 RECOVERY - phorge21 Puppet on phorge21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:55:37] <icinga-wikitide>	 PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[03:55:59] <icinga-wikitide>	 PROBLEM - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4328 bytes in 2.035 second response time
[03:56:09] <icinga-wikitide>	 PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[03:59:58] <icinga-wikitide>	 RECOVERY - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is OK: HTTP OK: HTTP/1.1 200 OK - 103602 bytes in 0.105 second response time
[04:00:56] <icinga-wikitide>	 RECOVERY - phorge21 phorge-static.wikitide.org HTTPS on phorge21 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 58196 bytes in 0.228 second response time
[04:22:48] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca
[05:08:06] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[05:21:20] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[05:50:36] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca
[05:55:52] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[06:25:47] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca
[06:36:04] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[06:37:38] <icinga-wikitide>	 PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[06:37:44] <icinga-wikitide>	 RECOVERY - distrowiki.net - reverse DNS on sslhost is OK: SSL OK - distrowiki.net reverse DNS resolves to cp5.wikitide.net - CNAME FLAT
[06:41:43] <icinga-wikitide>	 PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['grannbo.ns.cloudflare.com.', 'rodrigo.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[06:46:37] <icinga-wikitide>	 PROBLEM - beta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[06:47:31] <icinga-wikitide>	 PROBLEM - mason.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[06:47:56] <icinga-wikitide>	 PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[06:48:00] <icinga-wikitide>	 RECOVERY - wikitide.net - reverse DNS on sslhost is OK: SSL OK - wikitide.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[06:49:07] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[06:50:50] <icinga-wikitide>	 PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns3.digitalocean.com.', 'ns1.digitalocean.com.', 'ns2.digitalocean.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[06:51:31] <icinga-wikitide>	 RECOVERY - sdiy.info - reverse DNS on sslhost is OK: SSL OK - sdiy.info reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[06:52:17] <icinga-wikitide>	 RECOVERY - www.wikitide.org - reverse DNS on sslhost is OK: SSL OK - www.wikitide.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[06:53:09] <icinga-wikitide>	 PROBLEM - meta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[06:55:42] <icinga-wikitide>	 RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[06:55:43] <icinga-wikitide>	 PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[06:55:52] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[07:01:58] <icinga-wikitide>	 PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[07:02:28] <icinga-wikitide>	 RECOVERY - greatamerica.wiki - reverse DNS on sslhost is OK: SSL OK - greatamerica.wiki reverse DNS resolves to cp2.wikitide.net - CNAME FLAT
[07:07:23] <icinga-wikitide>	 PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alpha.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca
[07:13:34] <icinga-wikitide>	 RECOVERY - hkrail.wiki - reverse DNS on sslhost is OK: SSL OK - hkrail.wiki reverse DNS resolves to cp2.wikitide.net - CNAME FLAT
[07:16:48] <icinga-wikitide>	 PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[07:17:29] <icinga-wikitide>	 PROBLEM - mason.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mason.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca
[07:17:29] <icinga-wikitide>	 PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.wixdns.net.', 'ns0.wixdns.net.'], 'CNAME': 'mw-lb.wikitide.org.'}
[07:17:50] <icinga-wikitide>	 PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.lophocmatngu.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[07:19:54] <icinga-wikitide>	 PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chevrine.com reverse DNS resolves to vps-d267e842.vps.ovh.ca
[07:21:06] <icinga-wikitide>	 RECOVERY - farthestfrontier.wiki - reverse DNS on sslhost is OK: SSL OK - farthestfrontier.wiki reverse DNS resolves to cp2.wikitide.net - CNAME FLAT
[07:25:02] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[07:25:50] <icinga-wikitide>	 RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[07:27:46] <icinga-wikitide>	 PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[07:28:55] <icinga-wikitide>	 RECOVERY - realitylongterms.studio - reverse DNS on sslhost is OK: SSL OK - realitylongterms.studio reverse DNS resolves to cp2.wikitide.net - CNAME FLAT
[07:31:58] <icinga-wikitide>	 PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hsck.lophocmatngu.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[07:32:13] <icinga-wikitide>	 RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[07:34:44] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[07:37:09] <icinga-wikitide>	 PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[07:39:04] <icinga-wikitide>	 RECOVERY - thegreatwar.uk - reverse DNS on sslhost is OK: SSL OK - thegreatwar.uk reverse DNS resolves to cp5.wikitide.net - CNAME FLAT
[07:39:17] <icinga-wikitide>	 PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lgbtqia.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca
[07:44:11] <icinga-wikitide>	 PROBLEM - beta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - beta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca
[07:44:43] <icinga-wikitide>	 RECOVERY - polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - polandballwiki.com reverse DNS resolves to cp5.wikitide.net - CNAME FLAT
[07:46:34] <icinga-wikitide>	 PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - projects.dmvpetridish.com reverse DNS resolves to vps-d267e842.vps.ovh.ca
[07:48:59] <icinga-wikitide>	 PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.digitalocean.com.', 'ns3.digitalocean.com.', 'ns1.digitalocean.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[07:50:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 2.51, 3.56, 4.00
[07:50:48] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[07:52:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 3.92, 4.07
[07:54:44] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.greatamerica.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[07:57:21] <icinga-wikitide>	 PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[07:57:39] <icinga-wikitide>	 PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[08:03:19] <icinga-wikitide>	 PROBLEM - wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[08:04:04] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[08:08:05] <icinga-wikitide>	 PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['grannbo.ns.cloudflare.com.', 'rodrigo.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[08:12:58] <icinga-wikitide>	 PROBLEM - beta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[08:15:39] <icinga-wikitide>	 PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns0.wixdns.net.', 'ns1.wixdns.net.'], 'CNAME': 'mw-lb.wikitide.org.'}
[08:15:40] <icinga-wikitide>	 PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['cleo.ns.cloudflare.com.', 'may.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[08:16:48] <icinga-wikitide>	 PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.polandballwiki.com reverse DNS resolves to vps-d267e842.vps.ovh.ca
[08:17:10] <icinga-wikitide>	 RECOVERY - lophocmatngu.wiki - reverse DNS on sslhost is OK: SSL OK - lophocmatngu.wiki reverse DNS resolves to cp5.wikitide.net - CNAME FLAT
[08:17:24] <icinga-wikitide>	 PROBLEM - mason.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[08:24:26] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[08:25:50] <icinga-wikitide>	 PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[08:27:14] <icinga-wikitide>	 RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[08:27:39] <icinga-wikitide>	 PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[08:27:52] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[08:32:30] <icinga-wikitide>	 RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[08:36:40] <icinga-wikitide>	 PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alpha.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca
[08:41:03] <icinga-wikitide>	 PROBLEM - thegreatwar.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thegreatwar.uk reverse DNS resolves to vps-d267e842.vps.ovh.ca
[08:44:44] <icinga-wikitide>	 PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - projects.dmvpetridish.com reverse DNS resolves to vps-d267e842.vps.ovh.ca
[08:49:07] <icinga-wikitide>	 RECOVERY - wiki.burning.art - reverse DNS on sslhost is OK: SSL OK - wiki.burning.art reverse DNS resolves to cp5.wikitide.net - CNAME OK
[08:49:32] <icinga-wikitide>	 PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns29.hichina.com.', 'dns30.hichina.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[08:54:08] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.greatamerica.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[08:55:54] <icinga-wikitide>	 PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[08:56:45] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[08:57:36] <icinga-wikitide>	 RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[08:59:18] <icinga-wikitide>	 PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[09:01:58] <icinga-wikitide>	 PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[09:06:25] <icinga-wikitide>	 PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[09:09:37] <icinga-wikitide>	 RECOVERY - thegreatwar.uk - reverse DNS on sslhost is OK: SSL OK - thegreatwar.uk reverse DNS resolves to cp5.wikitide.net - CNAME FLAT
[09:13:50] <icinga-wikitide>	 PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns0.wixdns.net.', 'ns1.wixdns.net.'], 'CNAME': 'mw-lb.wikitide.org.'}
[09:14:06] <icinga-wikitide>	 PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.farthestfrontier.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[09:16:48] <icinga-wikitide>	 PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[09:17:27] <icinga-wikitide>	 PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[09:19:32] <icinga-wikitide>	 PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.colleimadcat.com reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[09:23:51] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[09:25:48] <icinga-wikitide>	 RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[09:25:55] <icinga-wikitide>	 RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[09:31:58] <icinga-wikitide>	 PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hsck.lophocmatngu.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[09:32:03] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[09:38:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.60, 3.69, 3.97
[09:43:19] <icinga-wikitide>	 PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['cleo.ns.cloudflare.com.', 'may.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[09:44:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 3.85, 3.94
[09:47:49] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[09:49:33] <icinga-wikitide>	 PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns29.hichina.com.', 'dns30.hichina.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[09:58:06] <icinga-wikitide>	 PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[10:02:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.12, 3.70, 3.94
[10:06:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 4.16, 4.07
[10:08:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.36, 3.85, 3.96
[10:10:45] <icinga-wikitide>	 PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.75, 4.14, 4.05
[10:23:15] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.greatamerica.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca
[10:28:09] <icinga-wikitide>	 RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK
[10:29:05] <icinga-wikitide>	 RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[10:31:58] <icinga-wikitide>	 PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'}
[10:33:49] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[10:52:57] <icinga-wikitide>	 PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['john.ns.cloudflare.com.', 'vida.ns.cloudflare.com.'], 'CNAME': 'mw-lb.wikitide.org.'}
[12:31:17] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[14:13:31] <icinga-wikitide>	 PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.89, 6.50, 5.48
[14:15:31] <icinga-wikitide>	 RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.57, 5.71, 5.31
[15:05:24] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[15:34:44] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK
[17:27:34] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8aa89bc83a3c...13ad607d9da1
[17:27:36] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0313ad607 - varnish: use cloud2, cloud3, and cloud4 in debug acl
[17:29:04] <WikiTideRelay>	 <songngu.xyz> UO you there? Sorry for budging but like, it's 2 days and my wiki still like unusable
[17:29:09] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/13ad607d9da1...878f1efaf5df
[17:29:10] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03878f1ef - prometheus: use cloud2, cloud3, and cloud4 IPs
[17:31:29] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/878f1efaf5df...b97d6598fa92
[17:31:30] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03b97d659 - Add cloud3 and cloud4 to bastion public ips
[17:40:58] <botifico-49a823b>	 [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/5d81678a9754...23013e42b823
[17:40:59] <botifico-49a823b>	 [02WikiTideOrg/dns] 07Universal-Omega 0323013e4 - Update services
[17:43:16] <botifico-49a823b>	 [02WikiTideOrg/WikiTideDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/WikiTideDebug/compare/f0804aa9a97a...fd153c131ed3
[17:43:18] <botifico-49a823b>	 [02WikiTideOrg/WikiTideDebug] 07Universal-Omega 03fd153c1 - Update servers
[17:45:24] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/6e80bc7a1cf2...1317df6b84d9
[17:45:25] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 031317df6 - Use test21
[17:46:22] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[17:47:30] <WikiTideLSBot>	 !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21]
[17:47:31] <WikiTideLSBot>	 !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw21.wikitide.net
[17:47:33] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[17:47:34] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[17:52:51] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/1317df6b84d9...816397c0adb8
[17:52:54] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03816397c - Update servers
[17:53:54] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[17:56:02] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/b97d6598fa92...b7b0348955c2
[17:56:05] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03b7b0348 - Use test21 and update versions
[17:58:19] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 036 commits to 03raid [+2/-2/±8] 13https://github.com/WikiTideOrg/puppet/compare/f3e65fdae015...7791675e2fea
[17:58:20] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 037791675 - Merge branch 'master' into raid
[17:58:22] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39
[17:58:54] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[17:58:56] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+1/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/55213bb2eb1a...d8abef3e7c73
[17:58:57] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03d8abef3 - Use test21
[17:59:42] <wikitide-github>	 WikiTideOrg/puppet - Universal-Omega the build passed.
[18:00:38] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 036 commits to 03mcrouter [+1/-1/±9] 13https://github.com/WikiTideOrg/puppet/compare/d8abef3e7c73...23037d179cfc
[18:00:39] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0323037d1 - Merge branch 'master' into mcrouter
[18:00:42] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[18:02:41] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 036 commits to 03thumbor [+2/-2/±7] 13https://github.com/WikiTideOrg/puppet/compare/78ef31e96f82...b6aadea1c1ca
[18:02:44] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03b6aadea - Merge branch 'master' into thumbor
[18:02:45] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#23: Add thumbor and haproxy support - 13https://github.com/WikiTideOrg/puppet/pull/23
[18:03:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 032 commits to 03cloud-monitoring [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/ed4e3d67d504...8665708b6976
[18:03:33] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 037ce5487 - cloud: add monitoring
[18:03:35] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 038665708 - Use lookup
[18:03:39] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#21: cloud: add monitoring - 13https://github.com/WikiTideOrg/puppet/pull/21
[18:03:48] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 036 commits to 03reports [+2/-2/±8] 13https://github.com/WikiTideOrg/puppet/compare/b674d2e706a1...99c88d6f671c
[18:03:49] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0399c88d6 - Merge branch 'master' into reports
[18:03:52] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#18: Add reports module - 13https://github.com/WikiTideOrg/puppet/pull/18
[18:04:03] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 032 commits to 03phpldapadmin [+22/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a3480cf632db...e0d32d594acd
[18:04:04] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03b2f1b3d - Add phpldapadmin module
[18:04:06] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07github-actions 03e0d32d5 - CI: lint puppet code to standards Check commit and GitHub actions for more details
[18:04:07] <wikitide-github>	 WikiTideOrg/puppet - Universal-Omega the build passed.
[18:04:08] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#15: Add phpldapadmin module - 13https://github.com/WikiTideOrg/puppet/pull/15
[18:07:13] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/816397c0adb8...eeed5639599d
[18:07:15] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03eeed563 - Use mem21
[18:08:13] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:11:00] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/eeed5639599d...14d4a769b04e
[18:11:01] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 0314d4a76 - Update servers
[18:12:02] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:13:33] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 032 commits to 03lists [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/964e65ebaa48...d05771670f2a
[18:13:35] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 033f93a94 - Use unsuffixed DB lists
[18:13:37] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03d057716 - Fix
[18:13:40] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#22: Use unsuffixed DB lists - 13https://github.com/WikiTideOrg/mw-config/pull/22
[18:14:30] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:17:03] <WikiTideLSBot>	 !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21]
[18:17:05] <WikiTideLSBot>	 !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw22.wikitide.net
[18:17:05] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:17:07] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:17:26] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#22: Use unsuffixed DB lists - 13https://github.com/WikiTideOrg/mw-config/pull/22
[18:17:29] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03lists [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/d05771670f2a...27043985b051
[18:17:32] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 032704398 - Remove suffixed lists
[18:18:22] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:18:56] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 032 commits to 03mcrouter [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/440b6d4dcab8...f3fda247ef4a
[18:18:57] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 035437054 - Use mcrouter
[18:19:00] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03f3fda24 - Use mcrouter
[18:19:01] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#26: Use mcrouter - 13https://github.com/WikiTideOrg/mw-config/pull/26
[18:19:13] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 032 commits to 03thumbor [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/31c057c8c760...e529fd567d8c
[18:19:16] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 037984ddf - Add thumbor config
[18:19:17] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03e529fd5 - Fix
[18:19:18] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#17: Add thumbor config - 13https://github.com/WikiTideOrg/mw-config/pull/17
[18:19:51] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:20:16] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:20:46] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#24: Update Wikibase.php for lhmnwikitide/lhmndatawikitide - 13https://github.com/WikiTideOrg/mw-config/pull/24
[18:21:03] <botifico-49a823b>	 [02mw-config] 07Universal-Omega edited pull request 03#24: Update Wikibase.php for lhmnwiki/lhmndatawiki - 13https://github.com/WikiTideOrg/mw-config/pull/24
[18:21:45] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:42:55] <WikiTideLSBot>	 !log [universalomega@db21] DROP DATABASE premiumtest*wiki;
[18:42:57] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:43:13] <WikiTideLSBot>	 !log [universalomega@db21] DROP DATABASE wfphorge*;
[18:43:15] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:47:51] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[18:47:53] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:47:54] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 3s
[18:47:56] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:48:36] <botifico-49a823b>	 [02mw-config] 07Universal-Omega closed pull request 03#22: Use unsuffixed DB lists - 13https://github.com/WikiTideOrg/mw-config/pull/22
[18:48:37] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/14d4a769b04e...255475739e54
[18:48:38] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 032554757 - Use unsuffixed DB lists
[18:48:40] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03lists
[18:48:41] <botifico-49a823b>	 [02mw-config] 07Universal-Omega deleted branch 03lists - 13https://github.com/WikiTideOrg/mw-config
[18:48:44] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03revert-22-lists [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/1050c5e28209
[18:48:47] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 031050c5e - Revert "Use unsuffixed DB lists" This reverts commit 255475739e5456d5e0dac6948e43fffd3852e0ed.
[18:48:48] <botifico-49a823b>	 [02mw-config] 07Universal-Omega created branch 03revert-22-lists - 13https://github.com/WikiTideOrg/mw-config
[18:48:51] <botifico-49a823b>	 [02mw-config] 07Universal-Omega opened pull request 03#27: Revert "Use unsuffixed DB lists" - 13https://github.com/WikiTideOrg/mw-config/pull/27
[18:48:51] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[18:48:52] <WikiTideLSBot>	 !log [universalomega@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw21.wikitide.net
[18:48:53] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:48:54] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:49:07] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[18:49:09] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:49:09] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s
[18:49:11] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[18:49:28] <botifico-49a823b>	 [02mw-config] 07Universal-Omega closed pull request 03#27: Revert "Use unsuffixed DB lists" - 13https://github.com/WikiTideOrg/mw-config/pull/27
[18:49:30] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03revert-22-lists
[18:49:31] <botifico-49a823b>	 [02mw-config] 07Universal-Omega deleted branch 03revert-22-lists - 13https://github.com/WikiTideOrg/mw-config
[18:49:33] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[18:49:46] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[19:17:18] <icinga-wikitide>	 PROBLEM - mon21 Check correctness of the icinga configuration on mon21 is CRITICAL: Icinga configuration contains errors
[19:27:29] <icinga-wikitide>	 PROBLEM - cloud3 MegaRAID on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[19:27:54] <icinga-wikitide>	 PROBLEM - cloud3 get_raid_status_megacli on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[19:28:34] <icinga-wikitide>	 PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[19:28:49] <icinga-wikitide>	 PROBLEM - cloud3 MegaRAID on cloud3 is UNKNOWN: NRPE: Unable to read output
[19:29:48] <icinga-wikitide>	 PROBLEM - cloud3 get_raid_status_megacli on cloud3 is UNKNOWN: NRPE: Unable to read output
[19:30:32] <icinga-wikitide>	 PROBLEM - cloud3 Puppet on cloud3 is WARNING: WARNING: Puppet is currently disabled, message: Networking issues, last run 1 day ago with 0 failures
[19:42:11] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b7b0348955c2...6b700718b00e
[19:42:14] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 036b70071 - Update CSP (add *.wikitide.net) Should fix T85
[20:09:27] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query.
[20:38:52] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[21:32:09] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6b700718b00e...9dbd904cbe50
[21:32:10] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 039dbd904 - mem21: increase memcached size and update options
[21:32:34] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 033 commits to 03mcrouter [+0/-0/±3] 13https://github.com/WikiTideOrg/puppet/compare/23037d179cfc...c6a8a14bd72a
[21:32:35] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03c6a8a14 - Merge branch 'master' into mcrouter
[21:32:36] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[21:39:28] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/255475739e54...b5adc86b08af
[21:39:31] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03b5adc86 - Update memcached config
[21:39:49] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[21:39:51] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:39:51] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s
[21:39:53] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:40:22] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[21:42:42] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/b5adc86b08af...68b73ea9bd17
[21:42:45] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 0368b73ea - Set $wgDLPMaxCacheTime
[21:43:36] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[21:47:21] <WikiTideLSBot>	 !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21]
[21:47:23] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:47:23] <WikiTideLSBot>	 !log [@jobrunner21] finished deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21] - SUCCESS in 1s
[21:47:25] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:48:00] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/68b73ea9bd17...7fbe674e28e0
[21:48:01] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 037fbe674 - Update more caching config
[21:48:58] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[21:58:01] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/7fbe674e28e0...03ca29b09e3f
[21:58:02] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 0303ca29b - More caching
[21:58:25] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[21:58:27] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:58:28] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s
[21:58:30] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:58:56] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:01:47] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/03ca29b09e3f...034d4e910660
[22:01:50] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03034d4e9 - Remove claimTTL
[22:02:24] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[22:02:26] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:02:27] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s
[22:02:29] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:02:48] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:08:04] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/034d4e910660...a92d8e156a1c
[22:08:05] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03a92d8e1 - Add srvsByBucket to lock config
[22:08:14] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[22:08:16] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:08:16] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s
[22:08:18] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:09:01] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:10:29] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 038 commits to 03mcrouter [+0/-0/±10] 13https://github.com/WikiTideOrg/mw-config/compare/f3fda247ef4a...664cfb965531
[22:10:31] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03664cfb9 - Merge branch 'main' into mcrouter
[22:10:34] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#26: Use mcrouter - 13https://github.com/WikiTideOrg/mw-config/pull/26
[22:11:30] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:12:42] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 032 commits to 03thumbor [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/e529fd567d8c...d3ea9c8e20dc
[22:12:43] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 0303dc6bb - Add thumbor config
[22:12:44] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03d3ea9c8 - Fix
[22:12:47] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#17: Add thumbor config - 13https://github.com/WikiTideOrg/mw-config/pull/17
[22:13:43] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:16:13] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/9dbd904cbe50...faac1b4bf025
[22:16:16] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03faac1b4 - Update purgeParserCache cron
[22:17:41] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a92d8e156a1c...4d70f0936cfc
[22:17:44] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 034d70f09 - Use pc1
[22:21:37] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03db-mainstash [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/db4889c0f938
[22:21:40] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 03db4889c - Use db-mainstash
[22:21:43] <botifico-49a823b>	 [02mw-config] 07Universal-Omega created branch 03db-mainstash - 13https://github.com/WikiTideOrg/mw-config
[22:21:44] <botifico-49a823b>	 [02mw-config] 07Universal-Omega opened pull request 03#28: Use db-mainstash - 13https://github.com/WikiTideOrg/mw-config/pull/28
[22:22:34] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:25:16] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/faac1b4bf025...746f805a0470
[22:25:18] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03746f805 - Add mainstash grants
[22:35:12] <WikiTideLSBot>	 !log [universalomega@db21] Create mainstash database
[22:35:14] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:36:35] <botifico-49a823b>	 [02mw-config] 07Universal-Omega closed pull request 03#28: Use db-mainstash - 13https://github.com/WikiTideOrg/mw-config/pull/28
[22:36:36] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/4d70f0936cfc...2f94a8eea5b7
[22:36:38] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 032f94a8e - Use db-mainstash
[22:36:40] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03db-mainstash
[22:36:43] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[22:36:43] <botifico-49a823b>	 [02mw-config] 07Universal-Omega deleted branch 03db-mainstash - 13https://github.com/WikiTideOrg/mw-config
[22:36:45] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:36:46] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 3s
[22:36:48] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[22:37:29] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[22:41:41] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 033 commits to 03mcrouter [+0/-0/±3] 13https://github.com/WikiTideOrg/mw-config/compare/664cfb965531...5be71e707fa2
[22:41:42] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 035be71e7 - Merge branch 'main' into mcrouter
[22:41:44] <botifico-49a823b>	 [02mw-config] 07Universal-Omega synchronize pull request 03#26: Use mcrouter - 13https://github.com/WikiTideOrg/mw-config/pull/26
[22:42:42] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[23:03:00] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/746f805a0470...9d1743f8ca07
[23:03:01] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 039d1743f - Use weight 100
[23:34:16] <icinga-wikitide>	 PROBLEM - db21 NTP time on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:34:21] <icinga-wikitide>	 PROBLEM - db21 Backups SQL wtglobal on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:34:55] <icinga-wikitide>	 PROBLEM - db21 Disk Space on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:35:16] <icinga-wikitide>	 PROBLEM - db21 ferm_active on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:35:34] <icinga-wikitide>	 PROBLEM - db21 conntrack_table_size on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:35:41] <icinga-wikitide>	 PROBLEM - db21 PowerDNS Recursor on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:35:46] <icinga-wikitide>	 PROBLEM - db21 Puppet on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:35:58] <icinga-wikitide>	 PROBLEM - db21 Current Load on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:36:03] <icinga-wikitide>	 PROBLEM - db21 Backups SQL on db21 is CRITICAL: connect to address 10.0.2.3 port 5666: Connection refusedconnect to host 10.0.2.3 port 5666: Connection refused
[23:37:11] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/2f94a8eea5b7...8948ebc184fb
[23:37:12] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega 038948ebc - Enable WarmParsoidParserCache
[23:37:34] <icinga-wikitide>	 RECOVERY - db21 conntrack_table_size on db21 is OK: OK: nf_conntrack is 0 % full
[23:37:41] <WikiTideLSBot>	 !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21]
[23:37:41] <icinga-wikitide>	 RECOVERY - db21 PowerDNS Recursor on db21 is OK: DNS OK: 0.212 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151
[23:37:43] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:37:44] <WikiTideLSBot>	 !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s
[23:37:45] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:37:47] <icinga-wikitide>	 RECOVERY - db21 Puppet on db21 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[23:37:58] <icinga-wikitide>	 RECOVERY - db21 Current Load on db21 is OK: LOAD OK - total load average: 6.47, 6.29, 5.96
[23:38:03] <icinga-wikitide>	 RECOVERY - db21 Backups SQL on db21 is OK: FILE_AGE OK: /var/log/backup-logs/sql-backup.log is 239979 seconds old and 1356 bytes
[23:38:11] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[23:38:20] <icinga-wikitide>	 RECOVERY - db21 NTP time on db21 is OK: NTP OK: Offset 0.004129201174 secs
[23:38:21] <icinga-wikitide>	 RECOVERY - db21 Backups SQL wtglobal on db21 is OK: FILE_AGE OK: /var/log/backup-logs/sql-wtglobal-backup-weekly.log is 153490 seconds old and 1087 bytes
[23:38:55] <icinga-wikitide>	 RECOVERY - db21 Disk Space on db21 is OK: DISK OK - free space: / 191891MiB (40% inode=99%);
[23:39:16] <icinga-wikitide>	 RECOVERY - db21 ferm_active on db21 is OK: OK ferm input default policy is set
[23:39:17] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --procs 256 (START)
[23:39:19] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:39:52] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --procs 256 (END - exit=2)
[23:39:54] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:40:45] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=polandballwiki --procs 256 (START)
[23:40:47] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:41:33] <icinga-wikitide>	 PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.52, 8.08, 6.69
[23:41:43] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (START)
[23:41:45] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:08] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (END - exit=0)
[23:42:10] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:20] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (START)
[23:42:22] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (END - exit=0)
[23:42:22] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:24] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (START)
[23:42:24] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:26] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (END - exit=0)
[23:42:26] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:27] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (START)
[23:42:28] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:29] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=rosettacodewiki --procs 500 (END - exit=0)
[23:42:30] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:42:31] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:43:29] <WikiTideLSBot>	 !log [agent@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=polandballwiki --procs 256 (END - exit=0)
[23:43:31] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:51:31] <icinga-wikitide>	 PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.40, 7.74, 7.72
[23:59:31] <icinga-wikitide>	 RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.37, 5.71, 6.68