[00:01:05] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [00:06:24] PROBLEM - swiftobject1 Current Load on swiftobject1 is CRITICAL: LOAD CRITICAL - total load average: 7.05, 4.27, 3.51 [00:06:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:08:24] PROBLEM - swiftobject1 Current Load on swiftobject1 is WARNING: LOAD WARNING - total load average: 3.27, 3.97, 3.50 [00:10:24] RECOVERY - swiftobject1 Current Load on swiftobject1 is OK: LOAD OK - total load average: 0.88, 2.81, 3.13 [00:28:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [00:36:22] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [01:06:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:12:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:16:24] PROBLEM - swiftobject1 Current Load on swiftobject1 is CRITICAL: LOAD CRITICAL - total load average: 8.25, 5.70, 3.63 [01:22:24] RECOVERY - swiftobject1 Current Load on swiftobject1 is OK: LOAD OK - total load average: 0.68, 2.88, 3.13 [01:28:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:36:23] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [01:38:55] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/0ba6698f5aa9...c831abbcf0a9 [01:38:57] [02WikiTideOrg/mw-config] 07AgentIsai 03c831abb - End maintenance [01:39:31] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/c831abbcf0a9...b24decfb2c2f [01:39:33] [02WikiTideOrg/mw-config] 07AgentIsai 03b24decf - Remove old suffix [01:39:53] WikiTideOrg/mw-config - AgentIsai the build passed. [01:40:27] WikiTideOrg/mw-config - AgentIsai the build passed. [01:40:54] !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [01:40:55] !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw21.wikitide.net [01:41:10] !log [@jobrunner21] starting deploy of {'folders': '1.40/extensions/WikiTideMagic', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [01:41:12] !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw22.wikitide.net [01:41:16] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:41:19] !log [@jobrunner21] starting deploy of {'folders': '1.41/extensions/WikiTideMagic', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [01:41:20] !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw22.wikitide.net [01:41:21] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:41:43] !log [agent@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [01:41:45] !log [agent@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [01:42:14] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [02:13:50] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 033 commits to 03suffix [+0/-0/±3] 13https://github.com/WikiTideOrg/mw-config/compare/e3f42a828fe0...d8192a6e4bfd [02:13:51] [02WikiTideOrg/mw-config] 07Universal-Omega 03d8192a6 - Merge branch 'main' into suffix [02:13:53] [02mw-config] 07Universal-Omega synchronize pull request 03#25: Use 'wiki' suffix - 13https://github.com/WikiTideOrg/mw-config/pull/25 [02:14:11] [02mw-config] 07Universal-Omega closed pull request 03#25: Use 'wiki' suffix - 13https://github.com/WikiTideOrg/mw-config/pull/25 [02:14:13] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±8] 13https://github.com/WikiTideOrg/mw-config/compare/b24decfb2c2f...64df03391bd8 [02:14:14] [02WikiTideOrg/mw-config] 07Universal-Omega 0364df033 - Use 'wiki' suffix [02:14:16] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03suffix [02:14:17] [02mw-config] 07Universal-Omega deleted branch 03suffix - 13https://github.com/WikiTideOrg/mw-config [02:14:49] WikiTideOrg/mw-config - Universal-Omega the build passed. [02:15:07] WikiTideOrg/mw-config - Universal-Omega the build passed. [02:16:58] !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21] [02:16:59] !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw21.wikitide.net [02:36:14] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:44:45] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/invalidateUserSessions.php --wiki=metawiki --user NoLongerOnTheBall (END - exit=0) [02:47:05] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/invalidateUserSessions.php --wiki=metawiki --user Universal Omega (END - exit=0) [02:47:16] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/invalidateUserSessions.php --wiki=metawiki --user Universal_Omega (END - exit=0) [02:48:43] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/64df03391bd8...33468d10b79c [02:48:44] [02WikiTideOrg/mw-config] 07Universal-Omega 0333468d1 - Bump wgAuthenticationTokenVersion [02:48:58] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [02:49:00] !log [universalomega@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw21.wikitide.net [02:49:15] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [02:49:17] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [02:49:34] WikiTideOrg/mw-config - Universal-Omega the build passed. [02:52:01] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/33468d10b79c...6039a1794640 [02:52:04] [02WikiTideOrg/mw-config] 07Universal-Omega 036039a17 - Change prefix [02:52:08] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [02:52:11] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [02:52:55] WikiTideOrg/mw-config - Universal-Omega the build passed. [02:53:33] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/6039a1794640...9426b47d49d6 [02:53:36] [02WikiTideOrg/mw-config] 07Universal-Omega 039426b47 - Revert [02:53:37] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [02:53:39] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [02:54:25] WikiTideOrg/mw-config - Universal-Omega the build passed. [03:00:15] RECOVERY - db1 Backups SQL on db1 is OK: FILE_AGE OK: /var/log/backup-logs/sql-backup.log is 14 seconds old and 273 bytes [03:01:51] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/9426b47d49d6...1f41671dbbed [03:01:54] [02WikiTideOrg/mw-config] 07Universal-Omega 031f41671 - Try mem for session cache [03:01:59] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:02:02] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 3s [03:04:28] WikiTideOrg/mw-config - Universal-Omega the build passed. [03:06:13] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [03:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [03:07:41] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:20:16] !log [universalomega@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [03:22:07] !log [universalomega@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=2) [03:23:32] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=2) [03:24:23] !log [universalomega@jobrunner21] starting deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:24:25] !log [universalomega@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw22.wikitide.net [03:24:35] !log [universalomega@jobrunner21] starting deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:24:37] !log [universalomega@jobrunner21] finished deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 1s [03:24:52] !log [universalomega@jobrunner21] starting deploy of {'force': True, 'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:25:19] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rosettacodewiki (END - exit=0) [03:25:28] !log [universalomega@jobrunner21] starting deploy of {'force': True, 'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:25:29] !log [universalomega@jobrunner21] finished deploy of {'force': True, 'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 1s [03:26:36] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:27:09] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:28:05] !log [universalomega@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [03:29:36] !log [universalomega@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=2) [03:30:37] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4masonwiki (END - exit=0) [03:31:00] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4betawiki (END - exit=0) [03:31:31] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=sagan4alphawiki (END - exit=0) [03:31:48] !log [universalomega@jobrunner21] starting deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:31:51] !log [universalomega@jobrunner21] finished deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [03:37:26] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [03:44:10] !log [universalomega@jobrunner21] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=farthestfrontierwiki (END - exit=0) [03:44:17] !log [universalomega@jobrunner21] starting deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [03:44:19] !log [universalomega@jobrunner21] finished deploy of {'force': True, 'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [03:53:51] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/1f41671dbbed...9add30316e86 [03:53:52] [02WikiTideOrg/mw-config] 07AgentIsai 039add303 - Revert [03:54:43] WikiTideOrg/mw-config - AgentIsai the build passed. [03:58:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [03:59:03] @agentisai memcached should be used for session anyway, regardless of session issues, whether it actually fixed them or not (which it did for me). I'm going to start work factoring redis session out and it'll eventually be switched to Kafka, but using memcached in the meantime with reduce disruption. [03:59:36] memcached doesn't seem to have solved anything [04:00:24] It did for me. The minute I deployed it I could login, but regardless if issues are fixed I'd rather fix the underlying issue while on memcached than redis which is deprecated here. [04:00:56] I'm able to log in but can't edit because of loss of session data [04:01:45] I figured it out [04:01:48] It's ferm [04:01:56] Either way, I'll soon redis will probably be removed if I find time to work on Kafka. Using memcached in the meantime just makes switching session easier [04:01:58] mw22 isn't registering in ferm on either Redis or memcached [04:03:05] Oh [04:06:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:15:24] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/868c78fbb9b4...195b5c49ed69 [04:15:25] [02WikiTideOrg/puppet] 07AgentIsai 03195b5c4 - Fix MW hostname [04:16:42] !log [@jobrunner21] starting deploy of {'config': True, 'versions': ['']} to [mw21, mw22, jobrunner21] [04:16:45] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [04:17:01] !log [@jobrunner21] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@mw22.wikitide.net [04:17:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [04:28:09] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2572 bytes in 0.120 second response time [04:28:12] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2569 bytes in 0.392 second response time [04:28:13] PROBLEM - cp6 HTTPS on cp6 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.569 second response time [04:28:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:28:47] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.309 second response time [04:30:06] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2918 bytes in 0.195 second response time [04:30:09] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2915 bytes in 0.746 second response time [04:30:10] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2917 bytes in 1.237 second response time [04:30:47] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2939 bytes in 0.576 second response time [04:36:23] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:28:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [05:29:50] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:58:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:06:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [07:28:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [07:58:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [08:28:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [08:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:01:06] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:06:23] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [09:12:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:31:03] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [09:36:14] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:42:16] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [09:58:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:06:13] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [10:36:14] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:42:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:58:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [11:03:25] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [11:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [11:28:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [11:33:10] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [11:36:09] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [11:58:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [12:02:57] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:12:14] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [12:28:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:02:28] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [13:12:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:32:15] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:06:23] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [14:12:16] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [14:28:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [14:31:46] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [14:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:42:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [15:01:33] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [15:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [15:12:18] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [15:58:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:01:04] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [16:12:19] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:42:16] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [17:01:06] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:06:23] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [17:31:01] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [17:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:58:38] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [18:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [18:28:39] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [19:32:43] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/9add30316e86...a56cf73229f4 [19:32:46] [02WikiTideOrg/mw-config] 07Universal-Omega 03a56cf73 - No redis lock manager for now [19:33:27] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [19:33:29] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:33:33] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 5s [19:33:35] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:33:40] WikiTideOrg/mw-config - Universal-Omega the build passed. [19:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [19:45:54] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 7.06, 6.81, 3.26 [19:47:48] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 1.38, 4.78, 2.92 [19:59:29] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a56cf73229f4...a783c100dd91 [19:59:32] [02WikiTideOrg/mw-config] 07Universal-Omega 03a783c10 - Use redisLockManager again [19:59:44] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [19:59:47] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:59:50] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 5s [19:59:52] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:00:24] WikiTideOrg/mw-config - Universal-Omega the build passed. [20:01:06] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [20:06:21] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [20:26:43] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a783c100dd91...44e5134d4585 [20:26:44] [02WikiTideOrg/mw-config] 07Universal-Omega 0344e5134 - Disable PageProperties [20:27:07] !log [universalomega@jobrunner21] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [20:27:09] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:27:10] !log [universalomega@jobrunner21] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 3s [20:27:12] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:27:36] WikiTideOrg/mw-config - Universal-Omega the build passed. [20:32:42] !log [universalomega@jobrunner21] starting deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [20:32:44] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:32:44] !log [universalomega@jobrunner21] finished deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [20:32:47] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:34:05] !log [universalomega@jobrunner21] starting deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [20:34:07] !log [universalomega@jobrunner21] finished deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 1s [20:34:07] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:34:09] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:34:59] !log [universalomega@jobrunner21] starting deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] [20:35:01] !log [universalomega@jobrunner21] finished deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw21, mw22, jobrunner21] - SUCCESS in 1s [20:35:02] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:35:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [20:37:15] !log [universalomega@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/pageproperties.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/toggleExtension.php pageproperties --disable (START) [20:37:17] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:37:23] !log [universalomega@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/pageproperties.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/toggleExtension.php pageproperties --disable (END - exit=0) [20:37:25] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:38:51] [02puppet] 07Universal-Omega closed pull request 03#43: Use wmgExternalDataCredsCslmodswiki - 13https://github.com/WikiTideOrg/puppet/pull/43 [20:38:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/195b5c49ed69...0ad6c16877af [20:38:57] [02WikiTideOrg/puppet] 07Universal-Omega 030ad6c16 - Use wmgExternalDataCredsCslmodswiki Also just use passwords::mediawiki::secretkey [20:39:00] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03wmgExternalDataCredsCslmodswiki [20:39:01] [02puppet] 07Universal-Omega deleted branch 03wmgExternalDataCredsCslmodswiki - 13https://github.com/WikiTideOrg/puppet [20:44:14] [02WikiTideOrg/puppet] 07Universal-Omega pushed 034 commits to 03raid [+0/-0/±8] 13https://github.com/WikiTideOrg/puppet/compare/4642b468a454...6f1a4efe1ff9 [20:44:16] [02WikiTideOrg/puppet] 07Universal-Omega 036f1a4ef - Merge branch 'master' into raid [20:44:18] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [20:45:56] WikiTideOrg/puppet - Universal-Omega the build has errored. [20:48:06] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [20:48:08] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6f1a4efe1ff9...8547d12481a7 [20:48:10] [02WikiTideOrg/puppet] 07Universal-Omega 038547d12 - Fix [20:49:31] WikiTideOrg/puppet - Universal-Omega the build has errored. [20:58:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-d267e842.vps.ovh.ca [21:01:29] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:01:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8547d12481a7...672f25ce93cf [21:01:34] [02WikiTideOrg/puppet] 07Universal-Omega 03672f25c - Update [21:03:16] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:03:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/672f25ce93cf...d3887e43b7d5 [21:03:18] [02WikiTideOrg/puppet] 07Universal-Omega 03d3887e4 - -md [21:03:19] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:04:46] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:05:01] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:05:04] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d3887e43b7d5...a5be595a0d38 [21:05:07] [02WikiTideOrg/puppet] 07Universal-Omega 03a5be595 - - [21:06:00] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:06:02] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a5be595a0d38...0dde86412474 [21:06:03] [02WikiTideOrg/puppet] 07Universal-Omega 030dde864 - - [21:06:17] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [21:06:33] [02WikiTideOrg/puppet] 07github-actions[bot] pushed 031 commit to 03raid [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/0dde86412474...436e0a278bb2 [21:06:36] [02WikiTideOrg/puppet] 07github-actions 03436e0a2 - CI: lint puppet code to standards Check commit and GitHub actions for more details [21:06:37] [02puppet] 07github-actions[bot] synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:07:32] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:12:44] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:12:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/436e0a278bb2...0157e4559d8b [21:12:49] [02WikiTideOrg/puppet] 07Universal-Omega 030157e45 - Fix [21:14:07] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:14:45] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:14:47] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0157e4559d8b...5b7164c40904 [21:14:50] [02WikiTideOrg/puppet] 07Universal-Omega 035b7164c - Fix [21:16:04] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [21:16:06] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5b7164c40904...0dfb0b16817b [21:16:08] [02WikiTideOrg/puppet] 07Universal-Omega 030dfb0b1 - Fix [21:16:18] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:17:26] WikiTideOrg/puppet - Universal-Omega the build has errored. [21:31:05] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [21:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:06:21] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:22] PROBLEM - wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:22] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chevrine.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:22] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:09:22] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:09:23] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:23] PROBLEM - cp2 PowerDNS Recursor on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:23] PROBLEM - cp5 Nginx Backend for puppet21 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:24] PROBLEM - cp5 Nginx Backend for mw22 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:24] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:09:24] PROBLEM - cp6 Nginx Backend for puppet21 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:09:24] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:09:25] PROBLEM - cp2 Nginx Backend for phorge21 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:25] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:09:26] PROBLEM - wikitide.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.net reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:09:26] PROBLEM - cp2 Nginx Backend for mail21 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:27] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alpha.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:27] PROBLEM - jobrunner21 MediaWiki Rendering on jobrunner21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.317 second response time [22:09:28] PROBLEM - cp4 ferm_active on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:09:29] PROBLEM - swiftac31 Swift Account Service on swiftac31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:29] PROBLEM - cloud3 conntrack_table_size on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:09:29] PROBLEM - farthestfrontier.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - farthestfrontier.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:30] PROBLEM - cloud3 NTP time on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:09:30] PROBLEM - cp5 conntrack_table_size on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:31] PROBLEM - cp5 Nginx Backend for matomo21 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:32] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:09:32] PROBLEM - cp3 conntrack_table_size on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:32] PROBLEM - cp3 Nginx Backend for test21 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:33] PROBLEM - cp3 Nginx Backend for mw21 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:33] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 15.235.167.159: 1 [22:09:34] PROBLEM - swiftproxy21 HTTPS on swiftproxy21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:34] PROBLEM - wikitide.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikitide.com could not be found [22:09:35] PROBLEM - cp1 SSH on cp1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:35] PROBLEM - cp2 Nginx Backend for puppet21 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:36] PROBLEM - cp5 Nginx Backend for mail21 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:37] PROBLEM - os21 Current Load on os21 is CRITICAL: LOAD CRITICAL - total load average: 4.45, 3.99, 3.84 [22:09:37] PROBLEM - cp6 Disk Space on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:09:37] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lgbtqia.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:38] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - beta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:38] PROBLEM - swiftobject31 SSH on swiftobject31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:39] PROBLEM - mail21 webmail.wikitide.net HTTPS on mail21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 289 bytes in 0.008 second response time [22:09:39] PROBLEM - ldap21 LDAP on ldap21 is CRITICAL: Could not bind to the LDAP server [22:09:40] PROBLEM - cp3 Nginx Backend for puppet21 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:40] PROBLEM - cloud2 Disk Space on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:09:41] PROBLEM - cp3 ferm_active on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:41] PROBLEM - cp6 Nginx Backend for mail21 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:09:42] PROBLEM - db21 Puppet on db21 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 day ago with 0 failures [22:09:42] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:09:43] PROBLEM - cp2 ferm_active on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:43] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:44] PROBLEM - ldap21 Current Load on ldap21 is CRITICAL: LOAD CRITICAL - total load average: 4.42, 3.99, 3.84 [22:09:45] PROBLEM - cp5 ferm_active on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:45] PROBLEM - cp5 Nginx Backend for test21 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:09:45] PROBLEM - mem21 APT on mem21 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [22:09:46] PROBLEM - puppet21 Backups SSLKeys on puppet21 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sslkeys-backup.log [22:09:46] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.farthestfrontier.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:09:47] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:48] PROBLEM - cp2 APT on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:48] PROBLEM - wiki.burning.art - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.burning.art reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:09:48] PROBLEM - os21 HTTPS on os21 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 146 bytes in 0.006 second response time [22:09:49] PROBLEM - cp4 PowerDNS Recursor on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:09:49] PROBLEM - cloud4 conntrack_table_size on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:09:50] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:50] PROBLEM - cloud2 NTP time on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:09:51] PROBLEM - cp1 HTTPS on cp1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [22:09:51] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:09:52] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:52] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.polandballwiki.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:53] PROBLEM - cp4 Nginx Backend for mw22 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:09:53] PROBLEM - ldap21 Puppet on ldap21 is UNKNOWN: NRPE: Unable to read output [22:09:54] PROBLEM - bots21 CVT Bot on bots21 is CRITICAL: PROCS CRITICAL: 0 processes with args 'cvtbot' [22:09:54] PROBLEM - cloud3 PowerDNS Recursor on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:09:55] PROBLEM - puppet21 APT on puppet21 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (3 critical updates). [22:09:55] PROBLEM - cp6 NTP time on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:09:56] PROBLEM - cp3 Nginx Backend for mw22 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:56] PROBLEM - db21 APT on db21 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [22:09:57] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - meta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:09:57] PROBLEM - cp4 Nginx Backend for phorge21 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:09:58] PROBLEM - cp3 GDNSD Datacenters on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:09:58] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:09:59] PROBLEM - puppet21 Backups Private on puppet21 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/private-backup.log [22:09:59] PROBLEM - cp2 conntrack_table_size on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:10:00] PROBLEM - cp5 NTP time on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:00] PROBLEM - newcascadia.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'newcascadia.net' expired on Tue 14 Nov 2023 11:52:59 PM GMT +0000. [22:10:01] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:01] PROBLEM - cp5 Disk Space on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:02] PROBLEM - mw22 Puppet on mw22 is UNKNOWN: NRPE: Unable to read output [22:10:02] PROBLEM - cloud4 ferm_active on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:10:03] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:03] PROBLEM - cp4 conntrack_table_size on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:04] PROBLEM - cp3 Nginx Backend for mail21 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:04] PROBLEM - sdiy.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - sdiy.info reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:05] PROBLEM - Host swiftobject31 is DOWN: PING CRITICAL - Packet loss = 100% [22:10:05] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:10:06] PROBLEM - cloud2 conntrack_table_size on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:10:06] PROBLEM - projects.dmvpetridish.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - projects.dmvpetridish.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:07] PROBLEM - db31 SSH on db31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:07] PROBLEM - cloud3 ferm_active on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:10:08] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:09] PROBLEM - db31 MariaDB Connections on db31 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db31.wikitide.n...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_c [22:10:10] on line 47Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db31.wikitide.n...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47 [22:10:11] PROBLEM - cp4 Nginx Backend for test21 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:11] PROBLEM - puppet21 HTTPS on puppet21 is WARNING: HTTP WARNING: HTTP/2 403 - 113 bytes in 0.007 second response time [22:10:11] PROBLEM - mw21 MediaWiki Rendering on mw21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.109 second response time [22:10:12] PROBLEM - Host db31 is DOWN: PING CRITICAL - Packet loss = 100% [22:10:16] PROBLEM - cp6 conntrack_table_size on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:16] PROBLEM - polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - polandballwiki.com reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:18] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:21] PROBLEM - cp1 Nginx Backend for test21 on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:22] PROBLEM - cp5 Nginx Backend for phorge21 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:22] PROBLEM - cp4 APT on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:22] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.colleimadcat.com reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:22] PROBLEM - cloud3 Current Load on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:10:23] PROBLEM - hsck.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hsck.lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:23] PROBLEM - db21 Backups SQL wtglobal on db21 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/backup-logs/sql-wtglobal-backup-weekly.log [22:10:23] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:24] PROBLEM - cp3 Nginx Backend for phorge21 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:24] PROBLEM - cp3 Nginx Backend for matomo21 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:24] PROBLEM - cp6 ferm_active on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:25] PROBLEM - mon21 Backups Grafana on mon21 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/grafana-backup.log [22:10:26] PROBLEM - jobchron21 APT on jobchron21 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (3 critical updates). [22:10:26] PROBLEM - cp1 ferm_active on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:26] PROBLEM - cloud4 Disk Space on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:10:27] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:10:28] PROBLEM - cp1 Nginx Backend for mail21 on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:28] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:29] PROBLEM - cp2 NTP time on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:10:30] PROBLEM - puppet21 Puppet on puppet21 is UNKNOWN: NRPE: Unable to read output [22:10:31] PROBLEM - cp6 Nginx Backend for mw21 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:32] PROBLEM - www.wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:32] PROBLEM - cp4 NTP time on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:33] PROBLEM - cp6 Nginx Backend for matomo21 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:33] PROBLEM - cp2 Nginx Backend for matomo21 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:10:33] PROBLEM - cp6 Nginx Backend for phorge21 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:33] PROBLEM - cloud4 APT on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:10:33] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:34] PROBLEM - cp6 APT on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:34] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:10:34] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:35] PROBLEM - cloud4 Puppet on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:10:35] PROBLEM - cp4 Nginx Backend for mail21 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:36] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:36] PROBLEM - thegreatwar.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thegreatwar.uk reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:37] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:37] PROBLEM - greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - greatamerica.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:38] PROBLEM - os21 Puppet on os21 is UNKNOWN: NRPE: Unable to read output [22:10:38] PROBLEM - jobchron21 Current Load on jobchron21 is CRITICAL: LOAD CRITICAL - total load average: 3.92, 3.95, 3.84 [22:10:39] PROBLEM - cloud3 Disk Space on cloud3 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 104.168.123.210: 1 [22:10:39] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mason.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:40] PROBLEM - mw22 MediaWiki Rendering on mw22 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.112 second response time [22:10:40] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:10:41] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:41] PROBLEM - cp6 Nginx Backend for mw22 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:42] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:42] PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:43] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:43] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:10:44] PROBLEM - mw21 Puppet on mw21 is UNKNOWN: NRPE: Unable to read output [22:10:44] PROBLEM - mail21 Puppet on mail21 is UNKNOWN: NRPE: Unable to read output [22:10:45] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:45] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:10:46] PROBLEM - swiftac31 SSH on swiftac31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:46] PROBLEM - realitylongterms.studio - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - realitylongterms.studio reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:47] PROBLEM - cp4 Nginx Backend for puppet21 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:47] PROBLEM - swiftac31 Disk Space on swiftac31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:48] PROBLEM - cloud2 ferm_active on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:10:48] PROBLEM - cp1 Nginx Backend for puppet21 on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:49] PROBLEM - cp1 Disk Space on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:49] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:50] PROBLEM - jobrunner21 Puppet on jobrunner21 is UNKNOWN: NRPE: Unable to read output [22:10:50] PROBLEM - cp1 Nginx Backend for mw22 on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:51] PROBLEM - swiftac31 Swift Container Service on swiftac31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:51] PROBLEM - cloud4 NTP time on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:10:52] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.94, 3.95, 3.84 [22:10:52] PROBLEM - cp4 Nginx Backend for matomo21 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:10:53] PROBLEM - cp5 APT on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:53] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:54] PROBLEM - distrowiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - distrowiki.net reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:54] PROBLEM - cloud2 APT on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:10:55] PROBLEM - os21 APT on os21 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (3 critical updates). [22:10:55] PROBLEM - cp5 Current Load on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:56] PROBLEM - www.greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.greatamerica.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:10:56] PROBLEM - ldap21 APT on ldap21 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [22:10:57] PROBLEM - cp1 Nginx Backend for phorge21 on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:10:57] PROBLEM - hkrail.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hkrail.wiki reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [22:10:58] PROBLEM - cp5 Nginx Backend for mw21 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [22:10:58] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:10:59] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [22:11:00] PROBLEM - Host swiftac31 is DOWN: PING CRITICAL - Packet loss = 100% [22:11:00] PROBLEM - cp1 Nginx Backend for mw21 on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:11:01] PROBLEM - cp2 Nginx Backend for mw22 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:11:04] PROBLEM - cloud4 PowerDNS Recursor on cloud4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 198.23.193.74: Connection reset by peer [22:11:13] PROBLEM - bots21 Puppet on bots21 is UNKNOWN: NRPE: Unable to read output [22:11:13] PROBLEM - cp1 conntrack_table_size on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:11:17] PROBLEM - swiftproxy21 HTTP on swiftproxy21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:20] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:11:22] PROBLEM - cp2 Nginx Backend for test21 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:11:22] PROBLEM - cp1 NTP time on cp1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:11:30] PROBLEM - cp2 Nginx Backend for mw21 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [22:11:31] PROBLEM - cloud3 APT on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [22:11:32] PROBLEM - cloud2 PowerDNS Recursor on cloud2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 23.95.103.18: Connection reset by peer [22:11:33] PROBLEM - cp4 Nginx Backend for mw21 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [22:11:33] PROBLEM - cp6 Nginx Backend for test21 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [22:11:51] PROBLEM - Host cp1 is DOWN: PING CRITICAL - Packet loss = 100% [22:13:10] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/671492f16686...671ee1dc2559 [22:13:12] [02WikiTideOrg/dns] 07Universal-Omega 03671ee1d - Use mon21 [22:18:33] PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.67, 4.07, 3.87 [22:20:29] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 2.74, 3.51, 3.68 [22:22:21] !log sudo icinga2 node setup --endpoint NodeName --zone ZoneName on mon21 [22:22:23] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [22:27:39] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:30:15] PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 4.10, 3.85 [22:32:15] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.42, 3.93, 3.82 [22:34:15] PROBLEM - mem21 Current Load on mem21 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 4.14, 3.92 [22:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:36:43] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/671ee1dc2559...5d81678a9754 [22:36:45] [02WikiTideOrg/dns] 07Universal-Omega 035d81678 - Use ldap21 [22:39:16] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:39:55] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:50:37] !log stop ircecho on mon1 and restart it on mon21 [22:50:39] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:02:53] !log [agent@jobrunner21] starting deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': ['ManageWiki']} to [mw21, mw22, jobrunner21] [23:02:54] !log [agent@jobrunner21] finished deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': ['ManageWiki']} to [mw21, mw22, jobrunner21] - SUCCESS in 0s [23:02:55] !log [agent@jobrunner21] starting deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': ['ManageWiki']} to [mw21, mw22, jobrunner21] [23:02:55] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:02:56] !log [agent@jobrunner21] finished deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': ['ManageWiki']} to [mw21, mw22, jobrunner21] - SUCCESS in 2s [23:02:57] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:02:57] !log [agent@jobrunner21] starting deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': ['ManageWiki']} to [mw21, mw22, jobrunner21] [23:02:58] !log [agent@jobrunner21] finished deploy of {'force': True, 'versions': ['1.40', '1.41'], 'upgrade_extensions': ['ManageWiki']} to [mw21, mw22, jobrunner21] - SUCCESS in 5s [23:02:59] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:03:01] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:03:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:03:05] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:08:21] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:08:24] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0dfb0b16817b...776c5b538ef8 [23:08:25] [02WikiTideOrg/puppet] 07Universal-Omega 03776c5b5 - Fix [23:08:54] [02WikiTideOrg/puppet] 07github-actions[bot] pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/776c5b538ef8...e6976edb160e [23:08:55] [02WikiTideOrg/puppet] 07github-actions 03e6976ed - CI: lint puppet code to standards Check commit and GitHub actions for more details [23:08:56] [02puppet] 07github-actions[bot] synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:09:05] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [23:09:16] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net reverse DNS resolves to vps-6006c2ae.vps.ovh.ca [23:09:28] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:09:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e6976edb160e...c0ec2551dc95 [23:09:32] [02WikiTideOrg/puppet] 07Universal-Omega 03c0ec255 - Fix [23:09:37] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:09:47] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com reverse DNS resolves to vps-d267e842.vps.ovh.ca [23:11:08] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:21:53] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:21:55] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c0ec2551dc95...1f764f7c4711 [23:21:57] [02WikiTideOrg/puppet] 07Universal-Omega 031f764f7 - Add CheckCommand [23:22:25] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:22:28] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1f764f7c4711...41e0de21a781 [23:22:31] [02WikiTideOrg/puppet] 07Universal-Omega 0341e0de2 - Fix [23:22:44] !log [agent@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/WikiTideMagic/maintenance/resetWikiCaches.php (START) [23:22:46] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:23:02] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:23:04] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/41e0de21a781...322cc259e553 [23:23:05] [02WikiTideOrg/puppet] 07Universal-Omega 03322cc25 - Fix [23:23:15] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:23:44] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:24:22] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:25:18] !log [agent@jobrunner21] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/WikiTideMagic/maintenance/resetWikiCaches.php (END - exit=0) [23:25:20] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:29:06] PROBLEM - cloud3 MegaRAID on cloud3 is UNKNOWN: execvpe(/usr/lib/nagios/pluginscheck_nrpe) failed: No such file or directory [23:30:47] PROBLEM - cloud3 get_raid_status_megacli on cloud3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 104.168.123.210: Connection reset by peer [23:32:47] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:32:49] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/322cc259e553...d2a54ef0327c [23:32:50] [02WikiTideOrg/puppet] 07Universal-Omega 03d2a54ef - Fix [23:32:52] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0ad6c16877af...937ce071fca2 [23:32:53] [02WikiTideOrg/puppet] 07AgentIsai 03937ce07 - Add cloud2 to nrpe allowlist [23:33:56] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:35:34] [02WikiTideOrg/puppet] 07Universal-Omega pushed 032 commits to 03raid [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/d2a54ef0327c...b21e5e6312c5 [23:35:36] [02WikiTideOrg/puppet] 07Universal-Omega 03b21e5e6 - Merge branch 'master' into raid [23:35:38] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:36:45] PROBLEM - cloud3 MegaRAID on cloud3 is WARNING: Could not resolve hostname Object of type 'Host': Name or service not known [23:36:52] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:38:41] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [23:38:45] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:38:45] RECOVERY - cloud3 conntrack_table_size on cloud3 is OK: OK: nf_conntrack is 0 % full [23:38:47] RECOVERY - cloud3 Disk Space on cloud3 is OK: DISK OK - free space: / 86731MiB (95% inode=98%); [23:38:52] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [23:39:02] PROBLEM - cloud3 get_raid_status_megacli on cloud3 is UNKNOWN: NRPE: Unable to read output [23:39:03] RECOVERY - cloud3 NTP time on cloud3 is OK: NTP OK: Offset 0.001073569059 secs [23:39:03] RECOVERY - cloud3 PowerDNS Recursor on cloud3 is OK: DNS OK: 0.514 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [23:39:10] RECOVERY - cp4 Nginx Backend for matomo21 on cp4 is OK: TCP OK - 0.004 second response time on localhost port 8204 [23:39:11] RECOVERY - cp4 Nginx Backend for puppet21 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8201 [23:39:12] RECOVERY - cp4 Current Load on cp4 is OK: LOAD OK - total load average: 0.21, 0.07, 0.02 [23:39:16] RECOVERY - cp4 APT on cp4 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [23:39:17] RECOVERY - cloud3 ferm_active on cloud3 is OK: OK ferm input default policy is set [23:39:20] RECOVERY - cp4 Disk Space on cp4 is OK: DISK OK - free space: / 5970MiB (31% inode=94%); [23:39:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [23:39:33] RECOVERY - cloud3 Current Load on cloud3 is OK: LOAD OK - total load average: 1.38, 1.37, 1.30 [23:39:44] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 1% [23:39:45] RECOVERY - cp4 Nginx Backend for test21 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8180 [23:39:47] RECOVERY - cp4 ferm_active on cp4 is OK: OK ferm input default policy is set [23:39:50] RECOVERY - cp4 Nginx Backend for mw21 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8100 [23:45:03] PROBLEM - cloud3 MegaRAID on cloud3 is WARNING: Could not resolve hostname Object of type 'Host': Name or service not known [23:45:06] RECOVERY - cp4 Nginx Backend for mw22 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8101 [23:45:11] RECOVERY - cp4 conntrack_table_size on cp4 is OK: OK: nf_conntrack is 0 % full [23:45:21] RECOVERY - cp4 Nginx Backend for mail21 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8202 [23:45:45] RECOVERY - cp4 Nginx Backend for phorge21 on cp4 is OK: TCP OK - 0.000 second response time on localhost port 8200 [23:45:54] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [23:46:06] RECOVERY - cloud4 ferm_active on cloud4 is OK: OK ferm input default policy is set [23:46:08] RECOVERY - cp4 NTP time on cp4 is OK: NTP OK: Offset 9.527802467e-05 secs [23:46:10] RECOVERY - cp4 PowerDNS Recursor on cp4 is OK: DNS OK: 0.528 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [23:46:40] PROBLEM - cloud3 get_raid_status_megacli on cloud3 is UNKNOWN: NRPE: Unable to read output [23:46:51] RECOVERY - cloud4 conntrack_table_size on cloud4 is OK: OK: nf_conntrack is 0 % full [23:46:54] RECOVERY - cloud4 PowerDNS Recursor on cloud4 is OK: DNS OK: 0.322 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [23:46:58] RECOVERY - cloud4 Current Load on cloud4 is OK: LOAD OK - total load average: 0.39, 0.15, 0.10 [23:47:12] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:19] RECOVERY - cloud4 NTP time on cloud4 is OK: NTP OK: Offset 0.0004002451897 secs [23:47:28] RECOVERY - cloud4 Disk Space on cloud4 is OK: DISK OK - free space: / 87547MiB (95% inode=98%); [23:47:29] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 6071MiB (31% inode=94%); [23:47:32] RECOVERY - cp6 Nginx Backend for mw21 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8100 [23:47:34] RECOVERY - cp6 Nginx Backend for mail21 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8202 [23:47:38] RECOVERY - cp6 Nginx Backend for matomo21 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8204 [23:47:38] RECOVERY - cp6 Nginx Backend for mw22 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8101 [23:47:39] RECOVERY - cp6 Nginx Backend for phorge21 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8200 [23:48:01] RECOVERY - cp6 PowerDNS Recursor on cp6 is OK: DNS OK: 1.309 second response time. wikitide.org returns 139.99.236.151,2402:1f00:8100:400::1d23 [23:48:05] RECOVERY - cp6 conntrack_table_size on cp6 is OK: OK: nf_conntrack is 0 % full [23:48:09] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:49:01] RECOVERY - cp6 APT on cp6 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [23:49:08] RECOVERY - cp6 ferm_active on cp6 is OK: OK ferm input default policy is set [23:49:13] RECOVERY - cp6 NTP time on cp6 is OK: NTP OK: Offset 0.0002121031284 secs [23:49:15] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [23:49:19] RECOVERY - cp6 Current Load on cp6 is OK: LOAD OK - total load average: 0.04, 0.06, 0.01 [23:49:20] RECOVERY - cp6 Nginx Backend for test21 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8180 [23:49:20] RECOVERY - cp6 Nginx Backend for puppet21 on cp6 is OK: TCP OK - 0.000 second response time on localhost port 8201 [23:49:26] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 2% [23:50:47] RECOVERY - cp3 Nginx Backend for phorge21 on cp3 is OK: TCP OK - 0.001 second response time on localhost port 8200 [23:50:51] RECOVERY - cp3 Nginx Backend for mail21 on cp3 is OK: TCP OK - 0.000 second response time on localhost port 8202 [23:50:55] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [23:51:07] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5697MiB (29% inode=94%); [23:51:08] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [23:51:12] RECOVERY - cp3 conntrack_table_size on cp3 is OK: OK: nf_conntrack is 0 % full [23:51:14] RECOVERY - cp3 Nginx Backend for mw22 on cp3 is OK: TCP OK - 0.003 second response time on localhost port 8101 [23:51:15] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:51:21] RECOVERY - cp3 Nginx Backend for matomo21 on cp3 is OK: TCP OK - 0.000 second response time on localhost port 8204 [23:51:29] RECOVERY - cp3 NTP time on cp3 is OK: NTP OK: Offset -0.0008716881275 secs [23:51:30] RECOVERY - cp2 conntrack_table_size on cp2 is OK: OK: nf_conntrack is 1 % full [23:51:32] RECOVERY - cp3 Nginx Backend for puppet21 on cp3 is OK: TCP OK - 0.000 second response time on localhost port 8201 [23:51:35] RECOVERY - cp3 Current Load on cp3 is OK: LOAD OK - total load average: 0.21, 0.19, 0.13 [23:51:39] RECOVERY - cp3 ferm_active on cp3 is OK: OK ferm input default policy is set [23:51:46] RECOVERY - cp2 ferm_active on cp2 is OK: OK ferm input default policy is set [23:51:51] RECOVERY - cp3 Nginx Backend for test21 on cp3 is OK: TCP OK - 0.000 second response time on localhost port 8180 [23:52:03] RECOVERY - cp3 Nginx Backend for mw21 on cp3 is OK: TCP OK - 0.001 second response time on localhost port 8100 [23:52:05] RECOVERY - cp3 APT on cp3 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [23:52:05] RECOVERY - cp2 Nginx Backend for mw22 on cp2 is OK: TCP OK - 0.001 second response time on localhost port 8101 [23:52:06] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 5441MiB (28% inode=94%); [23:52:33] RECOVERY - cp2 Nginx Backend for mw21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8100 [23:52:34] RECOVERY - cp5 Nginx Backend for mw22 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8101 [23:52:46] RECOVERY - cp5 Nginx Backend for matomo21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8204 [23:52:47] RECOVERY - cp2 PowerDNS Recursor on cp2 is OK: DNS OK: 0.330 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [23:52:48] RECOVERY - cp2 Nginx Backend for puppet21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8201 [23:52:49] RECOVERY - cp5 conntrack_table_size on cp5 is OK: OK: nf_conntrack is 0 % full [23:52:50] RECOVERY - cp2 Current Load on cp2 is OK: LOAD OK - total load average: 0.54, 0.38, 0.35 [23:52:58] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [23:52:58] RECOVERY - cp2 NTP time on cp2 is OK: NTP OK: Offset -0.003574073315 secs [23:52:59] RECOVERY - cp5 Nginx Backend for mw21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8100 [23:53:01] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b21e5e6312c5...7d78238d868f [23:53:01] RECOVERY - cp2 Nginx Backend for matomo21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8204 [23:53:02] RECOVERY - cp2 Nginx Backend for test21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8180 [23:53:03] RECOVERY - cp5 NTP time on cp5 is OK: NTP OK: Offset 0.00125643611 secs [23:53:04] [02WikiTideOrg/puppet] 07Universal-Omega 037d78238 - Fix [23:53:05] RECOVERY - cp5 Current Load on cp5 is OK: LOAD OK - total load average: 0.06, 0.03, 0.01 [23:53:11] RECOVERY - cp2 Nginx Backend for mail21 on cp2 is OK: TCP OK - 0.002 second response time on localhost port 8202 [23:53:12] RECOVERY - cp5 Nginx Backend for test21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8180 [23:53:12] RECOVERY - cp5 Disk Space on cp5 is OK: DISK OK - free space: / 5770MiB (30% inode=94%); [23:53:13] RECOVERY - cp2 APT on cp2 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [23:53:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [23:53:14] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:14] RECOVERY - cp2 Nginx Backend for phorge21 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8200 [23:53:15] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2% [23:53:18] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 4% [23:53:19] RECOVERY - cp5 Nginx Backend for phorge21 on cp5 is OK: TCP OK - 0.000 second response time on localhost port 8200 [23:53:27] RECOVERY - cp5 Nginx Backend for mail21 on cp5 is OK: TCP OK - 0.002 second response time on localhost port 8202 [23:53:27] RECOVERY - cp5 Nginx Backend for puppet21 on cp5 is OK: TCP OK - 0.006 second response time on localhost port 8201 [23:53:28] RECOVERY - cp5 ferm_active on cp5 is OK: OK ferm input default policy is set [23:53:28] RECOVERY - cp5 APT on cp5 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [23:53:36] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:53] RECOVERY - cp5 PowerDNS Recursor on cp5 is OK: DNS OK: 0.803 seconds response time. wikitide.org returns 15.235.167.159,2402:1f00:8000:800::1b3d [23:53:57] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [23:54:11] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:57:15] PROBLEM - mon21 Check correctness of the icinga configuration on mon21 is CRITICAL: Icinga configuration contains errors