[00:44:44] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:14:43] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [01:27:59] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mediawiki-repos [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/7b7bdd491410...6d88cec7b2e0 [01:28:02] [02WikiTideOrg/puppet] 07Universal-Omega 036d88cec - Add mediawiki-repos path [01:28:03] [02puppet] 07Universal-Omega synchronize pull request 03#46: Use seperate mediawiki-repos repository - 13https://github.com/WikiTideOrg/puppet/pull/46 [01:29:22] WikiTideOrg/puppet - Universal-Omega the build passed. [01:29:41] [02puppet] 07Universal-Omega synchronize pull request 03#46: Use seperate mediawiki-repos repository - 13https://github.com/WikiTideOrg/puppet/pull/46 [01:29:42] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mediawiki-repos [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6d88cec7b2e0...5048c53b1a31 [01:29:43] [02WikiTideOrg/puppet] 07Universal-Omega 035048c53 - Use mediawiki-repos for composer.local.json [01:30:42] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d8fa954bf12a...ef1fc5ec6c1f [01:30:44] [02WikiTideOrg/puppet] 07Universal-Omega 03ef1fc5e - Remove AWS [01:30:53] [02WikiTideOrg/puppet] 07Universal-Omega pushed 036 commits to 03mediawiki-repos [+0/-2/±4] 13https://github.com/WikiTideOrg/puppet/compare/5048c53b1a31...643c093c84ee [01:30:54] [02WikiTideOrg/puppet] 07Universal-Omega 036236efe - Use seperate mediawiki-repos repository [01:30:56] [02WikiTideOrg/puppet] 07Universal-Omega 036e9efc0 - Update path [01:30:58] WikiTideOrg/puppet - Universal-Omega the build passed. [01:30:59] [02WikiTideOrg/puppet] 07Universal-Omega 03ad26a1a - Remove [01:31:01] [02WikiTideOrg/puppet] ... and 3 more commits. [01:31:03] [02puppet] 07Universal-Omega synchronize pull request 03#46: Use seperate mediawiki-repos repository - 13https://github.com/WikiTideOrg/puppet/pull/46 [01:31:51] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiTideOrg/mediawiki-repos/compare/85f9efad8aa4...d162cbe0f561 [01:31:53] [02WikiTideOrg/mediawiki-repos] 07Universal-Omega 03d162cbe - Add composer.local.json [01:32:10] WikiTideOrg/puppet - Universal-Omega the build passed. [01:32:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mediawiki-repos [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/643c093c84ee...0fc34c732002 [01:32:27] [02WikiTideOrg/puppet] 07Universal-Omega 030fc34c7 - Remove [01:32:30] [02puppet] 07Universal-Omega synchronize pull request 03#46: Use seperate mediawiki-repos repository - 13https://github.com/WikiTideOrg/puppet/pull/46 [01:34:10] WikiTideOrg/puppet - Universal-Omega the build passed. [01:44:44] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:14:43] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [04:13:51] PROBLEM - hkrail.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hkrail.wiki reverse DNS resolves to 1-65-141-101.static.netvigator.com [10:14:44] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:44:43] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [15:10:31] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/bb17577fa274...b22139bd2746 [15:10:32] [02WikiTideOrg/dns] 07Universal-Omega 03b22139b - Add google-site-verification [15:36:14] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/b22139bd2746...51d06893cc54 [15:36:15] [02WikiTideOrg/dns] 07Universal-Omega 0351d0689 - Update google-site-verification [15:37:42] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/51d06893cc54...ea3776c5a3f4 [15:37:43] [02WikiTideOrg/dns] 07Universal-Omega 03ea3776c - Update google-site-verification [17:14:36] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildtextindex.php --wiki=lgtbqiawiki (END - exit=65280) [17:14:45] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [17:24:36] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildtextindex.php --wiki=lgbtqiawiki (END - exit=0) [17:24:45] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [17:30:44] PROBLEM - nexttide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nexttide.org' expires in 15 day(s) (Sun 14 Jan 2024 05:21:30 PM GMT +0000). [18:05:10] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/ea3776c5a3f4...1df8f54a50f3 [18:05:11] [02WikiTideOrg/dns] 07Universal-Omega 031df8f54 - Update MX [19:03:20] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/1df8f54a50f3...f08e89085a13 [19:03:23] [02WikiTideOrg/dns] 07Universal-Omega 03f08e890 - Add DKIM record [19:03:39] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/f08e89085a13...d34aaac25255 [19:03:42] [02WikiTideOrg/dns] 07Universal-Omega 03d34aaac - Remove newline [19:04:13] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/d34aaac25255...5812d4d41515 [19:04:16] [02WikiTideOrg/dns] 07Universal-Omega 035812d4d - Fix tabbing [19:10:34] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/5812d4d41515...f3b1e6f9c87f [19:10:36] [02WikiTideOrg/dns] 07Universal-Omega 03f3b1e6f - Add SPF record [19:36:34] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03smtp [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/65b3f8daa99e [19:36:36] [02WikiTideOrg/mw-config] 07Universal-Omega 0365b3f8d - Update SMTP settings [19:36:37] [02mw-config] 07Universal-Omega created branch 03smtp - 13https://github.com/WikiTideOrg/mw-config [19:36:58] [02mw-config] 07Universal-Omega opened pull request 03#29: Update SMTP settings - 13https://github.com/WikiTideOrg/mw-config/pull/29 [19:37:56] WikiTideOrg/mw-config - Universal-Omega the build passed. [20:30:58] [02mw-config] 07Universal-Omega synchronize pull request 03#29: Update SMTP settings - 13https://github.com/WikiTideOrg/mw-config/pull/29 [20:31:00] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03smtp [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/65b3f8daa99e...9a96d04de105 [20:31:01] [02WikiTideOrg/mw-config] 07Universal-Omega 039a96d04 - Fix [20:31:57] WikiTideOrg/mw-config - Universal-Omega the build passed. [20:32:10] [02mw-config] 07Universal-Omega edited pull request 03#29: Update SMTP settings - 13https://github.com/WikiTideOrg/mw-config/pull/29 [20:38:53] PROBLEM - puppet1 HTTPS on puppet1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to puppet1.wikitide.net port 443 after 0 ms: Couldn't connect to server [20:40:33] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/ef1fc5ec6c1f...c281afa1854b [20:40:34] [02WikiTideOrg/puppet] 07Universal-Omega 03c281afa - phorge: update SMTP [20:40:49] PROBLEM - puppet1 puppetserver on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 8140: Connection refused [20:40:50] PROBLEM - puppet1 ferm_active on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:40:53] PROBLEM - puppet1 puppetdb on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 8081: Connection refused [20:40:53] PROBLEM - puppet1 PowerDNS Recursor on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:41:07] PROBLEM - puppet1 Backups SSLKeys on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:41:08] PROBLEM - puppet1 Current Load on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:41:18] PROBLEM - puppet1 Backups Private on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:41:20] PROBLEM - puppet1 SSH on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 22: Connection refused [20:41:28] PROBLEM - puppet1 APT on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:41:28] [02mw-config] 07Universal-Omega closed pull request 03#29: Update SMTP settings - 13https://github.com/WikiTideOrg/mw-config/pull/29 [20:41:31] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/908d0cbaa72d...be6935e1203a [20:41:32] [02WikiTideOrg/mw-config] 07Universal-Omega 03be6935e - Update SMTP settings [20:41:35] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03smtp [20:41:37] [02mw-config] 07Universal-Omega deleted branch 03smtp - 13https://github.com/WikiTideOrg/mw-config [20:41:47] PROBLEM - puppet1 conntrack_table_size on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:42:07] PROBLEM - puppet1 NTP time on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [20:42:28] WikiTideOrg/mw-config - Universal-Omega the build passed. [20:49:22] !log [universalomega@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [20:49:26] !log [universalomega@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [20:49:31] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:49:39] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:10:51] RECOVERY - puppet1 ferm_active on puppet1 is OK: OK ferm input default policy is set [21:10:53] RECOVERY - puppet1 puppetdb on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8081 [21:10:53] RECOVERY - puppet1 PowerDNS Recursor on puppet1 is OK: DNS OK: 0.393 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [21:10:53] PROBLEM - puppet1 HTTPS on puppet1 is WARNING: HTTP WARNING: HTTP/2 403 - 113 bytes in 0.019 second response time [21:11:07] RECOVERY - puppet1 Backups SSLKeys on puppet1 is OK: FILE_AGE OK: /var/log/sslkeys-backup.log is 486663 seconds old and 52 bytes [21:11:08] RECOVERY - puppet1 Current Load on puppet1 is OK: LOAD OK - total load average: 2.80, 0.94, 0.33 [21:11:18] RECOVERY - puppet1 Backups Private on puppet1 is OK: FILE_AGE OK: /var/log/private-backup.log is 497354 seconds old and 93 bytes [21:11:20] RECOVERY - puppet1 SSH on puppet1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [21:11:29] RECOVERY - puppet1 APT on puppet1 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [21:11:47] RECOVERY - puppet1 conntrack_table_size on puppet1 is OK: OK: nf_conntrack is 0 % full [21:12:07] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.009038954973 secs [21:25:18] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/f3b1e6f9c87f...f1e98894189c [21:25:19] [02WikiTideOrg/dns] 07Universal-Omega 03f1e9889 - Add google-site-verification [21:26:07] PROBLEM - puppet1 NTP time on puppet1 is WARNING: NTP WARNING: Offset 0.1271471083 secs [21:28:07] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.08588254452 secs [21:31:54] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/f1e98894189c...db0c92cd96ad [21:31:56] [02WikiTideOrg/dns] 07Universal-Omega 03db0c92c - Add mail records [21:36:11] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/db0c92cd96ad...0b2928825910 [21:36:14] [02WikiTideOrg/dns] 07Universal-Omega 030b29288 - Add mail records [22:10:04] [02WikiTideOrg/Reports] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/Reports/compare/74760f404aba...4d6f86ec0d68 [22:10:07] [02WikiTideOrg/Reports] 07Universal-Omega 034d6f86e - reports: update SMTP host [22:10:23] [02WikiTideOrg/Reports] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/Reports/compare/4d6f86ec0d68...79583eaaf3c8 [22:10:26] [02WikiTideOrg/Reports] 07Universal-Omega 0379583ea - Fix