[00:00:07] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:00:37] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:00:56] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2570 bytes in 0.394 second response time [00:01:11] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2569 bytes in 0.151 second response time [00:01:39] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 5.503 second response time [00:01:42] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.559 second response time [00:01:51] PROBLEM - db1 MariaDB on db1 is CRITICAL: Access denied for user 'icinga'@'10.0.0.111' (using password: YES) [00:01:54] PROBLEM - greatamerica.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - greatamerica.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [00:02:00] PROBLEM - wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [00:02:06] PROBLEM - cp4 Puppet on cp4 is UNKNOWN: NRPE: Unable to read output [00:02:11] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 52% [00:02:18] @cosmicalpha ^ [00:02:35] PROBLEM - cp6 Puppet on cp6 is UNKNOWN: NRPE: Unable to read output [00:02:55] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2894 bytes in 0.793 second response time [00:03:04] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.461 second response time [00:03:08] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2938 bytes in 0.291 second response time [00:03:23] seems to be temporary [00:04:11] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 18% [00:13:58] Probably yeah [00:14:11] Sorry had to step away [00:31:05] RECOVERY - greatamerica.wiki - reverse DNS on sslhost is OK: SSL OK - greatamerica.wiki reverse DNS resolves to cp1.wikitide.net - CNAME FLAT [00:31:40] RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:00:59] RECOVERY - db1 Backups SQL on db1 is OK: FILE_AGE OK: /var/log/backup-logs/sql-backup.log is 58 seconds old and 275 bytes [03:07:14] PROBLEM - db1 Current Load on db1 is WARNING: LOAD WARNING - total load average: 7.06, 5.27, 2.77 [03:09:14] RECOVERY - db1 Current Load on db1 is OK: LOAD OK - total load average: 6.10, 5.56, 3.19 [03:21:01] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=lhmnwikitide (END - exit=65280) [03:21:04] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:21:32] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=rosettacodewikitide (END - exit=0) [03:21:34] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:22:11] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=lhmnwikitide (END - exit=0) [03:22:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:30:05] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_skins': ['Citizen']} to [mw1, mw2, jobrunner1] [03:30:06] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_skins': ['Citizen']} to [mw1, mw2, jobrunner1] - SUCCESS in 0s [03:30:07] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_skins': ['Citizen']} to [mw1, mw2, jobrunner1] [03:30:08] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:30:09] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_skins': ['Citizen']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [03:30:11] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:30:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:30:17] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [03:49:04] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d3e0c17b5848...486545677991 [03:49:05] [02WikiTideOrg/puppet] 07Universal-Omega 034865456 - Remove purged extensions [04:09:26] [02WikiTideOrg/TSPortal] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±12] 13https://github.com/WikiTideOrg/TSPortal/compare/c888ab9ebf61...697683b9c916 [04:09:27] [02WikiTideOrg/TSPortal] 07Universal-Omega 03697683b - Update for WikiTide [04:15:15] [02WikiTideOrg/TSPortal] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/WikiTideOrg/TSPortal/compare/697683b9c916...74760f404aba [04:15:18] [02WikiTideOrg/TSPortal] 07Universal-Omega 0374760f4 - Rename to Reports [04:17:55] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/486545677991...244de82d09e1 [04:17:56] [02WikiTideOrg/puppet] 07Universal-Omega 03244de82 - Auto-update WikiTideMagic [05:31:36] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [06:17:50] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [06:30:12] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:31:04] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:47:36] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:53:31] [02landing] 07Reception123 edited pull request 03#2: make landing page translatable per T71 - 13https://github.com/WikiTideOrg/landing/pull/2 [06:57:39] [02WikiTideOrg/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://github.com/WikiTideOrg/landing/compare/7cdf00d9eb73...e325b8f71fa7 [06:57:42] [02WikiTideOrg/landing] 07Reception123 03e325b8f - Create LICENSE [06:57:45] [02landing] 07Reception123 synchronize pull request 03#2: make landing page translatable per T71 - 13https://github.com/WikiTideOrg/landing/pull/2 [07:01:43] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [07:30:34] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:00:22] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:16:52] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:31:06] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [08:35:06] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [09:01:25] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [09:35:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:59:58] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:24:50] PROBLEM - www.rosettacode.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.rosettacode.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:32:49] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [13:33:14] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [13:33:38] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [14:02:09] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:02:54] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [14:05:29] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:20:53] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [14:22:49] RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Sat 20 Jan 2024 06:52:33 AM GMT +0000. [14:31:26] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [14:50:36] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [15:48:06] PROBLEM - puppet1 APT on puppet1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:48:09] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [15:48:22] PROBLEM - graylog1 APT on graylog1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:48:43] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:48:49] PROBLEM - bast1 APT on bast1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [15:49:47] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:50:23] PROBLEM - bots1 APT on bots1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [15:50:28] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:50:36] PROBLEM - prometheus1 APT on prometheus1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [15:51:25] PROBLEM - mw1 APT on mw1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:51:45] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:52:06] PROBLEM - mw2 APT on mw2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:53:17] PROBLEM - test1 APT on test1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:54:23] PROBLEM - os1 APT on os1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:56:15] PROBLEM - cp1 APT on cp1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:56:50] PROBLEM - phorge1 APT on phorge1 is CRITICAL: APT CRITICAL: 21 packages available for upgrade (1 critical updates). [15:57:37] PROBLEM - db1 APT on db1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:58:38] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:00:34] PROBLEM - swiftac1 APT on swiftac1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:00:38] PROBLEM - mem1 APT on mem1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [16:00:43] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 18 packages available for upgrade (1 critical updates). [16:01:32] PROBLEM - swiftobject1 APT on swiftobject1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:01:32] PROBLEM - swiftproxy1 APT on swiftproxy1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:04:23] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:01:29] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [17:06:56] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:14:03] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/97ae389937d0...f6a9d45e0294 [17:14:05] [02WikiTideOrg/mw-config] 07Universal-Omega 03f6a9d45 - Remove X-Wiki-Visibility header Not needed [17:15:00] WikiTideOrg/mw-config - Universal-Omega the build passed. [17:18:20] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [17:18:22] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [17:18:27] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [17:18:31] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [17:30:48] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:31:07] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [17:31:08] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [17:31:10] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [17:31:13] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [17:35:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/244de82d09e1...36e230b2ab40 [17:35:49] [02WikiTideOrg/puppet] 07Universal-Omega 0336e230b - wikitide-backup: fixing download of SQL, should be tar.gz not 7z [18:18:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+3/-4/±57] 13https://github.com/WikiTideOrg/puppet/compare/36e230b2ab40...0dad1f909e90 [18:18:29] [02WikiTideOrg/puppet] 07Universal-Omega 030dad1f9 - Update apt module to 9.1.0 [18:20:56] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [18:22:31] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:23:03] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/0dad1f909e90...7015211dd522 [18:23:07] [02WikiTideOrg/puppet] 07Universal-Omega 037015211 - Fix: readd apt::security [18:23:35] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:27:52] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:52] PROBLEM - os1 Puppet on os1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:29:40] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:44:15] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/7015211dd522...c78c84e6439f [18:44:17] [02WikiTideOrg/puppet] 07Universal-Omega 03c78c84e - xconsole -> console [18:46:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c78c84e6439f...1e38322d7fd9 [18:46:16] [02WikiTideOrg/puppet] 07Universal-Omega 031e38322 - graylog: upgrade to 5.2.1 [18:50:31] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:50:39] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [18:53:34] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:55:52] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:56:52] RECOVERY - os1 Puppet on os1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:57:40] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:08] !log [universalomega@puppet1] Upgraded packages intel-microcode on mem1 [19:17:11] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:18:08] RECOVERY - mem1 APT on mem1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:20:33] !log [universalomega@puppet1] Upgraded packages intel-microcode on cloud1 [19:20:36] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:20:43] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 17 packages available for upgrade (0 critical updates). [19:21:05] !log [universalomega@puppet1] Upgraded packages intel-microcode on bots1 [19:21:08] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:21:32] !log [universalomega@puppet1] Upgraded packages connect on cp1 [19:21:35] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:22:21] !log [universalomega@puppet1] Upgraded packages intel-microcode on jobchron1 [19:22:23] RECOVERY - bots1 APT on bots1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:22:23] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:22:43] RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:22:54] !log [universalomega@puppet1] Upgraded packages intel-microcode on jobrunner1 [19:22:57] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:23:30] !log [universalomega@puppet1] Upgraded packages intel-microcode on swiftac1 [19:23:33] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:23:50] !log [universalomega@puppet1] Upgraded packages Unmet on mon1 [19:23:53] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:24:24] !log [universalomega@puppet1] Upgraded packages intel-microcode on swiftproxy1 [19:24:27] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:24:28] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:24:35] RECOVERY - swiftac1 APT on swiftac1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:24:55] !log [universalomega@puppet1] Upgraded packages intel-microcode on db1 [19:24:58] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:25:32] RECOVERY - swiftproxy1 APT on swiftproxy1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:25:40] !log [universalomega@puppet1] Upgraded packages intel-microcode on ldap1 [19:25:43] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:25:51] RECOVERY - db1 APT on db1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:22] !log [universalomega@puppet1] Upgraded packages intel-microcode on swiftobject1 [19:26:25] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:26:37] RECOVERY - ldap1 APT on ldap1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:54] !log [universalomega@puppet1] Upgraded packages intel-microcode on puppet1 [19:26:57] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:27:30] !log [universalomega@puppet1] Upgraded packages intel-microcode on prometheus1 [19:27:31] RECOVERY - swiftobject1 APT on swiftobject1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:27:33] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:28:05] RECOVERY - puppet1 APT on puppet1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:28:09] !log [universalomega@puppet1] Upgraded packages intel-microcode on phorge1 [19:28:12] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:28:23] RECOVERY - os1 APT on os1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:28:36] RECOVERY - prometheus1 APT on prometheus1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:28:40] !log [universalomega@puppet1] Upgraded packages intel-microcode on os1 [19:28:43] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:28:50] RECOVERY - phorge1 APT on phorge1 is OK: APT OK: 20 packages available for upgrade (0 critical updates). [19:29:17] RECOVERY - test1 APT on test1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:29:34] !log [universalomega@puppet1] Upgraded packages intel-microcode on test1 [19:29:37] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:30:09] !log [universalomega@puppet1] Upgraded packages intel-microcode on mail1 [19:30:12] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:30:40] RECOVERY - bast1 APT on bast1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:30:54] !log [universalomega@puppet1] Upgraded packages intel-microcode on bast1 [19:30:57] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:31:44] RECOVERY - mail1 APT on mail1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:31:57] !log [universalomega@puppet1] Upgraded packages intel-microcode on services1 [19:32:00] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:32:42] !log [universalomega@puppet1] Upgraded packages intel-microcode on ns1 [19:32:45] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:33:08] RECOVERY - services1 APT on services1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:34:09] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:35:20] !log [universalomega@puppet1] Upgraded packages connect on mw1 [19:35:23] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:37:38] !log [universalomega@puppet1] Upgraded packages connect on mw2 [19:37:41] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:40:48] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:43:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1e38322d7fd9...16d33bfeb65a [19:43:16] [02WikiTideOrg/puppet] 07Universal-Omega 0316d33bf - Use 3gb [19:44:03] !log [universalomega@puppet1] Upgraded packages intel-microcode on graylog1 [19:44:06] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:44:21] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:45:01] !log [universalomega@puppet1] Upgraded packages intel-microcode on cp1 [19:45:04] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:45:18] RECOVERY - cp1 APT on cp1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:46:09] !log [universalomega@puppet1] Upgraded packages intel-microcode on mw1 [19:46:12] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:46:48] !log [universalomega@puppet1] Upgraded packages intel-microcode on mw2 [19:46:51] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:47:05] RECOVERY - mw2 APT on mw2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:47:10] RECOVERY - mw1 APT on mw1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:56:57] !log INSERT INTO cw_wikis ... VALUES ... for letteraturawikitide (restore following erroneous deletion) [19:57:00] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:58:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/5df56a59b41d [19:58:27] [02WikiTideOrg/puppet] 07Universal-Omega 035df56a5 - PostgreSQL: fix support for bookworm [19:58:30] [02puppet] 07Universal-Omega created branch 03postgresql-bookworm - 13https://github.com/WikiTideOrg/puppet [19:58:33] [02puppet] 07Universal-Omega opened pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [19:59:55] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5df56a59b41d...6d3af72c205a [19:59:58] [02WikiTideOrg/puppet] 07Universal-Omega 036d3af72 - Fix [19:59:59] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:00:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6d3af72c205a...e090c9c74476 [20:00:40] [02WikiTideOrg/puppet] 07Universal-Omega 03e090c9c - Fix [20:00:42] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:01:16] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e090c9c74476...6fac60d34e37 [20:01:19] [02WikiTideOrg/puppet] 07Universal-Omega 036fac60d - Remove [20:01:22] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:03:01] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/migrateRevisionCommentTemp.php --wiki=letteraturawikitide (END - exit=0) [20:03:04] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:03:11] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/migrateRevisionCommentTemp.php --wiki=letteraturawikitide --force (END - exit=0) [20:03:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:03:25] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6fac60d34e37...da7789ff4686 [20:03:26] [02WikiTideOrg/puppet] 07Universal-Omega 03da7789f - Fix [20:03:28] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:04:21] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/da7789ff4686...2121875d9fd2 [20:04:22] [02WikiTideOrg/puppet] 07Universal-Omega 032121875 - Remove [20:04:24] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:06:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2121875d9fd2...39366b342e92 [20:06:33] [02WikiTideOrg/puppet] 07Universal-Omega 0339366b3 - Fix [20:06:35] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:07:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03postgresql-bookworm [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/39366b342e92...a1e2f5739bca [20:07:13] [02WikiTideOrg/puppet] 07Universal-Omega 03a1e2f57 - Fix [20:07:15] [02puppet] 07Universal-Omega synchronize pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:09:17] [02puppet] 07Universal-Omega closed pull request 03#17: PostgreSQL: fix support for bookworm - 13https://github.com/WikiTideOrg/puppet/pull/17 [20:09:20] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03postgresql-bookworm [20:09:21] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/WikiTideOrg/puppet/compare/16d33bfeb65a...34dc94155e62 [20:09:22] [02WikiTideOrg/puppet] 07Universal-Omega 0334dc941 - PostgreSQL: fix support for bookworm [20:09:23] [02puppet] 07Universal-Omega deleted branch 03postgresql-bookworm - 13https://github.com/WikiTideOrg/puppet [20:15:53] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/34dc94155e62...fe37c13967e1 [20:15:56] [02WikiTideOrg/puppet] 07Universal-Omega 03fe37c13 - Remove bullseye [20:30:20] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/fe37c13967e1...96619f368999 [20:30:22] [02WikiTideOrg/puppet] 07Universal-Omega 0396619f3 - check_puppet_run: fix support for ruby 3.2.0+ [20:31:31] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:31:36] [02WikiTideOrg/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/3e704571899a [20:31:38] [02WikiTideOrg/mw-config] 07Reception123 033e70457 - add wgLogos for picrosswikitide T73 [20:31:40] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/WikiTideOrg/mw-config [20:31:43] [02mw-config] 07Reception123 opened pull request 03#13: add wgLogos for picrosswikitide T73 - 13https://github.com/WikiTideOrg/mw-config/pull/13 [20:32:36] WikiTideOrg/mw-config - Reception123 the build passed. [20:33:14] [02mw-config] 07Reception123 closed pull request 03#13: add wgLogos for picrosswikitide T73 - 13https://github.com/WikiTideOrg/mw-config/pull/13 [20:33:16] [02WikiTideOrg/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [20:33:17] [02WikiTideOrg/mw-config] 07Reception123 pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/f6a9d45e0294...5e356d40224c [20:33:20] [02WikiTideOrg/mw-config] 07Reception123 035e356d4 - add wgLogos for picrosswikitide T73 (#13) [20:33:21] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/WikiTideOrg/mw-config [20:34:02] !log [reception@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [20:34:05] !log [reception@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [20:34:05] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:34:08] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:34:10] WikiTideOrg/mw-config - Reception123 the build passed. [20:39:09] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:43:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/96619f368999...f28fe16151a8 [20:43:32] [02WikiTideOrg/puppet] 07Universal-Omega 03f28fe16 - expand_path: fix support for ruby 3.2.0+ [20:47:22] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:50:02] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:51:29] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:51:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:52:19] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:56:08] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f28fe16151a8...6ea51d3e1cea [20:56:10] [02WikiTideOrg/puppet] 07Universal-Omega 036ea51d3 - Fix part 2 [20:58:44] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6ea51d3e1cea...0cb2163809eb [20:58:45] [02WikiTideOrg/puppet] 07Universal-Omega 030cb2163 - Fix part 3 [21:01:00] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [21:01:01] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [21:01:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:01:06] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:03:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0cb2163809eb...0e6cfc601478 [21:03:31] [02WikiTideOrg/puppet] 07Universal-Omega 030e6cfc6 - Fix part 4 [21:05:20] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0e6cfc601478...d6d75b976673 [21:05:22] [02WikiTideOrg/puppet] 07Universal-Omega 03d6d75b9 - Fix part 5 [21:07:42] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d6d75b976673...f6d7155ed0a3 [21:07:44] [02WikiTideOrg/puppet] 07Universal-Omega 03f6d7155 - Fix part 6 [21:09:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f6d7155ed0a3...e1a8d3a17602 [21:09:48] [02WikiTideOrg/puppet] 07Universal-Omega 03e1a8d3a - Fix part 7 [21:14:18] PROBLEM - mail1 Puppet on mail1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 1 failures [21:16:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e1a8d3a17602...727df1ad37e0 [21:16:33] [02WikiTideOrg/puppet] 07Universal-Omega 03727df1a - Fix part 8 [21:18:08] PROBLEM - mail1 Puppet on mail1 is UNKNOWN: NRPE: Unable to read output [21:27:28] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/727df1ad37e0...399b254596c4 [21:27:30] [02WikiTideOrg/puppet] 07Universal-Omega 03399b254 - Revert and try new fix [21:29:38] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/etc/opendkim/keys/mail.private] [21:31:04] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 29 minutes ago with 7 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.39/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules],Exec[DataTransfer-REL1_39 composer] [22:31:29] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [22:34:04] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:00:48] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:13:47] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/deleteBatch.php --wiki=simpleelectronicswikitide incorrect_files.txt (START) [23:13:48] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/deleteBatch.php --wiki=simpleelectronicswikitide incorrect_files.txt (END - exit=0) [23:13:50] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:13:52] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:35:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.