[00:00:42] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0d9b93c9410b...0a286f2109ad
[00:00:44] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 030a286f2 - php::fpm: add monitoring Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:03:16] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03gdnsd [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/10f2a50e5512
[00:03:19] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0310f2a50 - dns: add monitoring Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:03:22] <botifico-49a823b>	 [02puppet] 07Universal-Omega created branch 03gdnsd - 13https://github.com/WikiTideOrg/puppet
[00:03:24] <botifico-49a823b>	 [02puppet] 07Universal-Omega opened pull request 03#36: dns: add monitoring - 13https://github.com/WikiTideOrg/puppet/pull/36
[00:04:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03gdnsd [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/10f2a50e5512...dd144e9a2a40
[00:04:33] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03dd144e9 - Add check_gdnsd_datacenters Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:04:35] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#36: dns: add monitoring - 13https://github.com/WikiTideOrg/puppet/pull/36
[00:05:27] <botifico-49a823b>	 [02puppet] 07Universal-Omega closed pull request 03#36: dns: add monitoring - 13https://github.com/WikiTideOrg/puppet/pull/36
[00:05:29] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0a286f2109ad...e5607269067c
[00:05:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e560726 - dns: add monitoring Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:05:35] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03gdnsd
[00:05:36] <botifico-49a823b>	 [02puppet] 07Universal-Omega deleted branch 03gdnsd - 13https://github.com/WikiTideOrg/puppet
[00:06:02] <wikitide-github>	 WikiTideOrg/puppet - Universal-Omega the build passed.
[00:06:55] <wikitide-github>	 WikiTideOrg/puppet - Universal-Omega the build passed.
[00:07:06] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e5607269067c...e3bdd8a5217a
[00:07:09] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e3bdd8a - Add prometheus::exporter::postfix Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:08:25] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e3bdd8a5217a...c6e568116258
[00:08:28] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03c6e5681 - Add prometheus::exporter::mariadb Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:09:18] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c6e568116258...8cfc5fb7672a
[00:09:21] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 038cfc5fb - Add prometheus::exporter::redis Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:10:30] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8cfc5fb7672a...2f19d1ef9dc2
[00:10:31] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 032f19d1e - Add prometheus::exporter::memcached Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:12:07] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2f19d1ef9dc2...3ac02f54142d
[00:12:10] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 033ac02f5 - include base::monitoring Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:12:30] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/3ac02f54142d...7602a4542924
[00:12:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 037602a45 - Remove duplicate Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:14:07] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/7602a4542924...49291c5c4fb1
[00:14:09] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0349291c5 - Add prometheus::exporter::varnishreqs Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:16:00] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/49291c5c4fb1...60fadf077f10
[00:16:01] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0360fadf0 - Add prometheus::exporter::nginx Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:18:11] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/60fadf077f10...8b48e5de473c
[00:18:12] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 038b48e5d - Add prometheus::exporter::statsd_exporter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:19:04] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[00:23:57] <icinga-wikitide>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_gdnsd_datacenters]
[00:24:40] <icinga-wikitide>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_gdnsd_datacenters]
[00:27:26] <icinga-wikitide>	 PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:28:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/8b48e5de473c...7186c222bacb
[00:28:34] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 037186c22 - Fix name Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:30:05] <wikitide-github>	 WikiTideOrg/puppet - Universal-Omega the build passed.
[00:30:33] <icinga-wikitide>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:31:15] <icinga-wikitide>	 PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:33:16] <icinga-wikitide>	 PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:33:35] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[00:39:46] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiTideOrg/puppet/compare/7186c222bacb...96e75d9eb9cc
[00:39:48] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0396e75d9 - mariadb: fix prometheus exporter
[00:41:11] <icinga-wikitide>	 PROBLEM - cp3 GDNSD Datacenters on cp3 is UNKNOWN: NRPE: Unable to read output
[00:47:14] <icinga-wikitide>	 PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[00:47:54] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/96e75d9eb9cc...0b5b2d3e1d43
[00:47:56] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 030b5b2d3 - Add prometheus::exporter::statsd_exporter::mappings Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:48:45] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0b5b2d3e1d43...c27f09afb9fa
[00:48:48] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03c27f09a - Add prometheus::exporter::statsd_exporter::mappings Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:49:40] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c27f09afb9fa...e3500161ec52
[00:49:43] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e350016 - Add prometheus::exporter::statsd_exporter::mappings Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[00:51:02] <icinga-wikitide>	 RECOVERY - cp3 GDNSD Datacenters on cp3 is OK: OK - all datacenters are online
[00:51:12] <icinga-wikitide>	 RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[00:51:26] <icinga-wikitide>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[01:00:23] <icinga-wikitide>	 RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[01:12:58] <icinga-wikitide>	 RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:14:57] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e3500161ec52...2df0c159c9b3
[01:14:58] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 032df0c15 - mariadb::config: add passwords Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[01:15:05] <icinga-wikitide>	 RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[01:16:34] <CosmicAlpha>	 !log add exporter user to mariadb on db1
[01:16:36] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[02:12:20] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2df0c159c9b3...4282f3aec4f0
[02:12:21] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 034282f3a - Fix using Role::Prometheus Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[02:15:10] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±12] 13https://github.com/WikiTideOrg/puppet/compare/4282f3aec4f0...e897fa6fca87
[02:15:11] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e897fa6 - Use Class[Role::Prometheus]
[02:20:15] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e897fa6fca87...fcc2614de1bf
[02:20:16] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03fcc2614 - Enable SSH monitoring Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[02:28:39] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/fcc2614de1bf...6962d0eace0e
[02:28:41] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 036962d0e - Try fix query_nodes Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[02:32:28] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6962d0eace0e...1b087d74e45b
[02:32:31] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 031b087d7 - revert Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[02:42:45] <icinga-wikitide>	 PROBLEM - cp4 SSH on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:45:54] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03nodes [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/35a175d314ea
[02:45:56] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0335a175d - prometheus: fix nodes for publicly accessible servers Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[02:45:58] <botifico-49a823b>	 [02puppet] 07Universal-Omega created branch 03nodes - 13https://github.com/WikiTideOrg/puppet
[02:46:01] <botifico-49a823b>	 [02puppet] 07Universal-Omega opened pull request 03#37: prometheus: fix nodes for publicly accessible servers - 13https://github.com/WikiTideOrg/puppet/pull/37
[02:50:26] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#37: prometheus: fix nodes for publicly accessible servers - 13https://github.com/WikiTideOrg/puppet/pull/37
[02:50:29] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03nodes [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/35a175d314ea...76e01cd4a5b1
[02:50:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0376e01cd - Don't need to for puppet Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[02:59:28] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1b087d74e45b...2ed3a6f12713
[02:59:30] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 032ed3a6f - Add cloud to firewall Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:01:36] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2ed3a6f12713...fb89c75f0182
[03:01:38] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03fb89c75 - Add cloud1 specifically Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:11:16] <botifico-49a823b>	 [02puppet] 07Universal-Omega closed pull request 03#37: prometheus: fix nodes for publicly accessible servers - 13https://github.com/WikiTideOrg/puppet/pull/37
[03:11:17] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03nodes
[03:11:19] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/fb89c75f0182...b1c4b997547c
[03:11:22] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03b1c4b99 - prometheus: fix nodes for publicly accessible servers Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:11:24] <botifico-49a823b>	 [02puppet] 07Universal-Omega deleted branch 03nodes - 13https://github.com/WikiTideOrg/puppet
[03:11:48] <icinga-wikitide>	 PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:11:53] <icinga-wikitide>	 PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:11:54] <icinga-wikitide>	 PROBLEM - cp2 SSH on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:12:16] <icinga-wikitide>	 PROBLEM - cp6 SSH on cp6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:12:16] <icinga-wikitide>	 PROBLEM - cp5 SSH on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:19:37] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b1c4b997547c...58a567c8e264
[03:19:38] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0358a567c - Add cloud1 to bastion public IPs Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:20:08] <icinga-wikitide>	 RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[03:20:24] <icinga-wikitide>	 RECOVERY - cp6 SSH on cp6 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[03:20:39] <icinga-wikitide>	 RECOVERY - cp5 SSH on cp5 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[03:22:20] <icinga-wikitide>	 RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[03:22:23] <icinga-wikitide>	 RECOVERY - cp2 SSH on cp2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[03:23:29] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:23:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0fe5eb51f7b6...42794a0e921c
[03:23:35] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0342794a0 - use roundcube pass for testing Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:24:01] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 0371 commits to 03snappymail [+7/-5/±200] 13https://github.com/WikiTideOrg/puppet/compare/42794a0e921c...805d1c44ee17
[03:24:02] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03805d1c4 - Merge branch 'master' into snappymail
[03:24:05] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:24:34] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07github-actions[bot] pushed 031 commit to 03snappymail [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/805d1c44ee17...6094e91e331b
[03:24:35] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07github-actions 036094e91 - CI: lint puppet code to standards Check commit and GitHub actions for more details
[03:24:36] <botifico-49a823b>	 [02puppet] 07github-actions[bot] synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:28:12] <icinga-wikitide>	 PROBLEM - mail1 webmail.wikitide.net HTTPS on mail1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:32:17] <icinga-wikitide>	 RECOVERY - mail1 webmail.wikitide.net HTTPS on mail1 is OK: HTTP OK: HTTP/1.1 200 OK - 138917 bytes in 0.021 second response time
[03:35:42] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query.
[03:39:06] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:39:07] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6094e91e331b...8cbf8f1451dc
[03:39:09] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 038cbf8f1 - Update nginx Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:39:40] <icinga-wikitide>	 RECOVERY - cp4 SSH on cp4 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[03:40:32] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:40:33] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8cbf8f1451dc...22996fb4285d
[03:40:34] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0322996fb - Remove Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:43:03] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:43:06] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/22996fb4285d...64ecf8132fcd
[03:43:07] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0364ecf81 - Fix socket Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:46:58] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:46:59] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/64ecf8132fcd...e969d282f108
[03:47:01] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e969d28 - Fix root Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:49:46] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:49:49] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e969d282f108...92f840652c35
[03:49:52] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0392f8406 - Fix Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[03:53:50] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[03:53:53] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/92f840652c35...be8533cd0514
[03:53:55] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03be8533c - Fix include path Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:05:10] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[04:06:35] <WikiTideRelay>	 <songngu.xyz> Finally ready for working with the db
[04:07:29] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:07:29] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/be8533cd0514...495621dc69d0
[04:07:32] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03495621d - Fix path Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:09:09] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:09:12] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/495621dc69d0...f7ddc2c4303f
[04:09:15] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03f7ddc2c - Don't clone git Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:14:18] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[04:16:10] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:16:12] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f7ddc2c4303f...495d0f6b296e
[04:16:13] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03495d0f6 - Use snappymail password Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:23:16] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:23:17] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/495d0f6b296e...5cc43f0fb75b
[04:23:19] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 035cc43f0 - Remove DB Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:23:42] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:23:45] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5cc43f0fb75b...d874e46aab36
[04:23:47] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03d874e46 - Remove from firewall Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:24:11] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#24: Use snappymail instead of roundcube - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:24:12] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03snappymail [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d874e46aab36...f71c589912eb
[04:24:15] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03f71c589 - Remove DB Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:25:55] <botifico-49a823b>	 [02puppet] 07Universal-Omega edited pull request 03#24: Replace roundcube with snappymail - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:25:58] <botifico-49a823b>	 [02puppet] 07Universal-Omega edited pull request 03#24: Replace roundcube with snappymail - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:27:58] <icinga-wikitide>	 PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[04:32:41] <botifico-49a823b>	 [02puppet] 07Universal-Omega closed pull request 03#24: Replace roundcube with snappymail - 13https://github.com/WikiTideOrg/puppet/pull/24
[04:32:42] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+6/-6/±2] 13https://github.com/WikiTideOrg/puppet/compare/58a567c8e264...092ea8cba347
[04:32:45] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03092ea8c - Replace roundcube with snappymail Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:32:47] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03snappymail
[04:32:50] <botifico-49a823b>	 [02puppet] 07Universal-Omega deleted branch 03snappymail - 13https://github.com/WikiTideOrg/puppet
[04:35:44] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/092ea8cba347...6cf73bfdfda0
[04:35:46] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 036cf73bf - Remove passwords::roundcubemail Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[04:36:48] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[04:37:58] <icinga-wikitide>	 RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[05:35:47] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[05:51:22] <CosmicAlpha>	 !log removed roundcubemail from mail1 (replaced with snappymail)
[05:51:23] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[06:16:19] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[06:21:46] <icinga-wikitide>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 864 bytes in 0.071 second response time
[06:23:46] <icinga-wikitide>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.153 second response time
[07:14:11] <WikiTideRelay>	 <songngu.xyz> `maintenance/importSites.php --wiki=lhmndatawikitide [THIS FILE]`?
[07:35:39] <CosmicAlpha>	 @songngu.xyz Maybe @reception123 can help I unfortunately got busy with other things today, apologies.
[07:37:30] <WikiTideRelay>	 <songngu.xyz> oh, wish that I can just apply and help you with that instead-
[08:11:32] <WikiTideRelay>	 <reception123> help with what exactly?
[08:11:50] <WikiTideRelay>	 <songngu.xyz> Set up my site links
[08:12:23] <WikiTideRelay>	 <songngu.xyz, replying to reception123> which can be accomplished with `populateSiteLinks.php`
[08:13:16] <WikiTideRelay>	 <songngu.xyz> Oh wait...
[08:13:42] <WikiTideRelay>	 <songngu.xyz> My site haven't added to `lhmndatawikitide`'s db isn't it?
[08:14:38] <WikiTideLSBot>	 !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/populateSiteLinks.php --wiki=lhmndatawikitide (END - exit=256)
[08:14:40] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[08:14:52] <WikiTideRelay>	 <reception123> I don't think that's a thing?
[08:15:01] <WikiTideRelay>	 <reception123> or is it to a specific extension?
[08:15:05] <WikiTideRelay>	 <reception123> (I'm missing the context here)
[08:15:10] <WikiTideRelay>	 <cosmicalpha> wikibase
[08:15:23] <WikiTideRelay>	 <songngu.xyz> Hold on, it's for Wikibase's site links functionality.
[08:15:47] <WikiTideRelay>	 <songngu.xyz> https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration#Add%20the%20wikis%20to%20the%20sites%20table
[08:16:17] <WikiTideRelay>	 <songngu.xyz> We have to run `addSites.php` in both client and repository first
[08:16:58] <WikiTideRelay>	 <reception123> sure, feel free to give me what command you need (as I don't know the pagepath, etc.) and I can run
[08:17:55] <WikiTideRelay>	 <cosmicalpha> maintenance/addSite.php maybe
[08:18:01] <WikiTideRelay>	 <songngu.xyz> it is
[08:18:36] <WikiTideRelay>	 <cosmicalpha> Thats why it didn't work for me yesterday is I also tried populateSiteLinks lol
[08:18:40] <WikiTideRelay>	 <reception123> oh that I know, I mean the options
[08:18:42] <WikiTideRelay>	 <songngu.xyz> ah...
[08:18:56] <WikiTideRelay>	 <songngu.xyz> [1/3] ```bash
[08:18:57] <WikiTideRelay>	 <songngu.xyz> [2/3] addSite.php --pagepath=https://lhmn.wikitide.org/wiki/\$1  --filepath=https://static.wikitide.net/lhmnwikitide/\$1 --language vi lhmn lhmngroup
[08:18:57] <WikiTideRelay>	 <songngu.xyz> [3/3] ```
[08:19:27] <WikiTideRelay>	 <songngu.xyz> have we changes the scripts yet?
[08:19:43] <WikiTideRelay>	 <cosmicalpha> what scripts
[08:20:13] <WikiTideRelay>	 <songngu.xyz> in the default one it will put my wiki's ID with an ID ending with `wiki`
[08:20:26] <WikiTideRelay>	 <cosmicalpha> oh
[08:20:41] <WikiTideRelay>	 <cosmicalpha> I want to rename all wikis back to wiki suffix anyway
[08:20:42] <WikiTideRelay>	 <songngu.xyz> we should have set it to wikitide instead
[08:21:25] <WikiTideRelay>	 <cosmicalpha> Its just a complex process to rename so many wikis...
[08:21:40] <WikiTideRelay>	 <cosmicalpha> I don't know how we will do it.
[08:22:19] <WikiTideRelay>	 <songngu.xyz> it will, of course increase the amounts of jobs that have to be done
[08:22:37] <WikiTideRelay>	 <cosmicalpha> hmm?
[08:23:02] <WikiTideRelay>	 <songngu.xyz> you mean like, we will rename the db names isn't it?
[08:23:37] <WikiTideRelay>	 <cosmicalpha> Yeah that involves dumping DBs, importing then under new names, and renaming in global tables. It is complex
[08:24:15] <WikiTideRelay>	 <songngu.xyz> I don't think we should-
[08:24:56] <WikiTideRelay>	 <cosmicalpha> It needs to eventually for reasons of ease in the backend. It makes less to maintain, and easier to remember
[08:25:13] <WikiTideRelay>	 <cosmicalpha> I want to do it without downtime though
[08:25:25] <WikiTideRelay>	 <songngu.xyz> Make senses
[08:28:25] <WikiTideRelay>	 <songngu.xyz> but anyways, we still need to add the sites to the table first, or they are never going to work
[08:30:41] <WikiTideRelay>	 <songngu.xyz, replying to songngu.xyz> we will try to add it as normal first, run the script as similar to this
[08:43:51] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[09:15:33] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[09:29:37] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 0372 commits to 03reports [+17/-17/±174] 13https://github.com/WikiTideOrg/puppet/compare/acfb2b43a8ee...b904a34d2a2a
[09:29:38] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03b904a34 - Merge branch 'master' into reports Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[09:29:39] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#18: Add reports module - 13https://github.com/WikiTideOrg/puppet/pull/18
[10:40:28] <botifico-49a823b>	 [02mw-config] 07pisceskaze opened pull request 03#23: Change wgMaxUploadSize - 13https://github.com/WikiTideOrg/mw-config/pull/23
[10:40:45] <botifico-49a823b>	 [02mw-config] 07pisceskaze edited pull request 03#23: Change wgMaxUploadSize for qixwikiwikitide - 13https://github.com/WikiTideOrg/mw-config/pull/23
[10:41:25] <wikitide-github>	 WikiTideOrg/mw-config - pisceskaze the build passed.
[10:42:26] <WikiTideRelay>	 <songngu.xyz> Merge PR #23 pls
[10:42:46] <botifico-49a823b>	 [02mw-config] 07Universal-Omega closed pull request 03#23: Change wgMaxUploadSize for qixwikiwikitide - 13https://github.com/WikiTideOrg/mw-config/pull/23
[10:42:49] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/051dfbef8298...d02a0b1bccb8
[10:42:52] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07pisceskaze 03d02a0b1 - Update LocalSettings.php (#23) Update wgMaxUploadSize
[10:43:40] <wikitide-github>	 WikiTideOrg/mw-config - Universal-Omega the build passed.
[10:45:10] <WikiTideRelay>	 <songngu.xyz> Oh and looking at the addSite script, there doesn't seems to be something should be changed
[10:45:29] <WikiTideRelay>	 <songngu.xyz> Can we try addSite.php right now?
[10:45:41] <WikiTideRelay>	 <cosmicalpha> im mobile
[10:45:46] <WikiTideRelay>	 <songngu.xyz, replying to songngu.xyz> oh
[10:46:02] <WikiTideRelay>	 <cosmicalpha> @reception123 ?
[10:47:56] <WikiTideLSBot>	 !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1]
[10:47:58] <WikiTideLSBot>	 !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 1s
[10:47:58] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[10:48:00] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[10:48:37] <WikiTideRelay>	 <reception123> so I should just run ```addSite.php --pagepath=https://lhmn.wikitide.org/wiki/\$1  --filepath=https://static.wikitide.net/lhmnwikitide/\$1 --language vi lhmn lhmngroup``` then?
[10:48:47] <WikiTideRelay>	 <reception123> for lhmndatawikitide's  ?
[10:48:51] <WikiTideRelay>	 <songngu.xyz> yes?
[10:49:09] <WikiTideRelay>	 <songngu.xyz> oh and lhmnwikitide too
[10:49:35] <WikiTideRelay>	 <songngu.xyz> as it require both client and repo to have it, then we can populateSiteLinks.php
[10:58:31] <WikiTideRelay>	 <songngu.xyz> in the entire WikiTide, the script haven't been run before?
[10:58:41] <WikiTideLSBot>	 !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/addSite.php --wiki=lhmndatawikitide --pagepath=https://lhmn.wikitide.org/wiki/ --filepath=https://static.wikitide.net/lhmnwikitide/ --language vi lhmn lhmngroup (END - exit=0)
[10:58:43] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[10:58:52] <WikiTideLSBot>	 !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/addSite.php --wiki=lhmnwikitide --pagepath=https://lhmn.wikitide.org/wiki/ --filepath=https://static.wikitide.net/lhmnwikitide/ --language vi lhmn lhmngroup (END - exit=0)
[10:58:54] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[10:58:57] <WikiTideRelay>	 <songngu.xyz> oh exit=0
[10:59:10] <WikiTideRelay>	 <songngu.xyz> looking good to me-
[10:59:12] <WikiTideRelay>	 <cosmicalpha, replying to songngu.xyz> 0 means success
[10:59:28] <WikiTideRelay>	 <songngu.xyz> now we have to populateSiteLinks and we are done
[11:00:07] <WikiTideRelay>	 <cosmicalpha> I can't find a script called that
[11:00:19] <WikiTideRelay>	 <reception123> there's only populateSitesTable.php
[11:00:50] <WikiTideRelay>	 <cosmicalpha> where is that one
[11:01:08] <WikiTideRelay>	 <reception123> see the bottom of https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration
[11:01:16] <WikiTideRelay>	 <cosmicalpha> oh
[11:01:43] <WikiTideLSBot>	 !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1]
[11:01:44] <WikiTideLSBot>	 !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s
[11:01:45] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[11:01:46] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[11:02:02] <WikiTideRelay>	 <songngu.xyz> I have must misread it somewhere
[11:03:11] <WikiTideRelay>	 <songngu.xyz, replying to reception123> let us try that
[11:04:16] <WikiTideRelay>	 <songngu.xyz, replying to WikiTideLSBot> did we forgot the `$1` somewhere?
[11:04:49] <WikiTideRelay>	 <songngu.xyz> that doesn't looks good to me
[11:05:08] <WikiTideRelay>	 <cosmicalpha> It might have just logged wrong
[11:05:36] <WikiTideRelay>	 <cosmicalpha> I need to sleep it is 6AM so night
[11:05:52] <WikiTideRelay>	 <songngu.xyz, replying to cosmicalpha> gn
[11:06:33] <WikiTideRelay>	 <songngu.xyz> look at the pagepath and the filepath we noticed that we still have to put `$1` after the URLs
[12:18:48] <WikiTideRelay>	 <songngu.xyz> Ah wait, the site links work, but we have to fix the page and file path
[12:26:39] <WikiTideRelay>	 <songngu.xyz> oof
[12:34:09] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query.
[12:54:14] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[12:57:43] <icinga-wikitide>	 PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query.
[13:07:39] <icinga-wikitide>	 PROBLEM - nexttide.org - LetsEncrypt on sslhost is CRITICAL: connect to address nexttide.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[13:11:40] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[13:27:44] <icinga-wikitide>	 RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[13:33:15] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[13:43:19] <icinga-wikitide>	 PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[13:54:17] <icinga-wikitide>	 RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[14:02:46] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query.
[14:32:43] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[14:36:44] <icinga-wikitide>	 RECOVERY - nexttide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nexttide.org' will expire on Sun 14 Jan 2024 05:21:30 PM GMT +0000.
[15:08:40] <icinga-wikitide>	 PROBLEM - ldap1 LDAP on ldap1 is CRITICAL: Could not bind to the LDAP server
[15:28:59] <icinga-wikitide>	 PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query.
[15:29:46] <icinga-wikitide>	 PROBLEM - os1 Puppet on os1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal]
[15:32:34] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query.
[15:40:42] <icinga-wikitide>	 PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[15:58:47] <icinga-wikitide>	 PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[16:02:30] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.
[16:05:43] <botifico-49a823b>	 [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/2b2b2467e8fc...c48db9699de8
[16:05:44] <botifico-49a823b>	 [02WikiTideOrg/ssl] 07WikiTideSSLBot 03c48db96 - Bot: Update SSL cert for wikitide.net
[16:28:31] <icinga-wikitide>	 PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query.
[16:39:07] <icinga-wikitide>	 RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[16:58:13] <icinga-wikitide>	 RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[17:19:00] <icinga-wikitide>	 PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[17:50:00] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6cf73bfdfda0...13affe1a7593
[17:50:03] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0313affe1 - Remove extra spacing Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[18:02:13] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[18:16:51] <icinga-wikitide>	 RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[18:27:57] <icinga-wikitide>	 PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: LOAD CRITICAL - total load average: 9.46, 11.68, 5.32
[18:29:50] <icinga-wikitide>	 PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 1.46, 8.00, 4.71
[18:31:44] <icinga-wikitide>	 RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.42, 5.58, 4.20
[20:07:43] <botifico-49a823b>	 [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/11b5ebccba7c...994ed6c24ac8
[20:07:44] <botifico-49a823b>	 [02WikiTideOrg/dns] 07Universal-Omega 03994ed6c - Add acme challenge for cert renewal Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[20:11:06] <botifico-49a823b>	 [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/c48db9699de8...50da889cd16f
[20:11:09] <botifico-49a823b>	 [02WikiTideOrg/ssl] 07WikiTideSSLBot 0350da889 - Bot: Update SSL cert for wikitide.net
[20:13:03] <CosmicAlpha>	 !log running puppet on all servers
[20:13:05] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:17:25] <icinga-wikitide>	 PROBLEM - swiftproxy1 SSH on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:18:05] <icinga-wikitide>	 PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[20:20:08] <icinga-wikitide>	 PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:20:18] <icinga-wikitide>	 PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:20:41] <icinga-wikitide>	 PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:21:03] <icinga-wikitide>	 PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:23:13] <icinga-wikitide>	 RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set
[20:25:02] <icinga-wikitide>	 PROBLEM - swiftproxy1 Disk Space on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:27:48] <icinga-wikitide>	 RECOVERY - swiftproxy1 Disk Space on swiftproxy1 is OK: DISK OK - free space: / 23550MiB (88% inode=95%);
[20:30:33] <icinga-wikitide>	 PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[20:32:40] <icinga-wikitide>	 RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set
[20:32:41] <icinga-wikitide>	 RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 9.753 seconds response time. wikitide.org returns 2604:4300:a:24::113,63.141.240.6
[20:35:22] <icinga-wikitide>	 RECOVERY - swiftproxy1 conntrack_table_size on swiftproxy1 is OK: OK: nf_conntrack is 0 % full
[20:36:22] <icinga-wikitide>	 RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures
[20:36:31] <icinga-wikitide>	 RECOVERY - swiftproxy1 SSH on swiftproxy1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0)
[20:43:05] <icinga-wikitide>	 RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[20:46:17] <CosmicAlpha>	 !log reload nginx on swiftproxy1, mw*, and cp*
[20:46:19] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:55:52] <icinga-wikitide>	 RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Sun 03 Mar 2024 07:10:33 PM GMT +0000.
[20:58:33] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on bots1
[20:58:35] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:58:52] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on db1
[20:58:54] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:58:56] <icinga-wikitide>	 RECOVERY - bots1 APT on bots1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[20:58:58] <icinga-wikitide>	 RECOVERY - db1 APT on db1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[20:59:04] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on prometheus1
[20:59:06] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:59:16] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on jobchron1
[20:59:18] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:59:20] <icinga-wikitide>	 PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 0.30, 0.82, 7.93
[20:59:28] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on graylog1
[20:59:30] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:59:40] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on mem1
[20:59:42] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[20:59:50] <icinga-wikitide>	 RECOVERY - jobchron1 APT on jobchron1 is OK: APT OK: 19 packages available for upgrade (0 critical updates).
[20:59:55] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, libtiff-dev, libtiff6, and libtiffxx6 on mw2
[20:59:56] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:00:07] <icinga-wikitide>	 RECOVERY - mem1 APT on mem1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:00:07] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, libtiff-dev, libtiff6, and libtiffxx6 on mw1
[21:00:09] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:00:14] <icinga-wikitide>	 RECOVERY - mw2 APT on mw2 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[21:00:17] <icinga-wikitide>	 RECOVERY - prometheus1 APT on prometheus1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:00:18] <icinga-wikitide>	 RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:00:19] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on ldap1
[21:00:21] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:00:51] <icinga-wikitide>	 RECOVERY - mw1 APT on mw1 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[21:00:58] <icinga-wikitide>	 RECOVERY - ldap1 APT on ldap1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:00:59] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages intel-microcode, libpq5, libnghttp2-14, and libtiff6 on mon1
[21:01:01] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:01:20] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on os1
[21:01:22] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:01:33] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on puppet1
[21:01:34] <icinga-wikitide>	 RECOVERY - puppet1 APT on puppet1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:01:35] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:01:46] <icinga-wikitide>	 RECOVERY - mon1 APT on mon1 is OK: APT OK: 32 packages available for upgrade (0 critical updates).
[21:01:47] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on cp1
[21:01:48] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:01:49] <icinga-wikitide>	 RECOVERY - os1 APT on os1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:02:06] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, libtiff-dev, libtiff6, and libtiffxx6 on jobrunner1
[21:02:08] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:02:21] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on bast1
[21:02:23] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:02:30] <icinga-wikitide>	 RECOVERY - cp1 APT on cp1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:02:33] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on mail1
[21:02:35] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:02:39] <icinga-wikitide>	 RECOVERY - mail1 APT on mail1 is OK: APT OK: 18 packages available for upgrade (0 critical updates).
[21:02:48] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on ns1
[21:02:49] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:02:59] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on phorge1
[21:03:00] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:03:05] <icinga-wikitide>	 RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:03:18] <icinga-wikitide>	 RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.36, 0.51, 6.18
[21:03:35] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages chromium, chromium-sandbox, chromium-common, libnghttp2-14, libtiff-dev, libtiff6, and libtiffxx6 on services1
[21:03:37] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:03:48] <icinga-wikitide>	 RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[21:03:48] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, libtiff-dev, libtiff6, and libtiffxx6 on test1
[21:03:50] <icinga-wikitide>	 RECOVERY - bast1 APT on bast1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:03:50] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:04:01] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on swiftac1
[21:04:03] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:04:14] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on swiftproxy1
[21:04:15] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:04:23] <icinga-wikitide>	 RECOVERY - phorge1 APT on phorge1 is OK: APT OK: 16 packages available for upgrade (0 critical updates).
[21:04:33] <icinga-wikitide>	 RECOVERY - swiftproxy1 APT on swiftproxy1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:04:47] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages intel-microcode, libnghttp2-14, and libtiff6 on matomo1
[21:04:49] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:04:50] <icinga-wikitide>	 RECOVERY - services1 APT on services1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:05:03] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on swiftobject1
[21:05:04] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:05:10] <icinga-wikitide>	 RECOVERY - swiftac1 APT on swiftac1 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:05:13] <icinga-wikitide>	 RECOVERY - swiftobject1 APT on swiftobject1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:05:19] <icinga-wikitide>	 RECOVERY - test1 APT on test1 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[21:05:39] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query.
[21:06:20] <icinga-wikitide>	 RECOVERY - matomo1 APT on matomo1 is OK: APT OK: 17 packages available for upgrade (0 critical updates).
[21:07:28] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on cp4
[21:07:30] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:07:39] <icinga-wikitide>	 RECOVERY - cp4 APT on cp4 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:09:56] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on cp2
[21:09:58] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:10:56] <icinga-wikitide>	 RECOVERY - cp2 APT on cp2 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:12:24] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on cp6
[21:12:25] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:14:06] <icinga-wikitide>	 RECOVERY - cp6 APT on cp6 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:14:54] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on cp3
[21:14:56] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:15:50] <icinga-wikitide>	 RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[21:17:31] <WikiTideLSBot>	 !log [universalomega@puppet1] Upgraded packages libnghttp2-14, and libtiff6 on cp5
[21:17:33] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[21:17:46] <icinga-wikitide>	 RECOVERY - cp5 APT on cp5 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[21:35:34] <icinga-wikitide>	 PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL
[22:05:28] <icinga-wikitide>	 RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK
[22:41:41] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/13affe1a7593...e699074c073b
[22:41:42] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e699074 - Add cadvisor Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:44:39] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/e699074c073b...2849cc24b3b3
[22:44:40] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 032849cc2 - Add cadvisor package Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:46:25] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2849cc24b3b3...d10c1fdd7064
[22:46:27] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03d10c1fd - Use deb package Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:47:40] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03prometheusexporternutcracker [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/cc8396bf2bed
[22:47:41] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03cc8396b - Add prometheus::exporter::nutcracker Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:47:44] <botifico-49a823b>	 [02puppet] 07Universal-Omega created branch 03prometheusexporternutcracker - 13https://github.com/WikiTideOrg/puppet
[22:47:47] <botifico-49a823b>	 [02puppet] 07Universal-Omega opened pull request 03#38: Add prometheus::exporter::nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/38
[22:49:12] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03prometheusexporternutcracker [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/cc8396bf2bed...13390587b12a
[22:49:15] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 031339058 - Add package Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:49:17] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#38: Add prometheus::exporter::nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/38
[22:50:38] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03prometheusexporternutcracker [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/13390587b12a...f45a2076e1f4
[22:50:40] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03f45a207 - Use deb Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:50:42] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#38: Add prometheus::exporter::nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/38
[22:51:24] <botifico-49a823b>	 [02puppet] 07Universal-Omega closed pull request 03#38: Add prometheus::exporter::nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/38
[22:51:25] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/d10c1fdd7064...19caedd173ac
[22:51:26] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0319caedd - Add prometheus::exporter::nutcracker Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:51:29] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03prometheusexporternutcracker
[22:51:32] <botifico-49a823b>	 [02puppet] 07Universal-Omega deleted branch 03prometheusexporternutcracker - 13https://github.com/WikiTideOrg/puppet
[22:53:52] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/d02a0b1bccb8...46fbd8ea0f41
[22:53:54] <botifico-49a823b>	 [02WikiTideOrg/mw-config] 07AgentIsai 0346fbd8e - Update Share extension options Signed-off-by: Agent_Isai <43097272+AgentIsai@users.noreply.github.com>
[22:54:49] <wikitide-github>	 WikiTideOrg/mw-config - AgentIsai the build passed.
[22:55:21] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 0338 commits to 03mcrouter [+19/-13/±98] 13https://github.com/WikiTideOrg/puppet/compare/d9de5291714b...2671b18016a5
[22:55:23] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 032671b18 - Merge branch 'master' into mcrouter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:55:26] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[22:56:23] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/2671b18016a5...41ad33da5fc3
[22:56:26] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 0341ad33d - Add prometheus-mcrouter-exporter deb Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[22:56:28] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[23:00:45] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/41ad33da5fc3...a8c287d252cb
[23:00:46] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03a8c287d - Add exporter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:00:48] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[23:01:43] <WikiTideLSBot>	 !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1]
[23:01:44] <WikiTideLSBot>	 !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s
[23:01:45] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:01:47] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:04:14] <icinga-wikitide>	 PROBLEM - mw2 APT on mw2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[23:04:45] <icinga-wikitide>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[prometheus-nutcracker-exporter]
[23:05:20] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a8c287d252cb...ae01f2a81f40
[23:05:21] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03ae01f2a - Add prometheus::exporter::mcrouter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:05:22] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[23:06:50] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/ae01f2a81f40...476aa821a7d9
[23:06:53] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03476aa82 - Add exporter hiera Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:06:56] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[23:09:37] <icinga-wikitide>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[prometheus-nutcracker-exporter]
[23:10:50] <icinga-wikitide>	 PROBLEM - mw1 APT on mw1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[23:13:53] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/19caedd173ac...076bf7262326
[23:13:55] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03076bf72 - role::prometheus add nutcracker exporter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:14:48] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 037 commits to 03mcrouter [+10/-1/±11] 13https://github.com/WikiTideOrg/puppet/compare/476aa821a7d9...703463198e74
[23:14:51] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03685aa5f - mediawiki: use mcrouter instead of nutcracker
[23:14:54] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07github-actions 037d498eb - CI: lint puppet code to standards Check commit and GitHub actions for more details
[23:14:55] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03ac13f1d - Add package
[23:14:57] <botifico-49a823b>	 [02WikiTideOrg/puppet] ... and 4 more commits.
[23:15:00] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[23:16:57] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03mcrouter [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/703463198e74...ed8b463c8b44
[23:16:58] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03ed8b463 - role::prometheus add mcrouter exporter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:17:00] <botifico-49a823b>	 [02puppet] 07Universal-Omega synchronize pull request 03#32: mediawiki: use mcrouter instead of nutcracker - 13https://github.com/WikiTideOrg/puppet/pull/32
[23:18:27] <WikiTideLSBot>	 !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1]
[23:18:29] <WikiTideLSBot>	 !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 1s
[23:18:29] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:18:30] <WikiTideLogbot>	 Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log
[23:19:54] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/076bf7262326...8abd9464792f
[23:19:56] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 038abd946 - prometheus::exporter::nutcracker: ensure python3-prometheus-client is installed Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:21:37] <icinga-wikitide>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[23:21:47] <icinga-wikitide>	 PROBLEM - jobrunner1 APT on jobrunner1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[23:22:45] <icinga-wikitide>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[23:22:46] <icinga-wikitide>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[prometheus-nutcracker-exporter]
[23:22:50] <icinga-wikitide>	 RECOVERY - mw1 APT on mw1 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[23:23:48] <icinga-wikitide>	 RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[23:24:14] <icinga-wikitide>	 RECOVERY - mw2 APT on mw2 is OK: APT OK: 22 packages available for upgrade (0 critical updates).
[23:24:46] <icinga-wikitide>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[23:26:49] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8abd9464792f...0cd03c7dc8ce
[23:26:52] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 030cd03c7 - Add firewall rule for nutcracker exporter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>
[23:47:41] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0cd03c7dc8ce...e0da5177c7af
[23:47:42] <botifico-49a823b>	 [02WikiTideOrg/puppet] 07Universal-Omega 03e0da517 - Disable nutcracker exporter Signed-off-by: CosmicAlpha <cosmic.alpha24@gmail.com>