[00:09:09] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:20:01] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:39:00] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [00:47:57] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [01:01:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e0da5177c7af...cfb3333e4b6e [01:01:16] [02WikiTideOrg/puppet] 07Universal-Omega 03cfb3333 - check_reverse_dns: don't cache Signed-off-by: CosmicAlpha [01:02:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/cfb3333e4b6e...eb5deb73b70c [01:02:40] [02WikiTideOrg/puppet] 07Universal-Omega 03eb5deb7 - Use 127.0.0.1 Signed-off-by: CosmicAlpha [01:02:41] WikiTideOrg/puppet - Universal-Omega the build passed. [01:15:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/eb5deb73b70c...8d11f1306c20 [01:15:33] [02WikiTideOrg/puppet] 07Universal-Omega 038d11f13 - Fix regex Signed-off-by: CosmicAlpha [01:16:52] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [01:40:41] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [02:08:30] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/commit/303c6c6c4a63 [02:08:32] [02WikiTideOrg/puppet] 07Universal-Omega 03303c6c6 - Add raid status checks Signed-off-by: CosmicAlpha [02:08:35] [02puppet] 07Universal-Omega created branch 03raid - 13https://github.com/WikiTideOrg/puppet [02:08:37] [02puppet] 07Universal-Omega opened pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:09:51] WikiTideOrg/puppet - Universal-Omega the build passed. [02:10:03] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:16:47] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/303c6c6c4a63...853cdb54b133 [02:16:48] [02WikiTideOrg/puppet] 07Universal-Omega 03853cdb5 - Add Signed-off-by: CosmicAlpha [02:16:49] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:17:29] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [02:17:43] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:18:05] WikiTideOrg/puppet - Universal-Omega the build passed. [02:18:51] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/853cdb54b133...d13d2528bf70 [02:18:54] [02WikiTideOrg/puppet] 07Universal-Omega 03d13d252 - Add Signed-off-by: CosmicAlpha [02:18:56] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:19:46] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:20:17] WikiTideOrg/puppet - Universal-Omega the build passed. [02:21:52] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/d13d2528bf70...488e9b60aaa3 [02:21:54] [02WikiTideOrg/puppet] 07Universal-Omega 03488e9b6 - Add Signed-off-by: CosmicAlpha [02:21:56] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:22:45] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:23:07] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:23:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/488e9b60aaa3...4b644be0724a [02:23:11] [02WikiTideOrg/puppet] 07Universal-Omega 034b644be - Add Signed-off-by: CosmicAlpha [02:23:14] WikiTideOrg/puppet - Universal-Omega the build passed. [02:23:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4b644be0724a...f2c36cc3c379 [02:23:48] [02WikiTideOrg/puppet] 07Universal-Omega 03f2c36cc - - for now Signed-off-by: CosmicAlpha [02:23:51] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:23:59] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:24:25] WikiTideOrg/puppet - Universal-Omega the build passed. [02:24:38] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:25:04] WikiTideOrg/puppet - Universal-Omega the build passed. [02:26:15] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f2c36cc3c379...f023d54b9840 [02:26:18] [02WikiTideOrg/puppet] 07Universal-Omega 03f023d54 - Add Signed-off-by: CosmicAlpha [02:26:19] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:27:08] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:28:03] WikiTideOrg/puppet - Universal-Omega the build passed. [02:38:50] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f023d54b9840...32839e22ef83 [02:38:53] [02WikiTideOrg/puppet] 07Universal-Omega 0332839e2 - Use phorge and remove DC Signed-off-by: CosmicAlpha [02:38:54] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:39:42] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:40:02] WikiTideOrg/puppet - Universal-Omega the build passed. [02:43:28] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [02:43:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/32839e22ef83...2a772f23ca21 [02:43:33] [02WikiTideOrg/puppet] 07Universal-Omega 032a772f2 - write to irc and use icinga2 Signed-off-by: CosmicAlpha [02:44:26] WikiTideOrg/puppet - Universal-Omega the build has errored. [02:44:42] [1701744282] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:44:52] WikiTideOrg/puppet - Universal-Omega the build passed. [02:47:11] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [02:47:29] [1701744449] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:47:31] [1701744451] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:49:35] [1701744575] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:49:36] [1701744575] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:02] [1701744662] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:03] [1701744663] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:04] [1701744664] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:06] [1701744666] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:07] [1701744667] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:08] [1701744668] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:10] [1701744670] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:11] [1701744671] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:12] [1701744672] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:14] [1701744674] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:15] [1701744675] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:16] [1701744676] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:18] [1701744678] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:19] [1701744679] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:20] [1701744680] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:22] [1701744681] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:23] [1701744683] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:24] [1701744684] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:25] oh.... that isn't what I meant to do [02:51:26] [1701744685] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:27] [1701744687] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:28] [1701744688] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:51:29] [1701744689] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T-1 [02:52:26] Whoops... [03:04:48] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/2a772f23ca21...3be76e9f62f4 [03:04:50] [02WikiTideOrg/puppet] 07Universal-Omega 033be76e9 - Add monitoring bot config Signed-off-by: CosmicAlpha [03:04:53] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:05:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/3be76e9f62f4...4e83a4b2be7a [03:05:29] [02WikiTideOrg/puppet] 07Universal-Omega 034e83a4b - Fix name Signed-off-by: CosmicAlpha [03:05:32] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:05:41] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:06:13] WikiTideOrg/puppet - Universal-Omega the build passed. [03:06:22] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:06:46] WikiTideOrg/puppet - Universal-Omega the build passed. [03:08:43] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:09:11] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4e83a4b2be7a...fd0376877726 [03:09:13] [02WikiTideOrg/puppet] 07Universal-Omega 03fd03768 - Add bot config Signed-off-by: CosmicAlpha [03:09:14] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:10:07] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:10:36] WikiTideOrg/puppet - Universal-Omega the build passed. [03:11:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8d11f1306c20...120698115f71 [03:11:39] [02WikiTideOrg/puppet] 07Universal-Omega 031206981 - Add phorge_token Signed-off-by: CosmicAlpha [03:12:21] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/120698115f71...51f49ffe7edc [03:12:22] [02WikiTideOrg/puppet] 07Universal-Omega 0351f49ff - Revert Signed-off-by: CosmicAlpha [03:12:40] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/fd0376877726...4b14157f509b [03:12:42] [02WikiTideOrg/puppet] 07Universal-Omega 034b14157 - Add phorge token Signed-off-by: CosmicAlpha [03:12:43] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:13:15] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:13:16] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4b14157f509b...fdb7618542e2 [03:13:19] [02WikiTideOrg/puppet] 07Universal-Omega 03fdb7618 - Enable Signed-off-by: CosmicAlpha [03:13:31] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:14:03] WikiTideOrg/puppet - Universal-Omega the build passed. [03:14:09] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:14:12] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:14:12] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/fdb7618542e2...9153a8b502cc [03:14:14] [02WikiTideOrg/puppet] 07Universal-Omega 039153a8b - Enable task creation logic Signed-off-by: CosmicAlpha [03:14:30] WikiTideOrg/puppet - Universal-Omega the build passed. [03:15:02] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:15:31] WikiTideOrg/puppet - Universal-Omega the build passed. [03:17:53] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:17:56] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/9153a8b502cc...9f21ed87eae0 [03:17:58] [02WikiTideOrg/puppet] 07Universal-Omega 039f21ed8 - Drop support for controllers we don't use Signed-off-by: CosmicAlpha [03:18:44] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:19:10] WikiTideOrg/puppet - Universal-Omega the build passed. [03:19:44] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:22:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/9f21ed87eae0...36dbf12939a8 [03:22:29] [02WikiTideOrg/puppet] 07Universal-Omega 0336dbf12 - Fix Signed-off-by: CosmicAlpha [03:22:30] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:23:48] WikiTideOrg/puppet - Universal-Omega the build passed. [03:25:01] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/36dbf12939a8...0a012334c54a [03:25:02] [02WikiTideOrg/puppet] 07Universal-Omega 030a01233 - Add megaraid Signed-off-by: CosmicAlpha [03:25:04] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:25:52] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:26:23] WikiTideOrg/puppet - Universal-Omega the build passed. [03:26:57] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0a012334c54a...09242fede81a [03:27:00] [02WikiTideOrg/puppet] 07Universal-Omega 0309242fe - Fix Signed-off-by: CosmicAlpha [03:27:01] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:27:12] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/09242fede81a...d9d9ac76b9da [03:27:15] [02WikiTideOrg/puppet] 07Universal-Omega 03d9d9ac7 - Fix Signed-off-by: CosmicAlpha [03:27:17] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:27:51] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:28:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/d9d9ac76b9da...9ef3049fc714 [03:28:18] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:28:18] [02WikiTideOrg/puppet] 07Universal-Omega 039ef3049 - Add get-raid-status-megacli Signed-off-by: CosmicAlpha [03:28:20] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:28:23] WikiTideOrg/puppet - Universal-Omega the build passed. [03:28:32] WikiTideOrg/puppet - Universal-Omega the build passed. [03:29:13] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:29:34] WikiTideOrg/puppet - Universal-Omega the build passed. [03:32:41] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/9ef3049fc714...94042620825c [03:32:42] [02WikiTideOrg/puppet] 07Universal-Omega 039404262 - Add raid Signed-off-by: CosmicAlpha [03:32:43] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:33:40] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:33:58] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/94042620825c...af04161d9762 [03:33:59] [02WikiTideOrg/puppet] 07Universal-Omega 03af04161 - Add check_raid Signed-off-by: CosmicAlpha [03:34:00] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:34:16] WikiTideOrg/puppet - Universal-Omega the build passed. [03:34:55] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:35:32] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/af04161d9762...159b6461c19e [03:35:35] [02WikiTideOrg/puppet] 07Universal-Omega 03159b646 - Add facter Signed-off-by: CosmicAlpha [03:35:36] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:35:37] WikiTideOrg/puppet - Universal-Omega the build passed. [03:36:25] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:40:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/159b6461c19e...4df22671890d [03:40:41] [02WikiTideOrg/puppet] 07Universal-Omega 034df2267 - Add raid monitoring Signed-off-by: CosmicAlpha [03:40:43] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:40:48] WikiTideOrg/puppet - Universal-Omega the build passed. [03:41:49] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:41:57] WikiTideOrg/puppet - Universal-Omega the build passed. [03:42:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4df22671890d...e48568b9bb29 [03:42:30] [02WikiTideOrg/puppet] 07Universal-Omega 03e48568b - Add raid hiera Signed-off-by: CosmicAlpha [03:42:31] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:43:22] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:43:46] WikiTideOrg/puppet - Universal-Omega the build passed. [03:45:33] [02puppet] 07Universal-Omega edited pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:47:29] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:47:31] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e48568b9bb29...61241db973a8 [03:47:34] [02WikiTideOrg/puppet] 07Universal-Omega 0361241db - - Signed-off-by: CosmicAlpha [03:48:26] WikiTideOrg/puppet - Universal-Omega the build has errored. [03:48:45] WikiTideOrg/puppet - Universal-Omega the build passed. [03:49:51] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [03:49:52] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/61241db973a8...80762ec4611f [03:49:53] [02WikiTideOrg/puppet] 07Universal-Omega 0380762ec - Fix Signed-off-by: CosmicAlpha [03:50:41] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:51:01] WikiTideOrg/puppet - Universal-Omega the build passed. [03:55:42] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [04:07:35] [1701749255] ACKNOWLEDGE_SVC_PROBLEM;TESTING;RAID Status;2;1;0;nagiosadmin;RAID handler auto-ack: https://issue-tracker.wikitide.org/T97 [04:09:50] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/80762ec4611f...f871ca664a33 [04:09:52] [02WikiTideOrg/puppet] 07Universal-Omega 03f871ca6 - Fix Signed-off-by: CosmicAlpha [04:09:55] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [04:10:14] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f871ca664a33...016e99c868b8 [04:10:15] [02WikiTideOrg/puppet] 07Universal-Omega 03016e99c - Fix path Signed-off-by: CosmicAlpha [04:10:17] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [04:11:27] WikiTideOrg/puppet - Universal-Omega the build passed. [04:15:02] WikiTideOrg/puppet - Universal-Omega the build passed. [04:16:35] [02puppet] 07Universal-Omega synchronize pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [04:16:37] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03raid [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/016e99c868b8...ffac0bac4ecc [04:16:40] [02WikiTideOrg/puppet] 07Universal-Omega 03ffac0ba - Fix icinga URL Signed-off-by: CosmicAlpha [04:21:45] WikiTideOrg/puppet - Universal-Omega the build passed. [04:25:16] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [04:45:00] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [04:46:30] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/commit/0bb0fbeefb20 [04:46:31] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 030bb0fbe - Cleanup code Signed-off-by: CosmicAlpha [04:46:33] [02MediaWikiDebugJS] 07Universal-Omega created branch 03cleanup - 13https://github.com/WikiTideOrg/MediaWikiDebugJS [04:46:34] [02MediaWikiDebugJS] 07Universal-Omega opened pull request 03#8: Cleanup code - 13https://github.com/WikiTideOrg/MediaWikiDebugJS/pull/8 [04:50:43] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideDebug/commit/1e114cbce26d [04:50:46] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega 031e114cb - Cleanup code Signed-off-by: CosmicAlpha [04:50:48] [02WikiTideDebug] 07Universal-Omega created branch 03cleanup - 13https://github.com/WikiTideOrg/WikiTideDebug [04:50:50] [02WikiTideDebug] 07Universal-Omega opened pull request 03#1: Cleanup code - 13https://github.com/WikiTideOrg/WikiTideDebug/pull/1 [04:53:04] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideDebug/compare/1e114cbce26d...b0ca53137564 [04:53:07] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega 03b0ca531 - Cleanup code Signed-off-by: CosmicAlpha [04:53:08] [02WikiTideDebug] 07Universal-Omega synchronize pull request 03#1: Cleanup code - 13https://github.com/WikiTideOrg/WikiTideDebug/pull/1 [04:54:51] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideDebug/compare/b0ca53137564...8facce7c6db3 [04:54:53] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega 038facce7 - Cleanup code Signed-off-by: CosmicAlpha [04:54:55] [02WikiTideDebug] 07Universal-Omega synchronize pull request 03#1: Cleanup code - 13https://github.com/WikiTideOrg/WikiTideDebug/pull/1 [04:56:21] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/compare/0bb0fbeefb20...06fdd81ab2f5 [04:56:24] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 0306fdd81 - Cleanup code Signed-off-by: CosmicAlpha [04:56:26] [02MediaWikiDebugJS] 07Universal-Omega synchronize pull request 03#8: Cleanup code - 13https://github.com/WikiTideOrg/MediaWikiDebugJS/pull/8 [04:57:32] [02MediaWikiDebugJS] 07Universal-Omega synchronize pull request 03#8: Cleanup code - 13https://github.com/WikiTideOrg/MediaWikiDebugJS/pull/8 [04:57:33] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/compare/06fdd81ab2f5...7739693bf1b1 [04:57:35] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 037739693 - Re-add comment Signed-off-by: CosmicAlpha [05:01:30] [02MediaWikiDebugJS] 07Universal-Omega closed pull request 03#8: Cleanup code - 13https://github.com/WikiTideOrg/MediaWikiDebugJS/pull/8 [05:01:33] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/compare/26abc4b55585...5e6a36c7254b [05:01:34] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 035e6a36c - Cleanup code Signed-off-by: CosmicAlpha [05:01:36] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega deleted branch 03cleanup [05:01:39] [02MediaWikiDebugJS] 07Universal-Omega deleted branch 03cleanup - 13https://github.com/WikiTideOrg/MediaWikiDebugJS [05:05:47] [02WikiTideDebug] 07Universal-Omega synchronize pull request 03#1: Cleanup code - 13https://github.com/WikiTideOrg/WikiTideDebug/pull/1 [05:05:50] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideDebug/compare/8facce7c6db3...12599a41b6ca [05:05:52] [02WikiTideOrg/WikiTideDebug] 07Universal-Omega 0312599a4 - Fix Signed-off-by: CosmicAlpha [05:32:18] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/compare/5e6a36c7254b...fddf6f729db3 [05:32:19] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 03fddf6f7 - Add support for getting DB name from comment Signed-off-by: CosmicAlpha [05:35:20] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/compare/fddf6f729db3...2ae4f080afd6 [05:35:21] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 032ae4f08 - replace cache-bfi-krnt Signed-off-by: CosmicAlpha [05:43:37] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:52:10] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03dbname-comment [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/commit/8fefb0b0ed5c [05:52:12] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 038fefb0b - Only get DB name from comment Signed-off-by: CosmicAlpha [05:52:15] [02MediaWikiDebugJS] 07Universal-Omega created branch 03dbname-comment - 13https://github.com/WikiTideOrg/MediaWikiDebugJS [05:52:16] [02MediaWikiDebugJS] 07Universal-Omega opened pull request 03#9: Only get DB name from comment - 13https://github.com/WikiTideOrg/MediaWikiDebugJS/pull/9 [07:40:57] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:11:53] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:33:52] [02WikiTideOrg/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiTideOrg/dns/compare/994ed6c24ac8...24e62ee4a25a [08:33:55] [02WikiTideOrg/dns] 07Reception123 0324e62ee - add mlrpgspeedruns.com zone Signed-off-by: Reception123 [12:30:52] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:01:48] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:43:20] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:43:24] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:43:24] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:45:45] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:45:46] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:49:55] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:50:45] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:51:01] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:51:17] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:51:37] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:51:54] PROBLEM - matomo1 Puppet on matomo1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:52:56] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:53:18] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:54:14] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:54:37] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:56:53] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:58:27] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:59:10] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:59:24] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [14:01:06] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:03:02] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:09] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:07:44] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:07:46] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:08:00] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:09:56] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:12:15] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:45] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:17:01] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:18:16] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:18:48] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:19:16] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:19:39] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:21:16] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:21:54] RECOVERY - matomo1 Puppet on matomo1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:22:42] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:22:56] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:24:52] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:25:02] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:26:27] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:27:10] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:31:19] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:32:09] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:33:02] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:35:44] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:35:45] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:35:58] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:37:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:40:15] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:41:20] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:41:24] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:41:24] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:43:45] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:50:58] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [14:51:22] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [15:06:51] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [15:14:28] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/50da889cd16f...e7a490145f87 [15:14:30] [02WikiTideOrg/ssl] 07WikiTideSSL 03e7a4901 - Bot: Adding SSL cert for mlrpgspeedruns.com Certificate committed by reception [15:19:37] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [15:21:03] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [15:22:56] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:23:18] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:26:18] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:27:46] @reception123 make sure to `chown -R puppet:puppet /etc/puppet/puppetlabs/ssl-keys` [15:27:51] or ssl-key, I forgot the dir name [15:28:03] and then use salt to restart nginx on all failed cp's [15:29:06] ah, thanks! I thought that was fixed already [15:29:10] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:30:26] [1/3] ```[root@puppet1.wikitide.net:/etc/puppet]# ls [15:30:27] [2/3] code puppetserver ssl [15:30:27] [3/3] ``` hmm, there should be a 'puppetlabs' here [15:33:49] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/51f49ffe7edc...07b46012294c [15:33:51] [02WikiTideOrg/puppet] 07AgentIsai 0307b4601 - Add bash defaults to all servers [15:33:55] PROBLEM - cp1 Nginx Backend for phorge1 on cp1 is CRITICAL: connect to address localhost and port 8200: Connection refused [15:34:02] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: connect to address wikitide.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:34:08] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:34:23] PROBLEM - www.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.polandballwiki.com reverse DNS resolves to vps-a7e4a0a2.vps.ovh.net [15:34:25] PROBLEM - cp1 HTTPS on cp1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp1.wikitide.net port 443 after 0 ms: Couldn't connect to server [15:34:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.79.55.151/cpweb, 2607:5300:205:200::2aa8/cpweb, 51.75.170.66/cpweb, 2001:41d0:801:2000::4089/cpweb, 15.235.167.159/cpweb, 2402:1f00:8000:800::1b3d/cpweb, 63.141.240.6/cpweb, 2604:4300:a:24::113/cpweb [15:34:39] PROBLEM - cp1 Nginx Backend for mail1 on cp1 is CRITICAL: connect to address localhost and port 8202: Connection refused [15:34:47] PROBLEM - cp1 Nginx Backend for mw2 on cp1 is CRITICAL: connect to address localhost and port 8101: Connection refused [15:34:51] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [15:34:52] PROBLEM - cp1 Nginx Backend for mw1 on cp1 is CRITICAL: connect to address localhost and port 8100: Connection refused [15:35:16] PROBLEM - cp1 Nginx Backend for puppet1 on cp1 is CRITICAL: connect to address localhost and port 8201: Connection refused [15:35:17] PROBLEM - cp1 Nginx Backend for matomo1 on cp1 is CRITICAL: connect to address localhost and port 8204: Connection refused [15:35:23] PROBLEM - cp2 Nginx Backend for mw1 on cp2 is CRITICAL: connect to address localhost and port 8100: Connection refused [15:35:25] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [15:35:27] PROBLEM - cp5 Nginx Backend for phorge1 on cp5 is CRITICAL: connect to address localhost and port 8200: Connection refused [15:35:27] PROBLEM - cp5 Nginx Backend for puppet1 on cp5 is CRITICAL: connect to address localhost and port 8201: Connection refused [15:35:37] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: connect to address localhost and port 8200: Connection refused [15:35:41] PROBLEM - cp3 Nginx Backend for phorge1 on cp3 is CRITICAL: connect to address localhost and port 8200: Connection refused [15:35:47] PROBLEM - cp1 Nginx Backend for test1 on cp1 is CRITICAL: connect to address localhost and port 8180: Connection refused [15:35:48] PROBLEM - cp5 Nginx Backend for matomo1 on cp5 is CRITICAL: connect to address localhost and port 8204: Connection refused [15:35:58] PROBLEM - cp3 GDNSD Datacenters on cp3 is CRITICAL: CRITICAL - 8 datacenters are down: 51.79.55.151/cpweb, 2607:5300:205:200::2aa8/cpweb, 51.75.170.66/cpweb, 2001:41d0:801:2000::4089/cpweb, 15.235.167.159/cpweb, 2402:1f00:8000:800::1b3d/cpweb, 63.141.240.6/cpweb, 2604:4300:a:24::113/cpweb [15:35:59]