[00:15:28] <MacFan4000>	 I’ve been around MH on and off since late 2016 and I’ve never seen our icinga not have any issues
[00:15:54] <MacFan4000>	 (Of course in the early days we were using icinga v1)
[00:20:59] <MacFan4000>	 Of course lately the alerts have mostly been SSL and APT
[00:24:12] <MacFan4000>	 lol now we just need to resolve the 237 open phorge tickets
[00:28:51] <CosmicAlpha>	 lol yeah
[00:36:55] <MirahezeRelay>	 <pixldev> only 237?
[00:37:09] <MirahezeRelay>	 <pixldev> I say that sarcastically but I’m also legitimately surprised
[00:40:17] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dd2de5e5b694...89e2447423eb
[00:40:19] <botifico>	 [02puppet] 07Universal-Omega 0389e2447 - mwdeploy: add user agent to requests headers
[00:41:56] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[01:12:16] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[01:21:07] <MirahezeRelay>	 <cosmicalpha> !log update cloudflare origin rules and cache rules to let requests from internal IP ranges be able to use X-WikiTide-Debug without access key so mwdeploy can work
[01:21:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:21:43] <MirahezeRelay>	 <cosmicalpha> MacFan4000: FYI, that + adding a UA to mwdeploy fixed it.
[01:28:49] <BlankEclair>	 huh, MirahezeLogbot also works from discord, neat
[01:33:07] <CosmicAlpha>	 BlankEclair: yeah, Pix to thank for that
[01:34:39] <BlankEclair>	 https://issue-tracker.miraheze.org/maniphest/query/o_aquIReey_Q/#R; well...
[01:35:51] <MirahezeLSBot>	 !log [universalomega@bots171] Restart irclogserverbot
[01:35:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:36:08] <CosmicAlpha>	 Yay its back lol
[01:36:23] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'OreDict'} to all
[01:36:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:36:29] <BlankEclair>	 wee
[01:36:36] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'OreDict'} to all - SUCCESS in 13s
[01:36:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:37:31] <CosmicAlpha>	 Busy couple of days fixing things lol
[01:41:32] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[01:47:49] <botifico>	 [02puppet] 07Universal-Omega pushed 0344 commits to 03patch-mwpatches [+11/-9/±87] 13https://github.com/miraheze/puppet/compare/d9b69a96ebde...dbfaf2af6763
[01:47:51] <botifico>	 [02puppet] 07Universal-Omega 03dbfaf2a - Merge branch 'master' into patch-mwpatches
[01:47:53] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3885: mwdeploy.py: add support for local patches - 13https://github.com/miraheze/puppet/pull/3885
[01:47:54] <botifico>	 [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3885: mwdeploy.py: add support for local patches - 13https://github.com/miraheze/puppet/pull/3885#issuecomment-2234785055
[01:49:06] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[02:03:35] <botifico>	 [02mw-config] 07hurohukidaikon commented on pull request 03#5643: Update UploadWizard settings for kagagawiki - 13https://github.com/miraheze/mw-config/pull/5643#issuecomment-2323116388
[02:19:21] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.99, 4.93, 2.30
[02:39:21] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.12, 7.45, 6.02
[02:39:56] <PixDeVl>	 what the hell is os162
[02:41:20] <icinga-miraheze>	 PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 6.38, 7.11, 6.07
[02:42:57] <CosmicAlpha>	 opensearch
[02:43:21] <icinga-miraheze>	 RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 4.41, 6.27, 5.90
[02:43:21] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[02:43:22] <PixDeVl>	 oh
[02:43:31] <PixDeVl>	 dont we use elastic?
[02:43:35] <CosmicAlpha>	 nope
[02:43:41] <PixDeVl>	 huh
[02:44:07] <PixDeVl>	 we need to get a community liasons to force the missing servers docs onto
[02:44:33] <CosmicAlpha>	 we use opensearch with elasticsearch compatability mode, but IIRC upstream says they will make true opensearch work for CirrusSearch by the end of the year but I don't fully buy it.
[02:45:07] <PixDeVl>	 yeah they were moving to move to OS then ES became OSS again
[02:47:03] <CosmicAlpha>	 I thought the ELK stack was EOL, did it come back or was I thinking completely wrong?
[02:48:57] <PixDeVl>	 ELK?
[02:49:52] <CosmicAlpha>	 Elasticsearch, Logstash, Kibana
[02:50:37] <CosmicAlpha>	 I thought it was EOL lol. If it isn't we should consider ELK rather than Graylog
[02:50:59] <PixDeVl>	 why?
[02:51:45] <CosmicAlpha>	 I dislike Graylog
[02:51:56] <PixDeVl>	 very well
[02:52:04] <PixDeVl>	 to the gallows with it
[02:52:05] <CosmicAlpha>	 It is inconvenient to access you have to proxy, etc...
[02:52:21] <PixDeVl>	 hm?
[02:52:37] <CosmicAlpha>	 See Tech:Graylog on meta
[02:59:00] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/89e2447423eb...a7f8c9ddec96
[02:59:01] <botifico>	 [02puppet] 07Universal-Omega 03a7f8c9d - Don't ignore some bots from relay
[03:12:36] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[03:16:05] <MirahezeRelay>	 <cosmicalpha> Well at least that works lol
[03:18:50] <Agent>	 quite nice to have again
[03:28:53] <CosmicAlpha>	 Yeah, it is
[05:22:17] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:28:35] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001989006996 secs
[05:37:26] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:39:24] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0004064440727 secs
[05:47:31] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[05:56:08] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[05:58:06] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001275241375 secs
[06:04:27] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:06:25] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001123249531 secs
[06:15:19] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:26:29] <MirahezeRelay>	 <orduin> ns2 having mini seizures
[07:00:08] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:02:06] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001544356346 secs
[07:02:14] <BlankEclair>	 ns2 also be having failures to dns
[07:02:50] <CosmicAlpha>	 Yeah... even when I am sshing in some random commands like ls fails because ns2 is not a valid hostname it thinks. That server acts up a lot
[07:03:03] <CosmicAlpha>	 Only once every 10 minutes or so
[07:04:02] <RhinosF1>	 CosmicAlpha: is something constantly resetting on it?
[07:04:33] <CosmicAlpha>	 No, network just fails
[07:05:46] <RhinosF1>	 Weird
[07:05:54] <RhinosF1>	 That needs fixing
[07:15:37] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[07:21:54] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[07:26:20] <BlankEclair>	 > [01/09/2024 17:02] <CosmicAlpha> Yeah... even when I am sshing in some random commands like ls fails because ns2 is not a valid hostname it thinks. That server acts up a lot
[07:26:22] <BlankEclair>	 what
[07:28:18] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 7.745623589e-05 secs
[07:36:00] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:44:09] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures
[07:44:20] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0003868341446 secs
[07:50:35] <RhinosF1>	 Also morning BlankEclair
[07:50:43] <BlankEclair>	 eh, close enough
[07:51:00] * RhinosF1 thinks it would be very nice if ns2 decided to stop being critical so everything could be happy
[07:51:13] <RhinosF1>	 BlankEclair: let me guess, 23:30 for you?
[07:51:20] <BlankEclair>	 17:51
[07:51:54] <RhinosF1>	 What timezone are you
[07:51:58] <BlankEclair>	 utc+10
[07:52:03] <RhinosF1>	 Ah
[07:52:18] <RhinosF1>	 We seriously need to create a clock app with volunteers timezones on
[07:52:40] <BlankEclair>	 yeah and how do we integrate that?
[07:52:50] <BlankEclair>	 oh wait
[07:53:05] <BlankEclair>	 i was thinking timezone info in discord/irc, but if you're talking about a separate webapp...
[07:53:18] <BlankEclair>	 alternatively, we could do that on the wiki lol
[07:53:26] <RhinosF1>	 I was thinking like time.releng.team
[07:53:56] <BlankEclair>	 holy shit that's neat
[07:54:28] <RhinosF1>	 It is
[07:54:50] <RhinosF1>	 We don't even need to host simple web stuff too
[07:55:01] <RhinosF1>	 Could just stick it in cloudflare directly at the edge
[07:57:07] <BlankEclair>	 imagine being these extensions right now: https://issue-tracker.miraheze.org/maniphest/query/MSDh7NfOtLB1/#R
[08:03:09] <BlankEclair>	 https://issue-tracker.miraheze.org/T11688; should i close?
[08:08:52] <RhinosF1>	 BlankEclair: yup
[08:11:47] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[08:13:45] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0005311071873 secs
[09:05:03] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sudoers]
[09:05:34] <BlankEclair>	 um!
[09:06:18] <MirahezeRelay>	 <ellie.rose._> that doesn't seem good
[09:21:53] <CosmicAlpha>	 its fine should resolve itself
[09:23:38] <BlankEclair>	 php could literally crash and someone'd say that
[09:30:52] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[09:32:26] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:34:23] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001528352499 secs
[09:36:36] <RhinosF1>	 BlankEclair: most of these things are just it being grumpy
[09:36:47] <RhinosF1>	 Sudoers is a weird one
[09:36:53] <BlankEclair>	 why's it happen?
[09:37:02] <RhinosF1>	 No idea
[09:37:17] <BlankEclair>	 :/
[09:41:51] <CosmicAlpha>	 BlankEclair: if PHP crashed I would have fixed lol, but that was just puppet being odd no real issue other than puppet.
[09:42:01] <BlankEclair>	 oh
[09:42:09] <BlankEclair>	 probably shouldn't have exaggerated that much, sorry ^^;
[09:43:01] <CosmicAlpha>	 lol no problem
[09:43:32] <RhinosF1>	 Php is supposed to fix itself
[09:43:41] <RhinosF1>	 Except on photge
[09:43:51] <RhinosF1>	 CosmicAlpha: can you make phorge php auto restart if it crashes
[09:44:45] <CosmicAlpha>	 I can in a few hours
[09:46:54] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:02:18] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 135.78 ms
[10:02:34] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2407:3641:2161:9774::1/cpweb
[10:02:37] <icinga-miraheze>	 PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089
[10:04:21] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 135.55 ms
[10:04:32] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.312 seconds response time. wikitide.net returns 2602:294:0:b13::110,38.46.223.205,38.46.223.206
[10:04:41] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[10:04:42] <icinga-miraheze>	 PROBLEM - cp41 Current Load on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[10:05:08] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/ssl/private],File[/usr/local/sbin/reload-vcl],File[/usr/lib/nagios/plugins/check_varnishbackends],File[/usr/lib/nagios/plugins/check_nginx_errorrate]
[10:06:26] <RhinosF1>	 icinga-miraheze: stop being sad we solved most of your errors
[10:06:31] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[10:06:36] <icinga-miraheze>	 RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 4%
[10:06:37] <icinga-miraheze>	 RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 0.62, 0.78, 0.87
[10:08:56] <BlankEclair>	 ...and just like that it's happy again
[10:26:10] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[10:28:08] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.002352356911 secs
[10:30:58] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[10:43:45] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[10:43:50] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#602: Delete zones/mirabeta.org - 13https://github.com/miraheze/dns/pull/602
[10:44:48] <RhinosF1>	 CosmicAlpha: you want to merge ^
[10:45:23] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#602: Delete zones/mirabeta.org - 13https://github.com/miraheze/dns/pull/602
[10:45:26] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/b154a40c4c72...74ee997ccbfd
[10:45:28] <botifico>	 [02dns] 07RhinosF1 0374ee997 - Delete zones/mirabeta.org (#602)
[10:46:01] <botifico>	 [02puppet] 07RhinosF1 opened pull request 03#3912: varnish/beta: remove - 13https://github.com/miraheze/puppet/pull/3912
[10:46:06] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3912: varnish/beta: remove - 13https://github.com/miraheze/puppet/pull/3912#issuecomment-2323281900
[10:46:27] <RhinosF1>	 CosmicAlpha: also ^ and https://github.com/miraheze/ssl/pull/793#pullrequestreview-2274308437
[10:47:10] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3912: varnish/beta: remove - 13https://github.com/miraheze/puppet/pull/3912
[10:47:12] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a7f8c9ddec96...03f65abeb01a
[10:47:13] <botifico>	 [02puppet] 07RhinosF1 0303f65ab - varnish/beta: remove (#3912)
[10:47:36] <botifico>	 [02ssl] 07Universal-Omega closed pull request 03#793: Switch Mirabeta to the Cloudflare origin certificate - 13https://github.com/miraheze/ssl/pull/793
[10:47:38] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a2379e30f553...8f0009651d28
[10:47:41] <botifico>	 [02ssl] 07redbluegreenhat 038f00096 - Switch Mirabeta to the Cloudflare origin certificate (#793)
[10:47:42] <botifico>	 [02ssl] 07Universal-Omega deleted branch 03mirabeta-cf-origin - 13https://github.com/miraheze/ssl
[10:47:45] <botifico>	 [02ssl] 07Universal-Omega deleted branch 03mirabeta-cf-origin
[10:48:08] <CosmicAlpha>	 RhinosF1: done 
[10:48:14] <RhinosF1>	 CosmicAlpha: ty
[10:48:25] <RhinosF1>	 the second batch can't be done for another 7 hours
[10:48:34] <CosmicAlpha>	 No problem.
[12:01:36] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw181 mw182
[12:01:43] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 162.68 ms
[12:01:47] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 135.56 ms
[12:02:34] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[12:02:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:02:48] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 14s
[12:02:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:03:06] <CosmicAlpha>	 Why did that randomly trigger?
[12:03:20] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[12:03:36] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[12:03:47] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.68 ms
[12:03:50] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 135.59 ms
[12:10:19] <icinga-miraheze>	 PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089
[12:11:26] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw151 mw152 mw161
[12:11:52] <Agent>	 Thank you ns2, very cool!
[12:12:18] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 1.314 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[12:13:28] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[12:13:31] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100%
[12:13:31] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 137.76 ms
[12:13:58] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[12:14:30] <RhinosF1>	 Agent: I'm pretty sure ns2 is just being grumpy to remind us it exists
[12:17:43] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 135.55 ms
[12:17:59] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[12:18:06] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3820 bytes in 1.266 second response time
[12:18:41] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 197.95 ms
[12:24:32] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds
[12:24:59] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/nginx/fastcgi_params]
[12:26:24] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 135.57 ms
[12:26:29] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw162 mw171 mw172 mw182
[12:26:36] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb, 2602:294:0:b23::112/cpweb
[12:26:36] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 6.175 second response time
[12:27:08] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 162.27 ms
[12:27:38] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 5 backends are down. mw152 mw162 mw171 mw172 mw181
[12:30:16] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[12:31:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 168.55 ms
[12:31:44] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[12:31:45] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[12:32:14] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3821 bytes in 1.466 second response time
[12:32:44] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 133.79 ms
[12:34:36] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb, 2400:d320:2161:9775::1/cpweb
[12:35:57] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 62%
[12:35:59] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[12:37:11] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 139.57 ms
[12:37:39] <icinga-miraheze>	 PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 53%
[12:37:57] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 54%
[12:39:08] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:39:12] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for mw161 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:39:15] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 134.00 ms
[12:39:23] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[12:39:25] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[12:39:36] <icinga-miraheze>	 RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 34%
[12:41:15] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for mw161 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8115
[12:42:12] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.06 ms
[12:42:15] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 0.779 second response time
[12:42:25] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[12:43:43] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 135.52 ms
[12:44:19] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[12:44:28] <icinga-miraheze>	 RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 1%
[12:44:33] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb
[12:46:31] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds
[12:47:52] <icinga-miraheze>	 PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 54%
[12:49:31] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 52%
[12:50:31] <icinga-miraheze>	 PROBLEM - cp51 PowerDNS Recursor on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:50:32] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for test151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:51:27] <icinga-miraheze>	 RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 2%
[12:51:41] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 170.19 ms
[12:51:48] <icinga-miraheze>	 RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 30%
[12:51:53] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 1.538 second response time
[12:52:09] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 133.70 ms
[12:52:29] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 1.101 second response time
[12:52:31] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for test151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8181
[12:52:33] <icinga-miraheze>	 RECOVERY - cp51 PowerDNS Recursor on cp51 is OK: DNS OK: 0.484 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167
[12:52:50] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[12:53:04] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[12:53:05] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[12:56:44] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 224.88 ms
[12:56:58] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 210.43 ms
[13:06:25] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[13:06:59] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 227.34 ms
[13:08:48] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 232.64 ms
[13:08:59] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 221.29 ms
[13:09:49] <MirahezeRelay>	 <bluemoon0332> [1/8] ```
[13:09:49] <MirahezeRelay>	 <bluemoon0332> [2/8] [alex@ldap171:~]$ openssl s_client -showcerts -servername ldap171.wikitide.net -connect ldap171.wikitide.net:636
[13:09:50] <MirahezeRelay>	 <bluemoon0332> [3/8] Certificate chain
[13:09:50] <MirahezeRelay>	 <bluemoon0332> [4/8]  0 s:CN = wikitide.net
[13:09:50] <MirahezeRelay>	 <bluemoon0332> [5/8]    i:C = US, O = Let's Encrypt, CN = E5
[13:09:51] <MirahezeRelay>	 <bluemoon0332> [6/8]    a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
[13:09:51] <MirahezeRelay>	 <bluemoon0332> [7/8]    v:NotBefore: Jun 18 13:51:38 2024 GMT; NotAfter: Sep 16 13:51:37 2024 GMT
[13:09:51] <MirahezeRelay>	 <bluemoon0332> [8/8] ```
[13:10:02] <MirahezeRelay>	 <bluemoon0332> CosmicAlpha: Looks like you forgot to restart slapd
[13:10:47] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 255.99 ms
[13:11:01] <MirahezeLSBot>	 !log [alex@ldap171] restart slapd
[13:11:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:11:33] <MirahezeRelay>	 <bluemoon0332> honestly this should be done automatically like with nginx
[13:13:26] <RhinosF1>	 Is there a conf test?
[13:13:34] <RhinosF1>	 Cause ldap rebooting and dying would be bad
[13:14:42] <BlankEclair>	 reboot it first before it reboots itself
[13:14:48] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 252.41 ms
[13:15:33] <RhinosF1>	 An ldap crash would break icinga, grafana at least
[13:16:47] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 162.41 ms
[13:16:58] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 155.30 ms
[13:18:37] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 181.87 ms
[13:18:50] <RhinosF1>	 Graylog as well
[13:18:51] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[13:18:54] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 172.74 ms
[13:20:00] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:21:57] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:22:07] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2407:3641:2161:9774::1/cpweb, 2400:d320:2161:9775::1/cpweb
[13:22:18] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 100%
[13:23:08] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 245.93 ms
[13:23:09] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[13:23:13] <icinga-miraheze>	 PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 50%
[13:23:19] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 213.29 ms
[13:23:21] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 250.42 ms
[13:24:16] <icinga-miraheze>	 PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:24:39] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:24:42] <icinga-miraheze>	 PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:12] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for matomo151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:12] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for mw181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:13] <icinga-miraheze>	 PROBLEM - cp51 Nginx Backend for mwtask171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:13] <icinga-miraheze>	 PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:13] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for swiftproxy171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:16] <icinga-miraheze>	 PROBLEM - ns2 Disk Space on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:18] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for mwtask171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:25:20] <icinga-miraheze>	 PROBLEM - cp41 Nginx Backend for phorge171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[13:26:01] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[13:26:13] <icinga-miraheze>	 RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0)
[13:26:30] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 135.56 ms
[13:26:38] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[13:26:39] <icinga-miraheze>	 RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 23%
[13:26:55] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[13:27:05] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 158.19 ms
[13:27:07] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 1.486 second response time
[13:27:08] <icinga-miraheze>	 RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 3%
[13:27:08] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[13:27:08] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for swiftproxy171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8207
[13:27:08] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205
[13:27:09] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for mw181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8119
[13:27:10] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for mwtask171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8161
[13:27:11] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for matomo151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8203
[13:27:13] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 0.795 second response time
[13:27:14] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 101.70 ms
[13:27:15] <icinga-miraheze>	 RECOVERY - ns2 Disk Space on ns2 is OK: DISK OK - free space: / 16544MiB (86% inode=94%);
[13:27:16] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for phorge171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8202
[13:27:18] <icinga-miraheze>	 RECOVERY - cp41 Nginx Backend for mwtask171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8161
[13:27:19] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 140.54 ms
[13:28:59] <RhinosF1>	 How can cp* be so grumpy
[13:29:19] <RhinosF1>	 The traffic on cp* is down like 50%
[13:30:51] <RhinosF1>	 CosmicAlpha: can you take a look at something? Why does cp41 get so much traffic?
[13:47:09] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:57:00] <MirahezeRelay>	 <bluemoon0332, replying to RhinosF1> I couldn't see anything about it
[13:58:39] <MirahezeRelay>	 <rhinosf1, replying to bluemoon0332> That's probably why it doesn't auto restart
[13:58:42] <MirahezeRelay>	 <bluemoon0332> We did experience slapd running an expired wikitide.net certificate a while back, I legit thought someone changed my LDAP password
[13:58:54] <MirahezeRelay>	 <bluemoon0332> because I couldn't login to Icinga or ldapwikiwiki
[13:58:54] <MirahezeRelay>	 <rhinosf1> Auto restarts should only be done where it's safe
[14:00:19] <MirahezeRelay>	 <bluemoon0332, replying to bluemoon0332> (https://meta.miraheze.org/wiki/Tech:Server_admin_log?diff=404375&oldid=403783)
[14:02:12] <MirahezeRelay>	 <bluemoon0332, replying to rhinosf1> if only the certificate changed it's safe to auto-restart
[14:02:24] <MirahezeRelay>	 <rhinosf1, replying to bluemoon0332> Yes probably
[14:02:36] <MirahezeRelay>	 <rhinosf1> As long as you validate the public & private cert
[14:14:07] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:18:55] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:20:53] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001495182514 secs
[14:42:53] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 135.54 ms
[14:44:56] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 135.57 ms
[14:49:56] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 162.09 ms
[14:50:14] <MirahezeRelay>	 <pixldev, replying to icinga-miraheze> What is WRONG with you
[14:50:32] <MirahezeRelay>	 <pixldev> You complain more then a small child fed green beans
[14:54:04] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 162.32 ms
[14:58:27] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 162.57 ms
[14:59:50] <icinga-miraheze>	 PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 135.54 ms
[15:02:46] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[15:02:54] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw151
[15:03:01] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 14s
[15:04:14] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/nagios/nrpe.cfg]
[15:04:32] <icinga-miraheze>	 PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089
[15:04:49] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 233.61 ms
[15:04:54] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[15:06:30] <icinga-miraheze>	 RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.332 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[15:09:12] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 5 backends are down. mw152 mw162 mw171 mw172 mw181
[15:09:14] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw161 mw162
[15:09:14] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[15:10:59] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 46.250.240.167/cpweb
[15:11:22] <MirahezeRelay>	 <pixldev> Does icinga normally complain this much
[15:12:29] <icinga-miraheze>	 RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 133.68 ms
[15:12:56] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[15:13:15] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[15:15:19] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 2.360 second response time
[15:16:05] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:17:29] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw151 mw171 mw181
[15:19:31] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503
[15:20:24] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 135.18 ms
[15:21:10] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 100%
[15:22:38] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 252.29 ms
[15:24:23] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds
[15:24:36] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 171.90 ms
[15:26:22] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3843 bytes in 1.252 second response time
[15:26:42] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 112.32 ms
[15:27:15] <MirahezeRelay>	 <pixldev> @cosmicalpha Icinga didn’t always yell this much did it
[15:27:35] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.76 ms
[15:27:40] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3821 bytes in 1.235 second response time
[15:27:45] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[15:27:45] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[15:31:52] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:06:14] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 226.91 ms
[16:08:12] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 174.58 ms
[16:13:15] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[16:19:31] <CosmicAlpha>	 RhinosF1: cp41 always gets a ton lets put everything left through our US cps they were always faster even from Asia then our Asian cps.
[16:22:56] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/74ee997ccbfd...c5c65213d667
[16:22:59] <botifico>	 [02dns] 07Universal-Omega 03c5c6521 - depool au and ja regions
[16:28:16] <MirahezeRelay>	 <pixldev> Uh
[16:28:22] <MirahezeRelay>	 <pixldev, replying to botifico> Is this a good idea?
[16:29:02] <MirahezeRelay>	 <pixldev> If US servers are too stressed, depool japan and Australia?
[16:29:27] <MirahezeRelay>	 <cosmicalpha> Yes, our on premise cp servers should be able to handle remaining traffic, even from asia where the other servers are, our us cps are faster. The issue is with Australia and Japan servers.
[16:29:28] <MirahezeRelay>	 <pixldev> Or wait oh is 41 not US
[16:29:40] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> Do we know why?
[16:29:53] <MirahezeRelay>	 <cosmicalpha> Networking that we can't seem to figure out the cause.
[16:30:14] <MirahezeRelay>	 <pixldev> Aren’t the only things using cp* the apex custom domains?
[16:30:29] <MirahezeRelay>	 <bluemoon0332> and matomo, grafana and icingaweb2
[16:30:32] <MirahezeRelay>	 <pixldev> And WikiTide.org ig
[16:30:37] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> Ah
[16:30:51] <MirahezeRelay>	 <pixldev> Matomo is called on each view right?
[16:30:58] <MirahezeRelay>	 <bluemoon0332> yes
[16:31:10] <MirahezeRelay>	 <bluemoon0332> unless the user has the noanalytics userright
[16:31:32] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> That’s why icinga was screaming I assume?
[16:31:37] <MirahezeRelay>	 <pixldev, replying to bluemoon0332> Who actually has that
[16:31:49] <MirahezeRelay>	 <bluemoon0332> at least 1 wiki does
[16:31:56] <MirahezeRelay>	 <bluemoon0332> I remember telling someone on Discord about it
[16:32:04] <MirahezeRelay>	 <cosmicalpha, replying to pixldev> Yes, plus others mentioned. That's why the us on-premise cps should be able to handle it alone.
[16:32:44] <MirahezeRelay>	 <cosmicalpha> They used to handle all our traffic even before we started going to CF and handled it well unless DDOSed then they didn't so much.
[16:32:47] <MirahezeRelay>	 <pixldev> Why do I get the feeling we’re gonna figure out the cause of the slowness the same day we decommission them
[16:33:17] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> When did the au and ja caches get ‘sick’ like this
[16:33:32] <MirahezeRelay>	 <cosmicalpha> Since the very beginning
[16:34:00] <MirahezeRelay>	 <pixldev> Really!
[16:34:12] <MirahezeRelay>	 <pixldev> But we didn’t alway have these slowdowns
[16:34:19] <MirahezeRelay>	 <pixldev> Or is that another issue
[16:34:45] <MirahezeRelay>	 <cosmicalpha> I think we are talking about 3 seperate issues here.
[16:35:00] <MirahezeRelay>	 <pixldev> Possibly
[16:35:05] <MirahezeRelay>	 <pixldev> Wait what’s the third
[16:35:25] <MirahezeRelay>	 <pixldev> [1/2] * au and ja being slow cause network 
[16:35:26] <MirahezeRelay>	 <pixldev> [2/2] * the random slow downs
[16:36:50] <MirahezeRelay>	 <cosmicalpha> From Asia and Oceania we always had it slower then Europe, North America, and South America (which all use the us cps for quite some time). Some slowness was caused by the to many FPM childs, then there was the consistent and large DDOSing issue which is the purpose for the switch to CF.
[16:38:31] <MirahezeRelay>	 <pixldev> You did a patch for the FPM children a few days ago right?
[16:38:37] <MirahezeRelay>	 <cosmicalpha> Yes
[16:38:58] <MirahezeRelay>	 <pixldev> Hm, I thought i saw a 503 report today/yesterday
[16:39:01] <MirahezeRelay>	 <pixldev> Lemme see
[16:39:08] <MirahezeRelay>	 <pixldev> Or are those also separate
[16:39:45] <MirahezeRelay>	 <pixldev> One 11h ago… on cp41
[16:39:48] <MirahezeRelay>	 <pixldev> Hmmmm
[16:40:38] <MirahezeRelay>	 <cosmicalpha> the FPM childern issue is not related to Varnish at all, that was MediaWiki. It also doesn't mean all errors go away just improved.
[16:41:35] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> Gotcha
[16:41:56] <MirahezeRelay>	 <pixldev> I’m assuming those are going to be looked at more closely once we’re done with migration
[16:47:48] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 226.19 ms
[16:55:42] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 193.68 ms
[17:32:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:35:33] <MirahezeRelay>	 <pixldev> Oh dear
[18:13:58] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (START)
[18:15:47] <MirahezeLSBot>	 !log [universalomega@bots171] Restart logbot
[18:15:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:18:58] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=holidayswiki --procs=12 --memory-limit=max (END - exit=0)
[18:19:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:19:30] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=endiversumwiki --procs=12 --memory-limit=max (START)
[18:19:31] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=endiversumwiki --procs=12 --memory-limit=max (END - exit=0)
[18:19:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:19:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:20:16] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=endiversumwiki --procs=12 --memory-limit=max (START)
[18:20:17] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=endiversumwiki --procs=12 --memory-limit=max (END - exit=0)
[18:20:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:20:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:20:47] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=bluearchivewiki --procs=12 --memory-limit=max (START)
[18:20:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:21:41] <MirahezeLSBot>	 !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=bluearchivewiki --procs=12 --memory-limit=max (END - exit=0)
[18:21:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:22:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[18:25:23] <MirahezeLSBot>	 !log [universalomega@bots171] Restart ircrcbot
[18:25:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:33:23] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 204.94 ms
[19:35:23] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 198.22 ms
[19:39:28] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 226.05 ms
[19:41:01] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 222.72 ms
[19:43:28] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 196.91 ms
[19:44:58] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 193.22 ms
[19:49:00] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 223.27 ms
[19:50:58] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 177.70 ms
[19:55:00] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 229.27 ms
[19:57:31] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[20:00:56] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 193.26 ms
[20:02:39] <RhinosF1>	 MacFan4000: https://meta.miraheze.org/wiki/Special:RequestSSLQueue/179 is setup in cf
[20:04:57] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 232.72 ms
[20:14:51] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 236.84 ms
[20:16:27] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[20:16:40] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 218.58 ms
[20:16:50] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 240.87 ms
[20:20:41] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 228.09 ms
[20:22:41] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 233.75 ms
[20:26:38] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[20:26:41] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 174.36 ms
[20:30:46] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 215.63 ms
[20:32:10] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=endiversumwiki (END - exit=0)
[20:32:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:32:46] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 191.24 ms
[20:32:54] <MacFan4000>	 RhinosF1: after clearing cache we still redirect to the landing page
[20:33:20] <MacFan4000>	 this is the first post-CF domain that wasn't already in the SSL repo
[20:36:28] <MacFan4000>	 (cc CosmicAlpha and @bluemoon0332)
[20:36:52] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 208.22 ms
[20:37:15] <MacFan4000>	 seems like we still would at least need an entry in certs.yaml?
[20:38:30] <MirahezeRelay>	 <pixldev> @bluemoon0332
[20:40:11] <MirahezeRelay>	 <cosmicalpha, replying to MacFan4000> Yes so the nginx server is loaded.
[20:40:51] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 184.12 ms
[20:41:18] <MirahezeRelay>	 <rhinosf1> @cosmicalpha I think we should change the default to mediawiki
[20:41:39] <MirahezeRelay>	 <rhinosf1> for new sites
[20:41:43] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8f0009651d28...ac35bac781d8
[20:41:45] <MirahezeRelay>	 <pixldev> hm?
[20:41:46] <botifico>	 [02ssl] 07MacFan4000 03ac35bac - +wiki.endiversum.com
[20:41:47] <MirahezeRelay>	 <rhinosf1> cause then that removes certs.yaml
[20:42:23] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> I don't understand?
[20:42:29] <RhinosF1>	 MirahezeRelay: Ca should be cloudflare & sslname the origin cert
[20:42:55] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> Have NGINX serve the origin cert and mediawiki by default not miraheze.org landing page
[20:43:02] <MirahezeRelay>	 <rhinosf1> for unconfigured domains
[20:44:48] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 222.34 ms
[20:44:57] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 219.71 ms
[20:45:10] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[20:45:29] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> I also like having certs.yaml for tracking which sites on a repo, but we could simplify it and make it auto update with a bot maybe. We also have redirects.yaml anyway. We can visit that later, it is almost 5AM for me so I can look and answer better later.
[20:46:38] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> redirects.yaml is staying I think for now. We could have an automatic list of sites from the cf api or something.
[20:46:47] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 226.07 ms
[20:46:57] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 176.09 ms
[20:46:59] <MirahezeRelay>	 <rhinosf1> TNT is working on something to allow cloudflare to be managed on-wiki
[20:47:14] <MirahezeRelay>	 <rhinosf1> so people can turn on/off some of the speed features
[20:47:20] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> certs.yaml is also the only thing that makes @Wiki-Bot work for us I think, but I could be wrong.
[20:47:21] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ac35bac781d8...2a23f8fde869
[20:47:23] <botifico>	 [02ssl] 07MacFan4000 032a23f8f - Update certs.yaml
[20:48:05] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> @Wiki-Bot should probably use the sites API
[20:48:41] <MirahezeRelay>	 <cosmicalpha> @markusrost might be able to shed some light on this potential issue for us?
[20:50:16] <MirahezeRelay>	 <pixldev> what issue
[20:50:29] <MirahezeRelay>	 <cosmicalpha> Although regardless of what we do, I do want some sort of public list of sites @rhinosf1
[20:51:02] <icinga-miraheze>	 PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 210.53 ms
[20:51:23] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> The Miraheze API has one
[20:51:39] <MirahezeRelay>	 <rhinosf1> We could expose the CF API's version too
[20:51:43] <MirahezeRelay>	 <pixldev> we have an api?
[20:51:55] <MirahezeRelay>	 <rhinosf1> although we'd probably have to cache that because it's paged
[20:52:02] <MirahezeRelay>	 <rhinosf1> max 50 results i think
[20:52:23] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> I mean on GitHub would be nice, I was thinking just a bit to create a yaml list or json list, or maybe exactly in the format services used to be done.
[20:52:36] <MirahezeRelay>	 <rhinosf1, replying to pixldev> https://meta.miraheze.org/w/api.php?action=wikidiscover
[20:52:47] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 175.51 ms
[20:52:54] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> I mean ye it's easy to enough to sync it
[20:53:02] <icinga-miraheze>	 RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 186.85 ms
[20:53:31] <MirahezeRelay>	 <rhinosf1> we should probably see how much fun is had with the API on https://issue-tracker.miraheze.org/T12519
[20:53:35] <MirahezeRelay>	 <rhinosf1> and then go from there
[20:53:49] <MirahezeRelay>	 <rhinosf1> I want to link cloudflare to monitoring too
[20:54:14] <MirahezeRelay>	 <rhinosf1> it can send webhooks like grafana does to icinga-miraheze
[20:54:23] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> I am also working on a CF API version of RequestSSL but maybe I should wait for that one and go from there?
[20:54:43] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> Talk to @theresnotime and share knowledge
[20:54:54] <MirahezeRelay>	 <rhinosf1> one thing made harder by the huge ocean
[20:55:07] <MirahezeRelay>	 <rhinosf1> the CF API is a nightmare so more people learning it is good tbh
[20:55:13] <icinga-miraheze>	 PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[20:56:35] <icinga-miraheze>	 PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[20:56:43] <MirahezeRelay>	 <rhinosf1> if volunteers stay active, i might try some voice technical drop ins
[20:56:49] <MirahezeRelay>	 <rhinosf1> like i did with fossbots
[20:56:50] <icinga-miraheze>	 PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 213.10 ms
[20:56:51] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[20:56:53] <MirahezeRelay>	 <rhinosf1> but from october
[20:57:03] <MirahezeRelay>	 <rhinosf1> @cosmicalpha
[20:57:36] <icinga-miraheze>	 PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[20:58:48] <icinga-miraheze>	 RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 195.32 ms
[20:59:36] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2a23f8fde869...69ee1195d9d4
[20:59:38] <botifico>	 [02ssl] 07Universal-Omega 0369ee119 - Fix
[20:59:39] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:02:52] <icinga-miraheze>	 PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:03:38] <MirahezeRelay>	 <cosmicalpha> MacFan4000 https://github.com/miraheze/ssl/compare/2a23f8fde869...69ee1195d9d4
[21:04:00] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[21:04:05] <RhinosF1>	 thanks CosmicAlpha 
[21:04:17] <MirahezeRelay>	 <pixldev, replying to rhinosf1> hm?
[21:04:48] <MirahezeRelay>	 <markusrost, replying to rhinosf1> Last I check the sites API had no pagination, so I can get like 100 sites top and can't even filter by custom domain
[21:04:51] <icinga-miraheze>	 RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures
[21:05:16] <icinga-miraheze>	 RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[21:05:22] <MirahezeRelay>	 <rhinosf1, replying to pixldev> So someone sits in a voice channel for an hour every so often to help with tech stuff
[21:05:36] <CosmicAlpha>	 No problem RhinosF1
[21:05:36] <icinga-miraheze>	 RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[21:05:39] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures
[21:05:44] <MirahezeRelay>	 <pixldev> With other tech volunteers or users
[21:05:52] <MirahezeRelay>	 <rhinosf1, replying to markusrost> You can get all sites cause wikistats does it
[21:06:01] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:06:13] <MirahezeRelay>	 <rhinosf1, replying to pixldev> Aimed at volunteers
[21:06:35] <icinga-miraheze>	 RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:06:41] <MirahezeRelay>	 <pixldev> like working on stuff, talk to each other
[21:06:51] <MirahezeRelay>	 <pixldev> the rubber duck but hoomab
[21:06:57] <MirahezeRelay>	 <rhinosf1> Ye
[21:07:18] <MirahezeRelay>	 <markusrost, replying to cosmicalpha> Wiki-Bot is using certs.yaml for revert buttons in the recent changes feed, as that project won't otherwise know which custom domains are miraheze
[21:07:28] <MirahezeRelay>	 <pixldev> sounds fun >:)
[21:07:45] <CosmicAlpha>	 !log run puppet and reload nginx on mw* and cp*
[21:07:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:08:10] <MirahezeRelay>	 <pixldev, replying to markusrost> why does it need to specifically know a miraheze wiki apart
[21:08:13] <MirahezeRelay>	 <cosmicalpha, replying to markusrost> Ah that is what I was remembering thanks
[21:09:13] <MirahezeRelay>	 <rhinosf1, replying to pixldev> Hopefully
[21:11:04] <MirahezeRelay>	 <markusrost, replying to pixldev> The buttons allow users to revert edits using OAuth, which is a global consumer on Miraheze
[21:11:49] <MirahezeRelay>	 <markusrost, replying to rhinosf1> Where do I get that? Because all I can find is action `action=wikidiscover` which has no pagination
[21:13:39] <MirahezeRelay>	 <originalauthority> Iirc sites is a wikimedia only thing
[21:14:08] <MirahezeRelay>	 <rhinosf1, replying to markusrost> It must be showing all then
[21:14:11] <MirahezeRelay>	 <originalauthority> Built with extension SiteMatrix if im remembering correctly
[21:14:21] <MirahezeRelay>	 <rhinosf1> Cause wikistats uses the wikidiscover api
[21:14:38] <MirahezeRelay>	 <markusrost, replying to rhinosf1> [1/2] It has a max limit of 5000 - but no offset for pagination
[21:14:38] <MirahezeRelay>	 <markusrost, replying to rhinosf1> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1279912322936082582/image.png?ex=66d62abd&is=66d4d93d&hm=9367742f3c1db107ae6d09f3b5f55e2d3c4f5ec83fe1168ecf646f1c181882d4&
[21:14:56] <MirahezeRelay>	 <cosmicalpha> I think fhat API needs improvement
[21:15:00] <MirahezeRelay>	 <cosmicalpha> *that
[21:16:20] <MirahezeRelay>	 <cosmicalpha> 5000 is a MediaWiki limit for single results but should allow pagination
[21:18:40] <MirahezeRelay>	 <markusrost, replying to cosmicalpha> well there is no continue in the response, just 5000 wikis listed
[21:19:31] <MirahezeRelay>	 <markusrost> I would love to have a better way for getting custom domain miraheze wikis instead of using certs.yaml
[21:19:46] <MirahezeRelay>	 <cosmicalpha> Yeah we need to add continue support
[21:20:06] <MirahezeRelay>	 <rhinosf1, replying to markusrost> Oh
[21:20:18] <MirahezeRelay>	 <rhinosf1> I guess I never noticed that
[21:20:25] <MirahezeRelay>	 <rhinosf1> That's a big
[21:23:07] <MirahezeRelay>	 <originalauthority> Its not a bug
[21:23:13] <MirahezeRelay>	 <originalauthority> Its just a not implemented feature
[21:23:44] <MirahezeRelay>	 <markusrost> considering normally big API limit (without higher limits) is just 500 and that API uses 5000 for everyone, I think it was just set to 5000 thinking no one would ever have more than 5000 wikis anyway
[21:24:44] <icinga-miraheze>	 RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[21:39:58] <MirahezeRelay>	 <rhinosf1, replying to originalauthority> It's a bug, it breaks functionality
[21:40:56] <MirahezeRelay>	 <originalauthority> No it isnt a bug. A bug is fundamentally when a feature doesn't work as expected. It was never expected that there would be more than 5000 results. So that feature doesn't exist.
[21:41:41] <MirahezeRelay>	 <bluemoon0332> MacFan4000: an entry in certs.yaml is needed so that a server is added to nginx
[21:41:51] <MirahezeRelay>	 <rhinosf1> That would 100% should have been expected
[21:41:53] <MirahezeRelay>	 <bluemoon0332> otherwise it will default to the landing page server
[21:42:05] <MirahezeRelay>	 <rhinosf1, replying to bluemoon0332> Can we just make it default to mediawiki instead of landing
[21:45:25] <MirahezeRelay>	 <bluemoon0332> [1/2] https://nginx.org/en/docs/http/request_processing.html
[21:45:25] <MirahezeRelay>	 <bluemoon0332> [2/2] if you set default_server on the *.miraheze.org server's listen directive, probably
[21:48:00] <MirahezeRelay>	 <rhinosf1> I'm off to sleep now anyway
[21:48:07] <MirahezeRelay>	 <rhinosf1> But I think we should consider it
[21:54:39] <MirahezeRelay>	 <bluemoon0332> it probably will not interact well with the backlog directive we have set if we did the change right now
[22:17:25] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:24:28] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[23:12:25] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:21:54] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.