[06:16:10] <brouberol|ooo>	 > herron: first prod host on kafka 3.7 \o/
[06:16:10] <brouberol|ooo>	 OH HELL YAASS, congrats and thanks herron!
[07:29:59] <Nikerabbit>	 Phabricator is down (unable to connect to mysql or too many requests)
[07:31:16] <XioNoX>	 Nikerabbit: works for me
[07:31:47] <Nikerabbit>	 not for me, and I got reports from two other team members too
[07:32:07] <moritzm>	 works for me as well
[07:34:13] <Nikerabbit>	 so what's the next step to restore access for us?
[07:35:28] <Nikerabbit>	 Request served via cp3069 cp3069, Varnish XID 128512171 Upstream caches: cp3069 int Error: 403, Too many requests. Please respect our robot policy https://w.wiki/4wJS. (2b107d9) at Tue, 21 Apr 2026 07:34:23 GMT
[07:36:01] <XioNoX>	 thanks, that's useful
[07:36:37] <XioNoX>	 Nikerabbit: which browser are you using?
[07:36:57] <Nikerabbit>	 Google Chrome 147.0.7727.55
[07:37:35] <marostegui>	 Nikerabbit: works for me at the moment too
[07:38:16] <Nikerabbit>	 same error for me still
[07:39:16] <XioNoX>	 Nikerabbit: give it a few min and it should be good. You were hitting a too-strict rule we had to set after an aggressive scraping
[07:40:10] <Nikerabbit>	 thanks!
[07:44:49] <moritzm>	 Nikerabbit: you were probably logged into Phab, right? scrapers won't use an account, so it might also a path to differentiate the rules based on that
[07:45:12] <Nikerabbit>	 yes, I should always be logged in
[07:45:35] <moritzm>	 ack, thx
[07:45:37] <Nikerabbit>	 and can confirm was able to load Phab just now
[08:08:36] <Emperor>	 I'm getting database errors from Phab now
[08:09:22] <Emperor>	 [fixed after a few retries]
[08:10:03] <marostegui>	 Yesterday I switched the proxy in front of phab db cause it was having lots of haproxy[1872421]: Connect() failed for backend mariadb: no free ports.
[08:10:12] <marostegui>	 I am seeing now those on the proxy that I promoted
[08:10:31] <marostegui>	 is there higher unusual traffic to phab?
[08:10:34] <marostegui>	 jelto: ^
[08:11:02] <jelto>	 yes phabricator is getting scraped
[08:11:08] <marostegui>	 :_(
[09:35:04] <claime>	 fabfur (or anyone from traffic), can I get a review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271804 pretty please ? I'd like to get it out before going OOO so that the rest of the team can move traffic when everything is in place
[09:35:24] <fabfur>	 👀
[09:36:11] <claime>	 tyvm <3
[09:39:44] <fabfur>	 I'd say ok for the regexes (not very expert with the overall work though) , do you want to disable puppet on A:cp and enable on a single host for testing? 
[09:39:58] <claime>	 Yep, that's what I was going to do :)
[09:40:19] <claime>	 I have a list of curls to test these routes on :3128 on cp nodes
[09:40:24] <claime>	 That's the right way, right?
[09:40:52] <fabfur>	 I usually test that way
[09:41:04] <claime>	 Cool my snippets are good then :P
[09:41:30] <claime>	 It should be a no-op fwiw
[10:01:21] <claime>	 fabfur: ok for me to do that now?
[10:06:58] <fabfur>	 ack!
[10:07:08] <claime>	 Cool
[11:01:27] <claime>	 fabfur: follow up because lua patterns are trash https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275852
[11:02:45] <fabfur>	 ah sigh
[11:02:51] <fabfur>	 damn lua regexes
[11:04:54] <claime>	 moritzm: merging your redis::slave change is ok?
[11:05:24] <moritzm>	 claime: please do!
[11:06:17] <claime>	 {{done}}
[11:07:34] <moritzm>	 cheers