[05:46:12] <jelto>	 GitLab maintenance starting soon 06:00–08:00 UTC, see T400252 for more details. I'll let you know when the maintenance is finished.
[05:46:13] <stashbot>	 T400252: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252
[07:50:58] <godog>	 full http 5xx stream is back in logstash cfr https://phabricator.wikimedia.org/T371366#11045857
[07:51:17] <elukey>	 \o/
[08:07:14] <fabfur>	 jelto: don't know if something known already but creating a new MR on gitlab results in a 404
[08:07:34] <jelto>	 Are you logged in? you have to login again
[08:07:56] <fabfur>	 ah damn, sorry
[08:08:53] <jelto>	 yeah the user experience is quite bad on that side, I can open a task and check if there is some workaround for that issue. But if you are logged in you should be able to create a MR again. The maintenance logged out all users
[08:09:44] <fabfur>	 it works flawlessly once I'm logged in thanks 
[08:09:56] <jelto>	 great, thanks
[09:13:21] <_joe_>	 jelto: i'm sure it's a feature you can only get with the enterprise version
[09:13:58] <_joe_>	 non confusing logout workflow: enterprise ultimate only
[09:14:07] <jelto>	 probably yes :D
[09:17:35] <_joe_>	 lol I made a joke, but apparently there is an "ultimate" version of gitlab 🙃
[09:17:56] <_joe_>	 and how can you miss, that's the 'with AI" edition
[12:28:20] <Krinkle>	 Looks like Debian has php-tideways but not yet php-xhprof https://packages.debian.org/stable/php-tideways
[12:28:39] <Krinkle>	 Its gone in testing. https://packages.debian.org/testing/php-tideways
[12:29:10] <Krinkle>	 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1027123
[12:29:25] <Krinkle>	 Looks like Mark H added it to the wishlist 3y ago
[13:05:05] <dcaro>	 topranks: XioNoX  I'm having issues connecting to gitlab from home using ip4, it gets to equinix but from there gets stuck
[13:05:08] <dcaro>	 https://www.irccloud.com/pastebin/NOViWZp3/
[13:05:17] <dcaro>	 ip6 works ok
[13:05:19] <dcaro>	 any ideas?
[13:06:02] <dcaro>	 it started happening today (after the planned outage/move of gitlab)
[13:06:16] <dcaro>	 it worked ~20min ago though
[13:06:59] <dcaro>	 oh, and it works again :/
[13:07:15] <dcaro>	 https://www.irccloud.com/pastebin/yZYe0cPE/
[13:08:10] <dcaro>	 I'll ask again if it starts failing anew, it's the second time today that it goes away for a few minutes
[13:17:59] <effie>	 dcaro: out of curiosity, is it expected to have cixp-salt.cern.ch     in your path ?
[13:18:46] <dcaro>	 I think so yes, cern is a big hub for the area
[13:19:18] <cdanis>	 dcaro: historically we've had recurring issues with services that are only fronted in eqiad/codfw being accessed remotely
[13:19:39] <taavi>	 effie: cixp apparently stands for cern ixp, https://cixp.net/
[13:19:50] <effie>	 yes, I understand 
[13:20:04] <cdanis>	 one easy option to work around is `tunnelencabulator -s`
[13:20:20] <dcaro>	 yep https://cixp.web.cern.ch/index.php/node/155
[13:20:31] <effie>	 I was not aware that it is an exchange for public consumption :p
[13:20:48] <effie>	 interesting 
[13:20:48] <dcaro>	 tunnelencabulator 👀
[13:22:19] <dcaro>	 "it may also be employed in conjunction with a loopback interface reciprocation dingle arm" xd
[13:24:47] <dcaro>	 awesome tool
[13:24:51] <dcaro>	 !
[13:31:20] <dcaro>	 hahahahah, that made my afternoon xd
[13:31:47] <jelto>	 thanks for reporting that dcaro. How long is the issue roughly? more like a few seconds or maybe 5 minutes? It could also be some of the nftables rate limiting we have in place.
[13:31:47] <jelto>	 It looks like it's triggered more than on the codfw hosts before the switchover surprisingly 
[13:32:54] <dcaro>	 I'd say more like 10 min
[13:32:56] <topranks>	 dcaro: can you run the mtr again?  but with these flags: mtr -T --port 443  -z -b -w -c 10 gitlab.wikimedia.org
[13:33:19] <dcaro>	 it's working now though
[13:33:23] <topranks>	 my guess it it may be the return path from us, if you can share your IP with me we can check in the other direction 
[13:33:40] <dcaro>	 https://www.irccloud.com/pastebin/4GpXUPYM/
[13:33:46] <dcaro>	 ip6 ^
[13:33:56] <topranks>	 well I guess the answer is "internet ¯\_(ツ)_/¯"
[13:34:15] <dcaro>	 xd
[13:34:25] <dcaro>	 ip4 https://www.irccloud.com/pastebin/Ql31pUXm/
[13:35:31] <topranks>	 we won't find much if it's sopped happening unfortunately, sorry I didn't see your ping till a few mins ago 
[13:35:44] <dcaro>	 okok, np, hopefully it will not happen again :)
[13:39:26] <XioNoX>	 jelto: is there doc on how to check what IPs are currently being rate-limited/blocked?
[13:41:14] <jelto>	 I think there are no docs at the moment for that, but either sudo nft list table inet filter or checking the kernel logs on the host
[13:41:14] <jelto>	 We can also try disabling the rate-limiting
[13:48:44] <dcaro>	 oh, happening again
[13:48:46] <dcaro>	 https://www.irccloud.com/pastebin/8nIrbn8q/
[13:49:21] <dcaro>	 topranks, jelto ^
[13:49:36] <topranks>	 dcaro: ok 
[13:49:51] <topranks>	 was talking with XioNoX he is probably right it may be the rate limiting rather than a routing thing 
[13:50:13] <topranks>	 dcaro: can you DM me your source IP from www.whatismyip.com ?
[13:50:23] <jelto>	 I have a french and or swiss IP in the denylist 213.55.xx, is that you?
[13:50:58] <dcaro>	 yep, that's me
[13:51:28] <dcaro>	 am I doing that big amount of queries?
[13:51:31] <jelto>	 ok then I'll disable the rate-limiting for now and we can troubleshoot why this is behaving differently in eqiad compared to the codfw host
[13:51:38] <jelto>	 probably not no
[13:51:38] <dcaro>	 (I'm building things, and developing, but not as much I think)
[13:53:40] <dcaro>	 let me know if you want me to do any tests
[13:54:47] <effie>	 dcaro: it is an on going contest actually, engineers caught by the gitlab ratelimits, enter a ruffle to win gitlab swag 
[13:55:00] <effie>	 go to the gitlab shop and choose your swag!
[13:55:25] <topranks>	 jelto: did I run the wrong command, I didn't see David's IP in the list from "sudo nft list table inet filter"
[13:55:37] <topranks>	 but I could also have just been too slow as I think you changed things 
[13:56:01] <jelto>	 the list has a "timeout" of 5 minutes and the IP is removed after that time
[13:56:51] <topranks>	 ok cool thanks, just checking that is the right command for my own reference 
[13:57:55] <dcaro>	 effie: xd I want a hoodie!
[13:58:41] <effie>	 ah, I just found out that the budget is enough only for stickers, but you are more than welcome to buy a sticker and attach it to a hoodie!
[13:59:51] <dcaro>	 hahaha
[14:09:00] <cdanis>	 dcaro: thanks <3 it's an official part of `wmf-sre-laptop` :D
[14:10:14] <cdanis>	 jelto: how difficult would it be to add the Report-To and NEL response headers to gitlab's responses?
[14:14:38] <jelto>	 dcaro: rate-limiting is disabled, so the issue should not happen again. We will troubleshoot this 
[14:14:50] <dcaro>	 jelto: thanks!
[14:15:50] <jelto>	 cdanis: currently all of the rate limiting is happening in firewall, I guess that's quite tricky to send any response headers on that layer. But we could try adding this headers to the standalone nginx
[14:16:02] <cdanis>	 jelto: oh, you'd want to add it to all responses yeah
[14:16:42] <jelto>	 that could be possible, although gitlab manages quite a lot of the nginx config. But it's possible to inject custom settings
[14:17:05] <cdanis>	 ack
[14:17:26] <cdanis>	 https://phabricator.wikimedia.org/T303725
[14:19:17] <jelto>	 thanks for the task, I added our tag and will add a subtask for gerrit/gitlab
[14:20:25] <cdanis>	 thanks <3 tbh I've put off doing it for too long because it seemed to hard to put something in hiera that could also be dumped into VCL, and meanwhile, we haven't changed our NEL configuration at all
[14:20:40] <cdanis>	 so for gitlab/gerrit feel free to just copy and paste
[14:35:09] <cdanis>	 unless anyone objects, in about 10 minutes I'm going to temp disable puppet on all cp hosts to deploy and test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1172402
[15:24:50] <cdanis>	 all done
[15:28:10] <sukhe>	 <3