[08:07:04] <dcaro>	 ok, interesting, that mean that we can drop the 2fa requirement from toolforge auth? (it was a blocking issue on adopting idp directly)
[08:16:05] <blancadesal>	 morning :)
[08:16:32] <blancadesal>	 fyi I deployed ingress-nginx 1.11.2 to toolsbeta this morning
[08:16:51] <blancadesal>	 doesn't seem to be causing any issues so far
[08:23:36] <dcaro>	 nice
[08:51:54] <blancadesal>	 dcaro: do you think it's safe to deploy in tools too?
[08:52:20] <dcaro>	 blancadesal: what have you tested so far in toolsbeta?
[08:52:50] <blancadesal>	 checking pod health/logs, running the func tests
[08:57:10] <dcaro>	 you can try setting up one of https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes/Reverse_proxy , we don't officially support it, but it's used by some tools
[09:10:39] <dcaro>	 tools ingresses look ok yep
[09:10:40] <dcaro>	 https://api-docs.beta.toolforge.org/docs
[09:13:22] <blancadesal>	 nice
[09:13:34] <blancadesal>	 i'll try the reverse proxy thing
[09:16:29] <blancadesal>	 are you doing something with the test tool atm?
[09:25:01] <dcaro>	 nope
[09:25:13] <dcaro>	 I might have a shell open
[11:54:39] <blancadesal>	 dcaro: reverse proxy looks ok
[11:54:42] <blancadesal>	 https://test.beta.toolforge.org/external-cdn/ajax/libs/axios/1.7.4/axios.js
[11:55:19] <blancadesal>	 proxying https://tools-static.wmflabs.org/cdnjs/ajax/libs/axios/1.7.4/axios.js
[12:02:37] <blancadesal>	 dacro: ready for k8s upgrade?
[12:02:41] <dcaro>	 nice
[12:02:52] <dcaro>	 yep, do you want to do it on irc or meet?
[12:03:38] <blancadesal>	 what do you prefer?
[12:04:46] <dcaro>	 I'm ok with any xd, you lead this one ;)
[12:05:09] <blancadesal>	 maybe start on meet, then continue on irc once it's just going through each node
[12:05:33] <blancadesal>	 meet.google.com/siv-bfoo-xdk
[13:24:57] <Guest7691>	 topranks: (this is Andrew, irc is broken) I think we should bump our network checkin until Arturo is back. Can you reschedule it for a future week that suits you?
[13:50:28] <topranks>	 Guest7691 has a nice ring to it :P 
[13:50:38] <topranks>	 but yes no probs - he is back from next week is he?
[13:50:52] <Guest7691>	 yep, he is.
[13:51:00] <Guest7691>	 I just moved the meeting but you're welcome to move it more.
[13:51:32] <topranks>	 cool, the new time works for me 
[13:51:36] <Guest7691>	 great!
[13:52:03] <Guest7691>	 I feel silly but when I /nick andrewbogott it tells me 'Nick/channel is temporarily unavailable'
[13:52:12] <Guest7691>	 possibly I have forgotten how to operate my chatmachine
[14:00:09] <topranks>	 That happens to me sometimes too
[14:00:42] <topranks>	 I think due to some stale/stuck session but not sure how to fix other than wait and retry
[14:09:42] * Guest7691 is good at waiting
[14:10:42] <Guest7691>	 but not so good at apache config it seems
[15:04:58] <blancadesal>	 dcaro: I think I'm feeling brave enough to try deploying the new ingress-nginx to tools, wdyt?
[15:07:31] <dcaro>	 blancadesal: sure, though maybe wait for tomorrow morning
[15:07:45] <dcaro>	 (so the one being paged if it breaks is me xd)
[15:07:57] <blancadesal>	 dcaro: ok!
[15:29:35] <dcaro>	 hmm... it seems tools prometheus is having trouble authenticating to openstack
[15:30:26] <dcaro>	 that happened right before prometheus dying
[15:30:42] <dcaro>	 Aug 28 15:10:28 tools-prometheus-6 prometheus@tools[76902]: ts=2024-08-28T15:08:04.173Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=openstack config=harbor msg="Unable to refresh target groups" err="could not authenticate to OpenStack: Post \"https://openstack.eqiad1.wikimediacloud.org:25000/v3/auth/tokens
[15:30:42] <dcaro>	 \": EOF"
[15:31:22] <dcaro>	 a couple minutes before it failed also with network error when querying k8s
[15:31:29] <dcaro>	 https://www.irccloud.com/pastebin/Pa7ZyGKa/
[15:32:18] <dcaro>	 :/
[15:49:29] <dcaro>	 I'll reboot the node and leave it not fetching the ingress stats for the night, retry the configs tomorrow
[15:54:30] * dcaro off
[15:54:34] <dcaro>	 cya tomorrow!
[15:54:35] <dcaro>	 \away
[15:55:02] <blancadesal>	 andrewbogott: looks like I was a bit too ambitious about diving into striker today! hopefully tomorrow will be the day 😅
[15:55:21] <andrewbogott>	 ok! no worries
[20:50:09] <RhinosF1>	 Rook: is the post to phab for PRs running from github actions by any chance?
[20:51:01] <Rook>	 Yes
[20:52:07] <RhinosF1>	 Rook: you want https://phabricator.wikimedia.org/T362401 then I think
[20:52:38] <RhinosF1>	 It seems to be a very poorly thought out limit that basically means no using phab from cloud providers
[20:52:56] <Rook>	 You're right, I think I do want that ticket. Thank you!
[20:53:22] <RhinosF1>	 Rook: your welcome, it seems to have caught a lot out