[08:26:13] <atsukoito>	 hi, do you know if we have gitlab.w.o under heavy scraping? my unauthenticated requests are getting trottled, and the API calls we are using from deploy1003.e.w are either frequently stalled or 500s?
[08:29:44] <jelto>	 yes GitLab is getting scraped at the moment. Is it possible to login and do your requests again?
[08:43:35] <atsukoito>	 jelto: thanks.. i'm looking into how to add a token to the application so we will do authenticated API requests
[08:54:36] <atsukoito>	 jelto: unfortunately, api returns 500 or stalls even with authenticated requests.. do we have authenticated only endpoint maybe?
[08:55:58] <jelto>	 we need some more minutes to enable additional rate limiting
[08:56:28] <jelto>	 hopefully this will reduce 500s for all other requests
[09:20:56] <Lucas_WMDE>	 atsukoito / jelto: https://phabricator.wikimedia.org/T428396 seems to be due to the same gitlab throttling btw :/
[09:21:38] <jelto>	 I'll take a look in a sec, currently we are trying to get gitlab online again
[09:21:47] <Lucas_WMDE>	 thanks <3
[09:21:50] <claime>	 jelto: can we help?
[09:22:55] <jelto>	 I think gitlab is back, cc arnaudb
[09:23:03] <arnaudb>	 it is!
[09:23:52] <jelto>	 I'll disable the useless per-ip rate limiting and leave the bug chrome + macos hammer in-place
[09:27:01] <jelto>	 claime: thank you! But i'm not sure there is much we can do beside moving gitlab behind the cdn (which is already work in progress) and adding some blocking in GitLabs nginx
[09:27:54] <jelto>	 atsukoito: I disabled the per-ip rate limiting again, your scripts should be unstuck 
[09:30:38] <claime>	 jelto: ack. I wonder if there's something to do so that scripts from deploy servers don't get rate limited though (which is the problem atsukoito was encountering afaict)?
[09:32:00] <jelto>	 mid-term I think they could use the discovery record when gitlab is behind the CDN. I don't think gitlabs built-in rate limiting has some kind of allow-list
[09:32:07] <atsukoito>	 jelto: ack, I'll try to move my DPE scripts behind `private_token` auth. is there any chances that we could have a separate endpoint that only allows authenticated API?
[09:32:09] <jelto>	 but I'll check the docs and settings
[09:32:25] <claime>	 jelto: <3
[09:32:33] <atsukoito>	 ah, discovery, lemme check
[09:33:11] <claime>	 atsukoito: That would be in the near-ish future when it's behind the CDN, unsure that would work rn
[09:33:36] <volans>	 also if the backend is the one having issues it would not change much
[09:34:05] <claime>	 volans: Oh for sure, I'm just ticking boxes :P
[09:34:08] <jelto>	 we have discovery records in place but they will not help at the moment, after the CDN migration
[12:03:41] <jynus>	 hi, I got an unexpected CI error on puppet for "error during compilation: Function lookup() did not find a value for the name 'profile::conftool::hiddenparma::api_tokens'" but my PR doesn't touch any of that, any suggestion? I seems it is complaining about some cleanup on labs-private
[12:04:04] <jynus>	 or on tests
[12:04:41] <jynus>	 (It is the tests because puppet compilation for my change works fine, as expected)
[12:05:53] <jynus>	 I will ask joe when he comes back from lunch, maybe it is some cleanup leftover from his work on requestctl leftovers
[12:41:56] <_joe_>	 atsukoito: merging your labs/private change
[12:53:05] <arnaudb>	 same hiddenparma error here
[12:55:08] <jynus>	 I have a meeting now, but if someone can have a further look?
[13:04:43] <fabfur>	 _joe_: ^^ maybe?
[13:05:00] <_joe_>	 rebase your changes?
[13:05:33] <_joe_>	 I did send a fix earleir
[13:05:42] <_joe_>	 *earlier
[13:10:52] <arnaudb>	 it works, thanks _joe_ 
[13:56:44] <atsukoito>	 _joe_: sorry, I forgot to merge it
[14:02:51] <_joe_>	 np!
[14:08:30] <cdanis>	 jynus: ok to merge your bacula patch too?
[14:09:25] <jynus>	 yes
[14:09:29] <jynus>	 I was about to
[14:09:39] <cdanis>	 proceeding
[14:09:42] <jynus>	 thanks
[16:42:01] <btullis>	 There is something blocking the deployment-charts repo sync on deploy1003. Looks like it's related to ml-serve. 
[16:51:51] <Raine>	 ^ I saved the file in my ~ and cleaned up the git dir
[16:52:24] <Raine>	 `-rw-rw-r-- 1 kamila wikidev 25770 Jun  8 16:50 /home/kamila/ml-serve.yaml`
[16:52:53] <urbanecm>	 ty!
[16:54:41] <Raine>	 whoever left it there, it would be great if you can be responsive here if you really need to make edits there
[16:55:12] <Raine>	 *here and in -operations ideally
[16:57:51] <claime>	 Yeah that's my fault
[16:57:53] <claime>	 I think
[16:58:11] <claime>	 I backported a change and got pulled into a meeting
[16:58:40] <claime>	 You can remove the file Raine 
[16:58:54] <Raine>	 ack, thanks for popping up
[17:19:04] <btullis>	 Thanks all for sorting it <3
[19:49:00] <kostajh>	 I’m having issues with a scap backport now (https://spiderpig.wikimedia.org/jobs/2210), is it related to the deployment-charts issue from earlier today?
[20:06:54] <kostajh>	 ^ Pressing “retry” worked