[12:45:09] <dcaro>	 Hi! I'm seeing errors connecting to gitlab from digitalocean gitlab runners
[12:45:12] <dcaro>	 like https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363466
[12:45:26] <dcaro>	 fatal: unable to access 'https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api.git/': Failed to connect to gitlab.wikimedia.org port 443 after 4 ms: Could not connect to server
[12:45:37] <dcaro>	 is there a known issue going on?
[12:47:05] <dcaro>	 oh, even some that seem to work, did fail to contact gitlab
[12:47:06] <dcaro>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363459
[12:47:14] <dcaro>	 ERROR: Uploading artifacts as "dotenv" to coordinator... error  error=couldn't execute POST against https://gitlab.wikimedia.org/api/v4/jobs/363459/artifacts?artifact_format=gzip&artifact_type=dotenv: Post "https://gitlab.wikimedia.org/api/v4/jobs/363459/artifacts?artifact_format=gzip&artifact_type=dotenv": dial tcp 208.80.153.8:443: connect: connection refused id=363459 token=glcbt-64
[12:51:08] <dcaro>	 maybe you know something? jelto^  (it's breaking all my ci builds)
[12:54:15] <dcaro>	 oh, some do pass still https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363487
[12:55:01] <dcaro>	 hmm, this is in the same runner, almost at the same time, and failing https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363488
[12:56:09] <dcaro>	 now it passed that one, but failed on the next, so it's not failing all the time, it's flaky enough to not pass a full pipeline
[13:03:04] <dcaro>	 if I retry enough times I might get the pipeline finished xd, but it's many times (the issue seems to happen ~75% of the time)
[14:26:18] <dcaro>	 oh, things seem to be running again :)
[14:28:53] <dcaro>	 oh no, I was just lucky twice in a row xd
[14:28:54] <dcaro>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/363560
[14:28:59] <dcaro>	 ^still failing :/
[14:33:42] <mutante>	 dcaro: would it be an option for you to use our own infra, the runners in cloud instead of digital ocean?
[14:33:59] <mutante>	 afaik those are experimental
[14:34:25] <mutante>	 the current issue should be gone if you avoid DO for right now
[14:37:21] <dcaro>	 would be an option yep, we need the big mem ones though for golang stuff, do we have those on cloud too?
[14:38:39] <mutante>	 dcaro: glad to hear that. I'm sorry I don't know that off hand. I can try t find out
[14:38:59] <dcaro>	 np, let me know if I can help in any way
[14:39:45] <mutante>	 dcaro: the main problem is that the DO runners open too many connections at once
[14:39:58] <mutante>	 it's probably timing that more than one user is on it at the same time 
[14:40:12] <dcaro>	 aaaahhh
[14:40:18] <mutante>	 one fix would be to somehow slow them down 
[14:40:46] <mutante>	 you would not be limited from our own runners
[21:19:43] <mutante>	 production gitlab server needs to be upgraded. please expect short downtime in a few minutes.