[03:33:36] <wikibugs>	 10GitLab: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295 (10Count_Count) 03NEW
[03:41:37] <countcount>	 All my Gitlab CI jobs fail currently. Is that already known? Otherwise see my report at https://phabricator.wikimedia.org/T424295
[09:39:28] <wikibugs>	 10GitLab: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11854609 (10Count_Count) Hmm, `docker pull docker-hub-mirror.cloud.releng.team/library/rust:latest` works for me locally. Not sure what is going on.
[09:47:55] <countcount>	 It actually does not work locally
[09:48:12] <countcount>	 ❯ docker pull docker-hub-mirror.cloud.releng.team/library/rust:latest
[09:48:12] <countcount>	 Error response from daemon: received unexpected HTTP status: 503 Service Unavailable
[10:03:25] <arnoldokoth>	 Gitlab needs a brief maintenance in about an hour.
[11:25:52] <arnoldokoth>	 Maintenance is complete.
[14:59:34] <wikibugs>	 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11855926 (10dancy)
[14:59:39] <dancy>	 countcount: Taking a look
[15:22:02] <wikibugs>	 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11855990 (10dancy) I ran `dmesg` on the node that is current running the docker-hub-mirror pod and I see that the nginx process has been killed several times due to OOM...
[15:37:10] <wikibugs>	 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856063 (10dancy) 05Open→03Resolved a:03dancy I doubled the memory request/limit for the nginx container.    @Count_Count Thanks for the report.  This should...
[15:40:24] <wikibugs>	 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856086 (10Count_Count) @dancy Works now, thanks!
[15:41:21] <wikibugs>	 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856091 (10dancy) 05Resolved→03Open I just saw another OOM kill so I'm making more adjustments.
[15:55:40] <wikibugs>	 10GitLab, 06Release-Engineering-Team, 13Patch-For-Review: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856119 (10dancy) 05Open→03Resolved
[16:42:50] <wikibugs>	 10GitLab (CI & Job Runners), 06collaboration-services, 07IPv6: IPv6 non-functional in GitLab CI environments - https://phabricator.wikimedia.org/T423310#11856314 (10CDobbins) How (assuming it's even possible) can we switch to the DigitalOcean runner? I tried adding the `cloud` tag to the `build` stage and ev...
[17:36:54] <derenrich>	 would turning on secret push protection be something we could do? https://docs.gitlab.com/user/application_security/secret_detection/secret_push_protection/ we had an issue where a key got through and only noticed when we mirrored to github
[17:37:57] <taavi>	 derenrich: that's not available on the free software variant of gitlab we run
[17:38:04] <derenrich>	 oh dear