[07:49:16] Getting "Something went wrong during merge pre-receive hook. 500 Internal Server Error. Try again." on"https://gitlab.wikimedia.org/countcount/common-ci-templates/-/merge_requests/1". Any maintenance or known problems right now? [07:50:13] Tried merging previously which did not work either. [07:58:32] Merging worked after trying a couple of times. [08:08:16] Issues are definitely still ongoing on multiple projects with message like e.g. "An error occurred while fetching folder content." [09:32:52] 10GitLab (CI & Job Runners), 06collaboration-services, 06Release-Engineering-Team: gitlab workers ulimit nofiles 1073741816 slows down fakeroot - https://phabricator.wikimedia.org/T426827#11954282 (10fgiunchedi) My `ulimit -n 4096` bandaid is deployed in cloud/cicd/gitlab-ci now; I don't know whether there's... [10:02:23] gitlab needs a short restart in a couple of seconds [10:07:09] gitlab should be back [10:29:08] Still seeing stability issues, e.g. getting "An error occurred while fetching the assigned milestone of the selected merge_request." on just looking at https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/85 [10:30:01] gitlab graphql giving 404 errors [10:37:14] HakanIST: can you provide more information about your 404 issue? which endpoint? are you authenticated? do you have a link? [10:37:48] countcount: yes gitlab is getting increased traffic lately, we are aware and troubleshooting the issue [10:38:18] POST https://gitlab.wikimedia.org/api/graphql → HTTP/2 404 [10:38:25] seeing this in chrome console (authenticated) [10:41:54] https://gitlab.wikimedia.org/dashboard/home [10:44:55] yes I can confirm this, thank you for reporting the issue [10:58:38] 👋 more people are reporting issues with GitLab in Slack. Myself I'm getting "An error occurred while fetching folder content" for any of my repos [11:00:50] I'm on it, give me two more minutes [11:01:56] didn't mean to add any more pressure, sorry about that... thanks for investigating [11:04:02] the issue should be resolved now [11:06:43] yes works great, thank you [11:27:57] thanks a lot [12:07:25] gitlab needs another short restart in a couple of seconds, sorry for the disruptions. [12:11:48] gitlab should be back [14:09:23] 10GitLab, 06collaboration-services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11955399 (10ABran-WMF) [14:29:04] 10GitLab, 06collaboration-services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11955494 (10ABran-WMF) [15:18:19] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [@hahmed] - https://phabricator.wikimedia.org/T427089#11955773 (10HAhmed-WMF) @Aklapper Hi, yes that banner did display. I linked my LDAP account to my Phabricator and have "wmf" group access [17:20:34] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11956500 (10brennen) I'm guessing [[ https://gitlab.wikimedia.org/repos/releng/reggie | reggie ]] just needs a restart here. [19:32:44] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11956931 (10thcipriani) Looks like the pvc for reggie is out of space. looking at how to clean that now. [20:03:03] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11957060 (10thcipriani) Opted to add more space to reggie for now: ` ❯ ./scripts/resize-pvc.py reggie 128Gi Using kubectl context "do-nyc3-c... [20:11:00] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11957078 (10thcipriani) Seeing layers get created now in the registry logs; e.g.: ` "POST /v2/repos/abstract-wiki/wikifunctions/function-eva... [20:23:53] 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324 (10A_smart_kitten) 03NEW [20:42:56] 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957163 (10brennen) Thanks for noting. I was //just// looking at the comments on T427250 and wondering what had... [20:52:37] 10GitLab, 06collaboration-services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#11957226 (10Dzahn) I switched the gitlab-lb IPs from reserved to active state in netbox and ran the netbox sync. Meaning now they actually exist in DNS.... [22:02:30] 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957439 (10A_smart_kitten) [22:07:46] 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957447 (10A_smart_kitten) >>! In T427324#11957163, @brennen wrote: > This triggers a faint itching somewhere i... [22:17:32] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: registry.cloud.releng.team returns 500 responses when pushing layers - https://phabricator.wikimedia.org/T427250#11957480 (10thcipriani) New size is now persisted in the statefulset. BUT we've added 10G to the registry in the past hour: {F84460301} Mo... [22:37:14] 10GitLab (Integrations), 06Release-Engineering-Team: @CodeReviewBot is sometimes only commenting when a GitLab MR is merged, and not also when it's opened - https://phabricator.wikimedia.org/T427324#11957553 (10brennen) p:05Triage→03Low I can confirm from logs that gitlab-webhooks got two `merge_request` e...