[02:04:21] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T403439 (10aliu) 03NEW [05:25:57] 10GitLab: Merge request "almost ready" forever - https://phabricator.wikimedia.org/T403440 (10Samwilson) 03NEW [05:26:19] 10GitLab: Merge request "almost ready" forever - https://phabricator.wikimedia.org/T403440#11137736 (10Samwilson) [05:46:39] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [YOUR DEVELOPER ACCOUNT USERNAME HERE] - https://phabricator.wikimedia.org/T403439#11137751 (10Waddie96) Hey @aliu, did you possibly submit this by mistake? [05:47:51] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for aliu - https://phabricator.wikimedia.org/T403439#11137752 (10Pppery) [05:48:16] 10GitLab: Merge request "almost ready" forever - https://phabricator.wikimedia.org/T403440#11137753 (10Samwilson) [05:48:54] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for aliu - https://phabricator.wikimedia.org/T403439#11137755 (10Pppery) 05Open→03Resolved a:03Pppery Welcome to the club! [09:49:57] I think I'm seeing the same as ^, rebasing just gets stuck, this seems to be the same that happened some time ago [10:33:11] jnuche: maybe you can give it a look? last time iirc restarting gitlab helped (I'm kinda blocked xd) [10:40:02] dcaro: I can try restarting, let me check [10:42:36] I'm away from the laptop, be back in 10 [10:49:40] dcaro: gitlab restarted, but that doesn't seem to have helped [10:49:53] arnaudb: would you be around to take a look? [10:57:15] (for the record, what I did was running `gitlab-ctl restart`) [11:38:22] checking jnuche, will keep you posted [11:41:36] arnaudb: ty, on the admin UI I can see sidekick is not picking up jobs anymore and the default queue keeps growing [11:47:14] not fully accurate, we're still processing some jobs it seems [11:48:56] merge request are impossible to create, there is something weird and a restart does not fix, I'll continue digging [11:50:33] I got a ' created a merge request' email for a merge request 16h old, 8 min ago [11:50:46] https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 [11:53:27] interesting, I created one just a bit after that https://gitlab.wikimedia.org/repos/sre/ansible-data-persistence-operations/-/merge_requests/2 and fabfur before [11:53:38] https://gitlab.wikimedia.org/fabfur/benchmark-scripts/-/merge_requests/1 [11:54:17] did you get emails for them? [11:56:54] none, but I'm not sure I should have [11:57:11] arnaudb, dcaro: a test pipeline I created a while ago has just started running: https://gitlab.wikimedia.org/repos/releng/scap/-/pipelines/131378 [11:57:24] I've created a fake merge request: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/209 [11:58:12] nice [11:58:14] and it goes through [11:58:48] and fabfur's also went through [11:59:17] and you said you restarted gitlab jnuche ? [11:59:41] connections on redis side seems to be piling up (<1600 currently, `root@gitlab1004:/opt/gitlab# bin/gitlab-redis-cli --stat`) [11:59:51] arnaudb: yeah, using `gitlab-ctl restart`. That was an hour ago [12:00:40] my pipeline started running though https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/30/pipelines [12:00:49] so maybe it's just starting to get the stuck load? [12:01:08] (no idea how many connections it usually has xd) [12:01:36] 10GitLab (Project Migration), 06Community-Tech, 06translatewiki.net, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11138592 (10Samwilson) 05Open→03Resolved [12:02:38] I'm guessing that the dip here was the restart [12:02:41] https://usercontent.irccloud-cdn.com/file/AFyhZM24/image.png [12:03:43] well, pipelines seem to be going through again [12:03:55] arnaudb: did you anything besides restarting again? [12:04:55] nothing special, I'm trying to see if I can spot something odd on the graphs and the logs [12:05:03] was redis restarted? (might have restarted itself?) [12:05:05] https://www.irccloud.com/pastebin/vPtbbIvE/ [12:05:11] it did: https://grafana-rw.wikimedia.org/d/Wx3FDfZnz/gitlab-omnibus-redis?orgId=1&from=now-24h&to=now&timezone=utc&var-instance=gitlab1004:9121&refresh=1m [12:05:40] but it is expected with the gitlab-ctl command [12:06:32] ack, that shows some pattern change yesterday night [12:07:29] thanks for getting it unstuck :) [12:09:05] I think I got a clue, jnuche dcaro : https://grafana.wikimedia.org/goto/-KYACSrNg?orgId=1 → it seems that the first restart failed to restart redis [12:09:26] so when I ran the command, it took care of the dangling bit and fixed it [12:10:04] `for i in $(seq 10); do gitlab-ctl restart; done` xd [12:11:38] arnaudb: mmh, I'm not sure how to read that graph, the bar went up at around 11.45 which is when I restarted. How can you tell it failed? [12:12:07] if I deciphered that metric properly, it stores the start time timestamp in second [12:12:23] so a restart should show you a bump to the last restart time [12:12:46] (uptime is computing now() - that metric) [12:14:26] 11:45 UTC right? [12:14:36] seems about right [12:15:05] jnuche: I think your restart was ~10:50 UTC [12:16:39] yeah, you're right, around 10:45, not 11 [12:17:39] you can see the time the command was run in the history with `HISTTIMEFORMAT="%F %T " history` if you have your shell open [12:18:16] so those graphs must actually be pointing at the second restart [12:18:23] dcaro: uh nice, I didn't know that [12:18:41] https://www.irccloud.com/pastebin/h1ltlGB1/ [12:18:46] there you go [12:19:02] (the host is on UTC) [12:19:19] https://www.irccloud.com/pastebin/IiOrgKyI/ [12:21:14] 10GitLab: Merge request "almost ready" forever - https://phabricator.wikimedia.org/T403440#11138722 (10dcaro) This should have been fixed now :) [12:23:31] 10GitLab: Merge request "almost ready" forever - https://phabricator.wikimedia.org/T403440#11138731 (10Samwilson) 05Open→03Resolved a:03Samwilson Thank you! Looks good now. [12:23:41] 10GitLab: Merge request "almost ready" forever - https://phabricator.wikimedia.org/T403440#11138736 (10Samwilson) a:05Samwilson→03dcaro [12:25:23] whelp, in any case, thank you arnaudb [12:25:34] np! hth! [12:34:46] thanks both! 👍 [12:48:25] yep, I can confirm https://grafana.wikimedia.org/goto/tG1w6I9NR?orgId=1 [12:48:38] oops sorry wrong chan ↑ [16:50:26] 10GitLab (Project Migration), 06collaboration-services, 10Fundraising analytics stack, 10Release-Engineering-Team (Priority Backlog 📥): Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11140308 (10Jgreen) a:05Jgreen→03AStein-WMF