[08:34:46] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 17 - https://phabricator.wikimedia.org/T365675#9933531 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade was started by jelto@cumin1002 on GitLab host gitlab2002.w... [09:38:14] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 17 - https://phabricator.wikimedia.org/T365675#9933660 (10Jelto) [10:32:24] GitLab needs a short maintenance in 30 minutes. [11:06:29] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 17 - https://phabricator.wikimedia.org/T365675#9933881 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade started by jelto@cumin1002 on GitLab host gitlab2002.wikim... [11:09:02] GitLab maintenance finished [11:09:40] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 17 - https://phabricator.wikimedia.org/T365675#9933905 (10Jelto) [11:13:44] 10GitLab (Infrastructure), 06collaboration-services, 06Release-Engineering-Team, 13Patch-For-Review: Upgrade GitLab to major version 17 - https://phabricator.wikimedia.org/T365675#9933928 (10Jelto) All instances (except the WMCS Bookworm Runner) are now on GitLab version 17. I'll leave the task open over t... [16:14:47] 10GitLab, 06Release-Engineering-Team: Reggie endpoints returning 500 - https://phabricator.wikimedia.org/T368751#9935157 (10brennen) [16:37:59] 10Tool-gitlab-account-approval: Switch to keyset based pagination - https://phabricator.wikimedia.org/T368761 (10bd808) 03NEW [16:56:18] attempting to fix some reggie issues post the gitlab 17 upgrade, looks like we missed a jwt endpoint [16:58:12] so there's a manual step next to deploy to prod? [16:58:29] there sure is, also a manual step for staging, but I felt bolder clicking it [16:59:04] looks like it just modified reggie, just like we wanted? [16:59:32] so I should definitely click the big red button? [16:59:33] 10GitLab, 06Release-Engineering-Team, 13Patch-For-Review: Reggie endpoints returning 500 - https://phabricator.wikimedia.org/T368751#9935356 (10cjming) we were seeing this too https://gitlab.wikimedia.org/repos/data-engineering/mpic/-/jobs/297300 - just started happening [16:59:50] seems like let it rip [16:59:57] i doubt it will get _more_ broken [17:02:37] now waiting on yet another pipeline: https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/pipelines/61029 [17:04:34] looks like it's done, lemme check what's deployed now [17:05:40] looks like a new reggie has started, and the REGGIE_JWKS_ENDPOINT is correct [17:05:57] success? let's try to re-run some tests [17:09:49] success https://gitlab.wikimedia.org/repos/releng/docpub/-/jobs/297335 [17:09:55] * thcipriani closes task [17:09:59] nice work. [17:10:00] 10GitLab, 06Release-Engineering-Team, 13Patch-For-Review: Reggie endpoints returning 500 - https://phabricator.wikimedia.org/T368751#9935385 (10thcipriani) Deployed what we think should be the fix. For posterity, GitLab 17 removed the old jwt endpoint. For our registry (reggie) this comes from the environme... [17:10:28] once again, vaugely knowing how things work and a reckless willingness to push buttons pays off [17:11:11] really the foundation of my career to date [17:13:44] 10GitLab, 06Release-Engineering-Team, 13Patch-For-Review: Reggie endpoints returning 500 - https://phabricator.wikimedia.org/T368751#9935401 (10thcipriani) 05Open→03Resolved a:03jnuche Success https://gitlab.wikimedia.org/repos/releng/docpub/-/jobs/297335 Retried a formerly failing job and it work... [17:53:57] thcipriani: I had to save https://bash.toolforge.org/quip/c1n6X5ABhuQtenzv1grk because I'm also in that picture [18:03:46] :D [21:15:25] 10GitLab-Application-Security-Pipeline, 06Security-Team: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468#9936233 (10sbassett) Write-up of some of this quarter's work is here: T335892#9936225