[06:35:18] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12452621 (10ABran-WMF) [09:28:48] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774 (10hashar) 03NEW [09:28:56] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453190 (10hashar) p:05Triage→03Unbreak! [09:29:05] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453193 (10hashar) [09:33:09] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453255 (10hashar) #collaboration-services can you check the 502 that occurred on Gitlab side at 01:02:12 UTC? ` 01:02:12 stderr: remote: GitL... [09:39:06] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453299 (10ABran-WMF) >>! In T441774#12453255, @hashar wrote: > #collaboration-services can you check the 502 that occurred on Gitlab side at... [09:39:31] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Nadia Isanga - https://phabricator.wikimedia.org/T441776 (10Nisangaa) 03NEW [11:01:47] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453582 (10ABran-WMF) 05Open→03Resolved p:05Unbreak!→03Medium a:03ABran-WMF Feel free to reopen if needed, that should be fixed :-) [11:03:02] 10GitLab, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12453593 (10ABran-WMF) 05In progress→03Resolved I think this is now fixed? feel free to reopen if needed [11:03:13] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12453598 (10ABran-WMF) [11:23:14] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12453703 (10gmodena) [13:18:35] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12454510 (10hashar) a:05ABran-WMF→03hashar > gitlab was heavily scraped tonight, it is now behind CDN per T425441: gitlab behind CDN so... [13:44:21] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454635 (10trueg) 05Resolved→03In progress Re-opening the ticket. We can now access Gitlab on `eqiad`, `codfw`, how... [13:45:07] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454638 (10ABran-WMF) the fix has been applied to codfw as well but there might be a catch, I'll check [13:45:18] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454639 (10ABran-WMF) p:05Unbreak!→03High [14:06:16] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454769 (10ABran-WMF) The codfw timeouts have the same cause as before, in a form the earlier app... [14:06:45] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12454781 (10ABran-WMF) [14:07:12] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12454783 (10ABran-WMF) p:05High→03Medium [14:09:44] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12454812 (10ABran-WMF) p:05High→03Medium [14:20:49] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12454904 (10ABran-WMF) [14:56:34] 10GitLab, 06Collaboration-Services, 10Deployments, 06Release-Engineering-Team: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12455123 (10dancy) Thanks for fixing things up @hashar! [15:20:08] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455288 (10trueg) `wdqs-proxy` [[ https://gitlab.wikimedia.org/repos/wikidata-pla... [15:26:55] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455315 (10trueg) @ABran-WMF The only thing I do not understand is the need for t... [15:30:38] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455330 (10trueg) CR https://gerrit.wikimedia.org/r/c/operations/deployment-chart... [15:31:48] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455335 (10trueg) 05In progress→03Resolved [16:41:53] 10GitLab, 06Collaboration-Services, 06Product Safety and Integrity, 06Research, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12455630 (10sbassett) [17:08:14] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Release-Engineering-Team (Radar): gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12455834 (10bd808) I haven't had time to try and make my logging move verbose yet, but {T441864} seems very likely directly related to the cutover announ... [17:44:48] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Nadia Isanga - https://phabricator.wikimedia.org/T441776#12456031 (10brennen) 05Open→03Resolved a:03brennen [17:45:02] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Nadia Isanga - https://phabricator.wikimedia.org/T441776#12456040 (10brennen) Approved. Happy hacking!