[08:33:15] Project mediawiki-core-doxygen build #17630: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17630/ [08:52:17] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 07ci-test-error (WMF-deployed Build Failure), 06Wikidata-Omega (Radar/Epics/Stalled): Zuul does not support Gerrit rebase submit strategy correctly, may merge deterministically broken code (bre... - https://phabricator.wikimedia.org/T400918#11595135 [08:54:37] 10Gerrit, 06collaboration-services: Opt-in testing of Gerrit-via-CDN - https://phabricator.wikimedia.org/T414719#11595141 (10Jelto) Great thanks for digging. Marking a file as reviewed works on my side on randomly selected changes. If you click on "MARK REVIEWED" when you hoover over a file this sends a `PUT`.... [09:05:36] 06Project-Admins: Create project tag for Wishlist-FY25-26 - https://phabricator.wikimedia.org/T415913#11595187 (10KSiebert) @Aklapper any more advice? I am keeping this ticket open only to collect a history of tips. We can close it after a find a solution. [09:13:22] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/HotCat] Add BetaFeatures phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1237721 (owner: 10Fomafix) [09:15:10] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/HotCat] Add BetaFeatures phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1237721 (owner: 10Fomafix) [09:30:47] (03PS1) 10Hashar: Zuul: [GrowthExperiments] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1237857 (https://phabricator.wikimedia.org/T389998) [09:32:23] Yippee, build fixed! [09:32:23] Project mediawiki-core-doxygen build #17631: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17631/ [09:45:52] (03CR) 10Hashar: [C:03+2] Zuul: [GrowthExperiments] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1237857 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:47:08] (03PS1) 10Hashar: Zuul: [CheckUser] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1237859 (https://phabricator.wikimedia.org/T389998) [09:47:23] (03Merged) 10jenkins-bot: Zuul: [GrowthExperiments] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1237857 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:49:45] (03PS1) 10Hashar: Zuul: [MetricsPlatform] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1237860 (https://phabricator.wikimedia.org/T389998) [09:49:47] (03PS1) 10Hashar: Zuul: [TestKitchen] disable recursion [integration/config] - 10https://gerrit.wikimedia.org/r/1237861 (https://phabricator.wikimedia.org/T389998) [10:15:56] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit switchover process - https://phabricator.wikimedia.org/T387833#11595674 (10ABran-WMF) [10:20:08] 06Release-Engineering-Team (Radar), 10Ceph, 06ServiceOps new, 10SRE-swift-storage, and 3 others: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - https://phabricator.wikimedia.org/T412951#11595680 (10elukey) To keep archives happy - me and Matthew will tentatively sc... [11:31:24] 10Phabricator: Silent batch-edit request for PHP 8.4/8.5 CI failure tasks currently tagged with the 'WMF-deployed Build Failure' project - https://phabricator.wikimedia.org/T413865#11596161 (10Aklapper) I guess I'm just sitting here with popcorn waiting for potential agreement / a decision. Or someone saying "le... [11:39:34] (03PS1) 10Santiago Faci: Renaming TestKitchen: Removing MetricsPlatform as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1237871 (https://phabricator.wikimedia.org/T414435) [11:47:41] 10Phabricator: Silent batch-edit request for PHP 8.4/8.5 CI failure tasks currently tagged with the 'WMF-deployed Build Failure' project - https://phabricator.wikimedia.org/T413865#11596220 (10Jdforrester-WMF) 8.4 will be blocking CI soon enough anyway, and 8.5 soon-ish. Let's just decline? [11:48:56] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1237871 (https://phabricator.wikimedia.org/T414435) (owner: 10Santiago Faci) [11:49:47] (03CR) 10Jforrester: [C:03+2] "Ia0e454bd2e9285572722276fef03a464c411cd69 is live in master and was back-ported to wmf.14, so it looks to me like this is good to land." [integration/config] - 10https://gerrit.wikimedia.org/r/1237871 (https://phabricator.wikimedia.org/T414435) (owner: 10Santiago Faci) [11:51:18] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1237871 (https://phabricator.wikimedia.org/T414435) (owner: 10Santiago Faci) [11:51:51] !log Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency, for T414435 [11:51:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:51:53] T414435: [Renaming TestKitchen] Update ReadingList extension - https://phabricator.wikimedia.org/T414435 [12:07:02] (03CR) 10Jforrester: "@hashar, your approval on this patch requested. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/1234514 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [12:10:17] 10Phabricator: Silent batch-edit request for PHP 8.4/8.5 CI failure tasks currently tagged with the 'WMF-deployed Build Failure' project - https://phabricator.wikimedia.org/T413865#11596314 (10A_smart_kitten) FWIW, In the interests of being on the same page / coming to a conclusion here, I'm happy for #ci-test-e... [14:26:37] 10Continuous-Integration-Config, 06Release-Engineering-Team (Seen), 06ServiceOps new, 06SRE: operations/docker-images/production-images has no CI - https://phabricator.wikimedia.org/T283855#11596821 (10bking) Thanks @hashar , I must have had my brain dripping out of my ears when I wrote that last update. A... [15:53:44] 06Release-Engineering-Team, 10Scap, 06ServiceOps new, 07Epic: Proposal: scap deploy-service - https://phabricator.wikimedia.org/T412941#11597216 (10brouberol) #data-platform-sre would be very interested to help out as well! We have 11 (soon 12) airflow instances we have to manually deploy one by one everyt... [15:55:22] (03open) 10hnowlan: Add status.wikimedia.org miscweb service to projects [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/144 (https://phabricator.wikimedia.org/T414098) [15:56:00] (03merge) 10dancy: Add status.wikimedia.org miscweb service to projects [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/144 (https://phabricator.wikimedia.org/T414098) (owner: 10hnowlan) [16:36:23] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 17 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11597592 (10Jgiannelos) [17:11:12] 10Continuous-Integration-Config, 10Quibble: Core tests aren't run with extensions installed - https://phabricator.wikimedia.org/T206243#11597888 (10A_smart_kitten) I've ended up coming across this myself (ie., that `mw/core`'s tests aren't run in CI for MediaWiki extensions) in the last few days, after discove... [17:37:42] dancy: thank you for being so quick with the merge! <3 [17:37:57] No problem. That was an easy one [17:49:00] I think my phone is being monitored thru this method [17:51:14] Guest27: wrong place [18:06:09] I wonder if someone of us (RelEng) who is more GitLab than me should look into https://phabricator.wikimedia.org/T414781 which is coming closer? [18:09:22] (03open) 10jforrester: Dockerfile: Use Sury's builds of PHP so we get latest modules [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/108 (https://phabricator.wikimedia.org/T414036) [18:15:17] 10Gerrit, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912 (10Reedy) 03NEW [18:15:30] 10Gerrit, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598392 (10Reedy) p:05Triage→03High [18:19:05] andre: having a look [18:19:12] thanks [18:22:40] 10Gerrit, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598413 (10Reedy) [18:27:30] 10Gerrit, 06collaboration-services: Opt-in testing of Gerrit-via-CDN - https://phabricator.wikimedia.org/T414719#11598426 (10Dzahn) 05Open→03Resolved calling it enough testing based on "one of three 5xx we've served for Gerrit via the CDN in the past month. The other two were healthchecks from Liberica" [18:27:46] 10Gerrit, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598429 (10Jdforrester-WMF) This also breaks our gerrit -> GitHub -> Packagist integrations, of course. [18:31:29] 10Gerrit, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598464 (10bd808) * Last github mention in gerrit1003:/var/log/gerrit/replication_log was 2026-02-09 13:51:05 (4+ hours ago). * Last git@github.com:wikimedia/mediawiki-core mention was 2026-02... [18:32:57] Project mediawiki-core-doxygen build #17640: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17640/ [18:32:57] hashar: do you know more about gerrit->gitlab replication debugging than https://wikitech.wikimedia.org/wiki/Gerrit/Administration ? [18:34:04] 10Gerrit, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598473 (10Jgiannelos) [18:43:50] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598513 (10brennen) [18:59:10] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598590 (10bd808) The server doesn't seem to know that it is behind: `lang=shell-session bd808@mbp03:~/projects/wmf$ ssh -i ~/.ssh/id_ed25519 -o IdentityAgent=$SS... [19:03:53] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T413806#11598609 (10DAlangi_WMF) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1233340 * **Summ... [19:15:28] i just got a unicorn while trying to log in to github, so it definitely seems like they're not done having issues. [19:18:10] aha yaeh https://www.githubstatus.com/incidents/lcw3tg2f6zsd [19:18:14] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598640 (10brennen) GitHub [[ https://www.githubstatus.com/ | appears ]] to have quite a few ongoing issues. {F71778040} [19:18:44] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598641 (10Reedy) https://www.githubstatus.com/incidents/lcw3tg2f6zsd seems/feels related... [19:24:20] bd808: nothing at all and I am pretty sure I vetoed having Gerrit repos being replicated by GitLab [19:24:35] oh [19:24:37] that is github [19:25:58] lol [19:26:02] naming thing is hard [19:26:18] gotta restart gerrit [19:29:33] yeah sorry [19:29:36] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598661 (10hashar) We have a dashboard at https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream though it is hard to read and does not make the issue sta... [19:29:44] bd808: the replica plugin loose track of stuff when the config is changed :/ [19:30:06] that sounds like a bug or 5 [19:30:52] and my daughters loves the ascii art eyes when connecting to gerrit1003 :) [19:32:50] Yippee, build fixed! [19:32:51] Project mediawiki-core-doxygen build #17641: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/17641/ [19:33:57] Project beta-code-update-eqiad build #586897: 04FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/586897/ [19:36:11] done [19:36:13] bd808: solved! [19:36:21] Gerrit is running a replication to Github [19:36:33] (well really it replicate all repos to all replicas) [19:39:11] (03PS1) 10Reedy: Zuul: Add extension-apitests to WikimediaCustomizations extension [integration/config] - 10https://gerrit.wikimedia.org/r/1238006 [19:40:15] (03PS2) 10Reedy: Zuul: Add extension-apitests to WikimediaCustomizations extension [integration/config] - 10https://gerrit.wikimedia.org/r/1238006 (https://phabricator.wikimedia.org/T415680) [19:43:16] (03CR) 10Reedy: [C:03+2] Zuul: Add extension-apitests to WikimediaCustomizations extension [integration/config] - 10https://gerrit.wikimedia.org/r/1238006 (https://phabricator.wikimedia.org/T415680) (owner: 10Reedy) [19:43:32] 10Gerrit, 06Release-Engineering-Team, 10Wikimedia-GitHub: Replication to GitHub seems to have stalled - https://phabricator.wikimedia.org/T416912#11598746 (10hashar) 05Open→03Resolved a:03hashar @bd808 mentioned it to me over IRC. I have encountered the bug previously but honestly I never went to... [19:43:34] thank you Bryan [19:43:53] 06Release-Engineering-Team (Radar): Create jenkins job for creating deployment artifacts for `docker-pkg-deploy` - https://phabricator.wikimedia.org/T179562#11598751 (10MLechvien-WMF) [19:45:18] Yippee, build fixed! [19:45:18] Project beta-code-update-eqiad build #586898: 09FIXED in 2 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/586898/ [19:45:58] (03Merged) 10jenkins-bot: Zuul: Add extension-apitests to WikimediaCustomizations extension [integration/config] - 10https://gerrit.wikimedia.org/r/1238006 (https://phabricator.wikimedia.org/T415680) (owner: 10Reedy) [19:46:34] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1238006 T415680 [19:46:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:46:37] T415680: [Placeholder] Release beta module for basic Attribution - https://phabricator.wikimedia.org/T415680 [20:15:16] 10Gerrit, 06collaboration-services, 06Traffic: ATS/Gerrit: validate TLS hosts for gerrit (revert workaround that skips validation) - https://phabricator.wikimedia.org/T411904#11598836 (10Dzahn) 05Stalled→03Open Yes, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1215684/6/hieradata/common/profile/t... [20:17:51] 06Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic: Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11598842 (10Dzahn) [20:17:52] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11598841 (10Dzahn) [20:21:18] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11598866 (10Dzahn) [20:41:17] 06Release-Engineering-Team, 10Scap, 06ServiceOps new, 07Epic: Proposal: scap deploy-service - https://phabricator.wikimedia.org/T412941#11598962 (10bking) Balthazar already mentioned our (as in #data-platform-sre ) interest, but it would also be just as handy for [[ https://wikitech.wikimedia.org/wiki/Data... [20:47:24] <_Gerges_> Can a member of the data platform engineering team review this request https://meta.wikimedia.org/wiki/Steward_requests/Global_permissions#apihighlimits_for_Gerges [21:00:27] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11599036 (10Dzahn) [21:04:40] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11599108 (10cmooney) @taavi what is the rationale for having these on the NAT exemption list? I would have thought cloud vps instances could just access gerrit like anything else on the inte... [21:29:47] 10Gerrit, 07Puppet: Puppet should restart Gerrit when changing it's replication config - https://phabricator.wikimedia.org/T416929 (10bd808) 03NEW [21:33:40] Not sure why Gerges thought this channel would be a good place to talk to data platform. ¯\_(ツ)_/¯ [21:43:08] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit behind CDN - https://phabricator.wikimedia.org/T411895#11599266 (10Andrew) [21:51:08] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job [integration/config] - 10https://gerrit.wikimedia.org/r/1238038 (https://phabricator.wikimedia.org/T410941) [21:51:09] (03PS1) 10Jforrester: [WIP] Zuul: Add PHP 8.5 CI job to various things blocked by Phan [integration/config] - 10https://gerrit.wikimedia.org/r/1238039 (https://phabricator.wikimedia.org/T410941) [21:53:45] (03CR) 10Daimona Eaytoy: [C:03+1] Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job [integration/config] - 10https://gerrit.wikimedia.org/r/1238038 (https://phabricator.wikimedia.org/T410941) (owner: 10Jforrester) [21:55:10] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job [integration/config] - 10https://gerrit.wikimedia.org/r/1238038 (https://phabricator.wikimedia.org/T410941) (owner: 10Jforrester) [21:55:41] (03CR) 10Hashar: [C:03+1] "I have no idea but PHP 8.5 + Phan v6 sounds like we get a newer stack and I really appreciate this happening 🎉" [integration/config] - 10https://gerrit.wikimedia.org/r/1238038 (https://phabricator.wikimedia.org/T410941) (owner: 10Jforrester) [21:56:41] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job [integration/config] - 10https://gerrit.wikimedia.org/r/1238038 (https://phabricator.wikimedia.org/T410941) (owner: 10Jforrester) [21:57:14] 10Gerrit, 07Puppet: Puppet should restart Gerrit when changing it's replication config - https://phabricator.wikimedia.org/T416929#11599302 (10bd808) Now that I understand what I'm reading, https://wikitech.wikimedia.org/wiki/Gerrit/Administration#Troubleshooting documents this problem as well: > Any changes... [21:58:32] 10Gerrit, 07Puppet: Puppet should restart Gerrit when changing it's replication config - https://phabricator.wikimedia.org/T416929#11599305 (10Reedy) [21:58:45] !log Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job, for T410941 [21:58:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:58:48] T410941: PHP 8.5: "Phan 5.5.2 crashed when parsing/analyzing 'plugins/RedundantExistenceChecksPlugin/empty-miscnodes/test.php')" - https://phabricator.wikimedia.org/T410941 [22:00:05] (03open) 10daimona: releases: Bump mediawiki-phan-config to 0.19.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/118 (https://phabricator.wikimedia.org/T406326) [22:01:48] (03approved) 10jforrester: releases: Bump mediawiki-phan-config to 0.19.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/118 (https://phabricator.wikimedia.org/T406326) (owner: 10daimona) [22:01:56] (03merge) 10jforrester: releases: Bump mediawiki-phan-config to 0.19.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/118 (https://phabricator.wikimedia.org/T406326) (owner: 10daimona) [22:05:14] 10Gerrit, 13Patch-For-Review, 07Puppet: Puppet should restart Gerrit when changing it's replication config - https://phabricator.wikimedia.org/T416929#11599335 (10hashar) `git log -GautoReload modules/gerrit` has the history: That got disable by Paladox in October 2019 referencing "a bug". I went to enabl... [22:07:50] 10Gerrit, 13Patch-For-Review, 07Puppet: Puppet should restart Gerrit when changing it's replication config - https://phabricator.wikimedia.org/T416929#11599347 (10Dzahn) > Or we should have some other mechanism to ensure that Gerrit is restarted following a replication config change. Or we should do restart... [22:18:28] 10Gerrit, 07Upstream: Opening the "reply" dialog should focus the comment field, not the "reviewers" field - https://phabricator.wikimedia.org/T416934 (10Daimona) 03NEW [23:08:47] 10Continuous-Integration-Config, 10Quibble: Core tests aren't run with extensions installed - https://phabricator.wikimedia.org/T206243#11599601 (10bd808) →14Duplicate dup:03T249674 [23:08:49] 10Continuous-Integration-Config, 06Release-Engineering-Team (Seen), 10MW-on-K8s, 07Epic, 13Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674#11599598 (10bd808) [23:24:09] Daimona: Thanks for taking care of that unblock request from Lauren. [23:25:16] oh it looks like you got several. even more thanks. :) [23:25:18] No problem, I wanted to unblock (no pun intended) her work on another task and I figured I'd take the chance to refresh my memory on how IP unblocks are done [23:40:40] 10GitLab (Integrations), 06Release-Engineering-Team: Gitlab data-engineering resource access tokens expiring - https://phabricator.wikimedia.org/T414781#11599704 (10brennen) 05Open→03Resolved a:03brennen I set the expiry for this token to "never".