[00:26:29] (03CR) 10Mhurd: [C:03+1] "lgtm" [integration/config] - 10https://gerrit.wikimedia.org/r/1289280 (https://phabricator.wikimedia.org/T426729) (owner: 10Phedenskog) [00:31:55] 06Release-Engineering-Team (Doing 😎), 06ServiceOps new, 05Goal: [FY25-26 WE6.1.4] Establish Pretrain production design for MVP - https://phabricator.wikimedia.org/T417704#11938809 (10Scott_French) Thank you for doing so, and apologies for the delayed response. I'll give some thought to how we can distill the... [07:30:50] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#11939323 (10ABran-WMF) [07:30:52] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Quibble, and 2 others: Implement a retry policy for network errors in CI - https://phabricator.wikimedia.org/T424990#11939322 (10ABran-WMF) [07:34:21] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Quibble, and 2 others: Implement a retry policy for network errors in CI - https://phabricator.wikimedia.org/T424990#11939326 (10ABran-WMF) 05Open→03Resolved a:03ABran-WMF >>! In T424990#11902262, @A_smart_kitten wrote: > so I assu... [07:37:00] 10GitLab (Project Migration), 06Community-Tech, 10Wikimedia OCR: Migrate wikimedia/wikimedia-ocr from GitHub to GitLab - https://phabricator.wikimedia.org/T420317#11939348 (10Samwilson) Updated some wikis: * https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource%3AWikisource%2FWikimedia_OCR&diff=2416... [08:21:12] (03PS3) 10Arnaudb: zuul: retry policy on network errors [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) [08:21:12] (03CR) 10Arnaudb: "nested retry cleaned up, the CI tests are failing for PHP reasons that don't seem linked to this change" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [08:21:30] (03CR) 10Arnaudb: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [08:26:24] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [08:26:29] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T426822 (10wmcs-alerts) 03NEW [08:39:43] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:07:36] 06Release-Engineering-Team, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 15 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11939596 (10Sfaci) [09:08:55] 10GitLab (CI & Job Runners), 06cloud-services-team, 06collaboration-services: webservice-cli package deb gitlab CI job went from 9 minutes to 27 minutes - https://phabricator.wikimedia.org/T426827 (10fgiunchedi) 03NEW [09:35:27] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 3 others: api-gateway: run make test in CI - https://phabricator.wikimedia.org/T424824#11939704 (10hashar) @daniel and I exchanged on that topic. `operations/deployment-charts` has a `helm-lint`... [09:46:02] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525#11939734 (10Viktoria_Hillerud_WMSE) 05Resolved→03Open [09:46:05] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525#11939737 (10Viktoria_Hillerud_WMSE) It seems to be down again: `Request from 62.63.229.82 via deployment-cache-text08.deployment-prep.eqiad1.wikime... [09:49:04] 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525#11939742 (10taavi) 05Open→03Resolved Please file a new task instead of re-using one that's 4 years old and almost certainly caused by an unre... [10:00:05] 10Beta-Cluster-Infrastructure: Beta cluster down: Error: 502, Backend fetch failed - https://phabricator.wikimedia.org/T426831 (10Viktoria_Hillerud_WMSE) 03NEW [10:10:18] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912#11939830 (10Urbanecm_WMF) [10:19:40] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912#11939864 (10Urbanecm_WMF) FWIW, {T426832} warrants a rollback (cf. T426832#11939827 for the reasoning). [10:49:07] PSA: a JRE security update will also be applied upon contint reboot [10:50:06] erratum: please ignore my previous message ↑ [10:52:23] sorry for the back & forth haha I've been told there will be an upgrade when the server reboots [11:15:03] 10Beta-Cluster-Infrastructure: Beta cluster down: Error: 502, Backend fetch failed - https://phabricator.wikimedia.org/T426831#11940093 (10Viktoria_Hillerud_WMSE) [11:15:04] 10Beta-Cluster-Infrastructure, 07Epic: 502 errors on beta cluster - https://phabricator.wikimedia.org/T312253#11940094 (10Viktoria_Hillerud_WMSE) [11:58:28] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Handle invalid names gracefully - https://phabricator.wikimedia.org/T426843 (10jnuche) 03NEW [11:58:45] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Handle invalid names gracefully - https://phabricator.wikimedia.org/T426843#11940289 (10jnuche) [12:18:29] (03PS1) 10Kosta Harlan: zuul: [mediawiki/extensions/Wikibase] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1289940 (https://phabricator.wikimedia.org/T426829) [12:32:53] 10Beta-Cluster-Infrastructure, 06Traffic: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T426822#11940439 (10ssingh) ` Error: Failed to apply catalog: Parameter source failed on File[/etc/haproxy/ip-reputation.d/top_10000_ips_reque... [13:09:38] (03PS2) 10Kosta Harlan: zuul: [mediawiki/extensions/Wikibase] Add ConfirmEdit dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1289940 (https://phabricator.wikimedia.org/T426089) [13:16:03] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 3 others: api-gateway: run make test in CI - https://phabricator.wikimedia.org/T424824#11940675 (10JMeybohm) >>! In T424824#11939704, @hashar wrote: > An easy path is to add Lua/Make to the image... [13:34:17] (03merge) 10thcipriani: backport: Warn when a l10n-touching change is backported [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1187 (https://phabricator.wikimedia.org/T397089) (owner: 10dancy) [13:44:27] oooooh that sounds nice [14:01:28] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 3 others: api-gateway: run make test in CI - https://phabricator.wikimedia.org/T424824#11941030 (10daniel) >>! In T424824#11940675, @JMeybohm wrote: >>>! In T424824#11939704, @hashar wrote: >> An... [14:05:38] (03open) 10dancy: Release 4.266.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1197 [14:10:21] (03update) 10hashar: Reenable doctest and fix up the one that fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1194 [14:11:27] (03merge) 10dancy: Release 4.266.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1197 [14:26:45] 10Scap, 13Patch-For-Review: scap backport should warn if it knows it will take a long time - https://phabricator.wikimedia.org/T397089#11941099 (10dancy) 05In progress→03Resolved Deployed in scap 4.266.0 [15:02:19] hashar: https://integration.wikimedia.org/ci/job/trigger-cxserver-pipeline-test/1133/console - should I file a bug or temporary issue? [15:14:22] kart_: always file a bug I guess ;) [15:14:26] they are cheap! [15:14:47] and can be used as a reference in the future or might be the starting point for a wide scale issue of doom [15:15:28] you can paste the trace from https://integration.wikimedia.org/ci/job/cxserver-pipeline-test/1134/console [15:16:44] it is probably the same as https://phabricator.wikimedia.org/T420865 [15:16:46] git unreachable [15:19:08] OH found it [15:19:16] Java got upgraded on contint which requires Jenkins to be restarted [15:19:36] else Jenkins JVM runs with a different version of java than the one that is now used to load integration/pipelinelib [15:19:41] moritz upgraded Java earlier today [15:20:52] I am restarting the CI Jenkins [15:24:26] and I did recheck https://gerrit.wikimedia.org/r/c/mediawiki/services/cxserver/+/1289739 [15:36:40] 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11941454 (10Aklapper) Theory: * `rMWe50642231cfb` (threshold commit be passed as p... [15:43:09] hashar: Thanks. [15:43:29] hi folks, I mistakenly deleted some messages defined in the DonationInterface extension that were in use on Donatewiki. I have just restored them to the DonationInterface master branch and would like to deploy them to donatewiki this week if possible. Is this the correct procedure? [15:43:58] 1) cherry-pick the restoration patches to the wmf/1.47.0-wmf.3 branch of DonationInterface [15:44:58] 2) submit a patch to the same branch of core updating the submodule pointer [15:45:15] 3) schedule a backport window to get it deployed [15:45:17] ?? [15:45:23] 2) in theory should be done automatically.... But DonationInterface may be an edge case [15:45:39] it definitely is for other extensions [15:45:50] oh cool, no, i think CentralNotice is the only edge case as far as release config [15:45:56] well, among fundraising extensions [15:46:15] if DonationInterface deployed branch is the same as the mediawiki/core deployment branch (wmf/1.47.0-wmf.3), then Gerrit should deal with the submodule update automatically [15:46:23] cool cool [15:46:31] so just steps 1 and 3 then? [15:46:53] Yeah.. .And run scap and wait a long time :) [15:47:05] that sounds a bout right. Given the l10n bot probably has injected some more updates since the wmf/1.47.0-wmf.3 has been branched [15:47:12] ah right, the localization cache :) [15:47:28] so you might want to find another window, rather than using a backport window (unless it's quiet/empty) [15:47:41] ok, i'll look for a clear spot [16:22:42] (03update) 10hashar: Reenable doctest and fix up the one that fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1194 [16:25:37] (03update) 10hashar: Reenable doctest and fix up the one that fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1194 [16:45:00] (03CR) 10Ssingh: "I will give the gdnsd build a try on gerrit and see how it works out." [integration/config] - 10https://gerrit.wikimedia.org/r/1288954 (owner: 10BCornwall) [16:55:33] arnaudb > PSA: a JRE security update will also be applied upon contint reboot [16:56:05] the java upgrade requires Jenkins to be restarted, cause the java mismatch breaks some builds. I did it an hour or so ago after it got mentioned by kart_ [16:56:22] it is not using wmf autoupgrade to prevent it to magically restart Jenkins at unwanted time [16:56:23] ;) [16:56:41] cool. Thanks hashar! [16:56:50] ;) [17:00:28] hashar: ack, thanks for the restart, it'll be restarted again with the reboot then! [17:01:23] arnaudb: I think moritz has a note that the java upgrade requires Jenkins to be restarted [17:01:39] do you plan to reboot contint machine for a kernel upgrade? [17:02:14] cause that flushes the whole CI queue ( https://integration.wikimedia.org/zuul/ ), so usually we do that during European morning before the backport window [17:02:29] or after it if there is there is no train running that week [17:06:19] hashar: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/X5CXSBCR3RE2SNRTTSEF3Z6R3UGH4YIJ/ [17:08:02] Ah yeah I haven't read my emails yet :] [17:08:08] merci andre ! [17:08:45] and they will occur before the backport window, so we are all set 🎉 [17:08:58] I am off for diner etc [17:22:03] (03update) 10dancy: Reenable doctest and fix up the one that fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1194 (owner: 10hashar) [17:24:40] (03merge) 10dancy: Reenable doctest and fix up the one that fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1194 (owner: 10hashar) [17:50:21] 06Release-Engineering-Team (Priority Backlog 📥), 10Catalyst (Luka Ijo Pimeja Jan): Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#11942062 (10thcipriani) a:03SDunlap [17:51:32] 06Release-Engineering-Team (Priority Backlog 📥), 10Catalyst (Luka Ijo Pimeja Jan): Cannot select REL1_46 on PatchDemo - https://phabricator.wikimedia.org/T425165#11942065 (10thcipriani) API docs to grab this info from Gerrit are [[https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html#list... [18:41:56] (03PS1) 10Vaughn Walters: RelatedArticles: Drop selenium CI jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) [18:42:45] (03PS2) 10Vaughn Walters: RelatedArticles: Drop selenium CI jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) [18:44:19] (03CR) 10CI reject: [V:04-1] RelatedArticles: Drop selenium CI jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) (owner: 10Vaughn Walters) [18:49:16] oops, I went and merged the changes to the wmf/1.47.0-wmf.3 branch before hitting the 'schedule' button [18:49:46] anyway, it looks like there's a free hour on the schedule before the UTC late backports window [18:51:14] I just edit the Deployments page directly [18:51:23] (oops hit enter too fast) [18:51:41] so I just edit it directly to notify that I'm doing a one-off, it looks like [18:58:36] ejegg: If you've merged to a production branch you should deploy now now now (or revert). [18:58:49] Also, this conversation needs to be in -operations. [19:44:12] (03PS9) 10Vaughn Walters: jjb: [selenium-daily-beta-RelatedArticles] Drop daily tests, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) [19:44:13] (03CR) 10Vaughn Walters: "done, annnnd updated global gitignore for future git add * fails" [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) (owner: 10Vaughn Walters) [19:44:58] 06Project-Admins: Create project tag for User-SomeRandomDeveloper - https://phabricator.wikimedia.org/T426896 (10SomeRandomDeveloper) 03NEW [19:46:14] (03CR) 10Jforrester: [C:03+2] jjb: [selenium-daily-beta-RelatedArticles] Drop daily tests, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) (owner: 10Vaughn Walters) [19:48:14] (03Merged) 10jenkins-bot: jjb: [selenium-daily-beta-RelatedArticles] Drop daily tests, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1290040 (https://phabricator.wikimedia.org/T423958) (owner: 10Vaughn Walters) [20:33:24] 06Project-Admins: Create project tag for User-SomeRandomDeveloper - https://phabricator.wikimedia.org/T426896#11942610 (10Peachey88) 05Open→03Resolved a:03Peachey88 Project #User-SomeRandomDeveloper has been created :) [20:35:11] 06Project-Admins: Create project tag for User-SomeRandomDeveloper - https://phabricator.wikimedia.org/T426896#11942619 (10SomeRandomDeveloper) Thank you! [21:11:35] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T426822#11942741 (10bd808) p:05Triage→03High a:03ssingh This is currently blocking an unblock request for Beta Clu... [21:17:06] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://phabricator.wikimedia.org/T426822#11942764 (10bd808) >>! In T426822#11942742, @bd808 wrote: > This is currently blocking an unblock request for Be... [21:24:44] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:33:29] is there anyone around who can help with releases-jenkins.w.o? [21:42:55] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cache-upload08 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:47:05] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Quibble, and 2 others: Implement a retry policy for network errors in CI - https://phabricator.wikimedia.org/T424990#11942886 (10A_smart_kitten) →14Duplicate dup:03T420865 [21:47:06] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#11942883 (10A_smart_kitten) [21:47:10] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Quibble, and 2 others: Implement a retry policy for network errors in CI - https://phabricator.wikimedia.org/T424990#11942888 (10A_smart_kitten) (boldly closing as a duplicate instead, as IMO it is a bit confusing to have this task as 'r... [21:57:49] (thanks for flagging urandom, responding out of band.) [23:52:13] 10Phabricator (2026-05-19), 07CSS: Remove blur when expanding hidden bot comments - https://phabricator.wikimedia.org/T425283#11943297 (10matmarex) Thanks, this is just what I had in mind!