[00:08:17] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11269980 (10xcollazo) @dancy https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/151 indeed fixed the issue. Thanks! [00:09:31] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11269985 (10xcollazo) A question: there was no bump of CI attributes from our side when this broke on a Friday. Can we add a feature so that `kokkuri`, from a CI standpoint, is versioned, and thus d... [00:16:51] 10GitLab (Project Migration), 06Community-Tech, 10SVG Translate Tool: Migrate SVG Translate from GitHub to GitLab - https://phabricator.wikimedia.org/T402505#11269990 (10Samwilson) [05:33:56] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T407169#11270174 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF same pattern as the one I was trying to fix in https://phabricator.wikimedia.org/T406403#11253645 β†’ https://grafana.wikimedia.org/goto/ENn... [05:34:18] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T407169#11270180 (10ABran-WMF) [05:34:29] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T407169#11270182 (10ABran-WMF) [08:04:47] 10Continuous-Integration-Config, 10Math, 06MediaWiki-Platform-Team (Radar), 05MW-1.45-notes (1.45.0-wmf.23; 2025-10-14), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11270479 (10hashar) [08:20:11] 10Gerrit, 06Release-Engineering-Team, 06Data-Engineering: Reduce size of analytics/superset/deploy.git Gerrit repo - https://phabricator.wikimedia.org/T406856#11270575 (10brouberol) I think we can delete it without any issue. @BTullis do you concur? [08:29:37] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11270646 (10jcrespo) Gilab + Gerrit backups were now close to 70 TB in total, which was starting to ca... [09:05:05] Project beta-code-update-eqiad build #569862: 04FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569862/ [09:13:01] Project beta-code-update-eqiad build #569863: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569863/ [09:18:02] Project mediawiki-core-doxygen build #14348: 04FAILURE in 0.22 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14348/ [09:23:01] Project beta-code-update-eqiad build #569864: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569864/ [09:35:15] Yippee, build fixed! [09:35:15] Project beta-code-update-eqiad build #569865: 09FIXED in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569865/ [09:43:01] Project beta-code-update-eqiad build #569866: 04FAILURE in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569866/ [09:47:22] 10GitLab, 10Growth Team Maintenance, 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration, 07Technical-Debt: Archive the Community Configuration Example repository in GitLab - https://phabricator.wikimedia.org/T407192#11270973 (10Michael) [09:47:53] 10GitLab, 10Growth Team Maintenance, 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration, and 2 others: Archive the Community Configuration Example repository in GitLab - https://phabricator.wikimedia.org/T407192#11270974 (10Michael) [09:51:42] 10GitLab (Project Migration), 06Community-Tech, 10CopyPatrol: Migrate CopyPatrol repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395401#11270985 (10A_smart_kitten) >>! In T395401#11247984, @bd808 wrote: >>>! In T395401#11232164, @Samwilson wrote: >> @brennen @bd808 Is there a preference for V... [09:53:01] Project beta-code-update-eqiad build #569867: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569867/ [10:03:01] Project beta-code-update-eqiad build #569868: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569868/ [10:15:16] Yippee, build fixed! [10:15:16] Project beta-code-update-eqiad build #569869: 09FIXED in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569869/ [10:15:25] 10GitLab, 10Growth Team Maintenance, 06Growth-Team, 10MediaWiki-extensions-CommunityConfiguration, and 2 others: Archive the Community Configuration Example repository in GitLab - https://phabricator.wikimedia.org/T407192#11271072 (10Urbanecm_WMF) Re tagging #gitlab: We can self-service this, given we're a... [10:23:01] Project beta-code-update-eqiad build #569870: 04FAILURE in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569870/ [10:29:07] Yippee, build fixed! [10:29:07] Project mediawiki-core-doxygen build #14349: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14349/ [10:35:13] Yippee, build fixed! [10:35:13] Project beta-code-update-eqiad build #569871: 09FIXED in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569871/ [10:43:26] FIXED as Gerrit is back up [10:44:13] I see gerrit. I don't yet see my changes getting tested on push [10:44:23] does Zuul need a kick? [10:44:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [10:44:36] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T407205 (10wmcs-alerts) 03NEW [10:50:27] seems better now - thanks! [11:12:16] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Archive #code-review-workgroup - https://phabricator.wikimedia.org/T393394#11271377 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Done. [11:18:31] 06Project-Admins, 10Projects-Cleanup, 10Wikidata: Consider archiving #Data_Import - https://phabricator.wikimedia.org/T404310#11271398 (10Aklapper) @conny-kawohl_WMDE: Hi, would you know and could answer the last comment? Thanks! [13:02:37] PROBLEM - gerrit process on gerrit2003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit/review_site/bin/gerrit.war daemon -d /var/lib/gerrit/review_site https://wikitech.wikimedia.org/wiki/Gerrit [13:07:27] 10Phabricator, 07SecTeam-Processed: Change the dropdown in security ticket dropdown to not include WMF Product and WMF Technology as two separate departments - https://phabricator.wikimedia.org/T384243#11271999 (10Aklapper) I checked again that aliases in the codebase are only supported for Priority field valu... [13:09:01] Heh RelEng, is it possible nowadays to rename Gerrit accounts? I really wish my Gerrit account can be renamed for discoverability reasons. [13:09:48] Years ago, I created several abouts (I think 2 of them) but finally sticked to a particular one but my username moved quite a bit and I wish a rename could happen on Gerrit to reflect my current handle. [13:12:47] https://wikitech.wikimedia.org/wiki/SRE/LDAP/Renaming_users [13:14:26] so I guess T171417 shouldn't be in open state :-/ [13:14:27] T171417: Request rename of "Alangi derick" to "Alangi Derick" on wikitech/LDAP/Gerrit - https://phabricator.wikimedia.org/T171417 [13:15:02] 10Gerrit: Access issue for golson-wmf - https://phabricator.wikimedia.org/T378187#11272062 (10Aklapper) @GOlson-WMF Is this still an issue nowadays? [13:18:01] andre, I think that task is still accurate, just a little out of date. I'll rename it. [13:18:10] thanks [13:22:04] 10Gerrit: Searching for users by display name doesn't seem to work - https://phabricator.wikimedia.org/T407232 (10Reedy) 03NEW [13:27:23] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07LDAP: Request rename of "d3r1ck01" to "xSavitar" on wikitech/LDAP/Gerrit - https://phabricator.wikimedia.org/T171417#11272187 (10xSavitar) [13:27:38] (03open) 10aklapper: Adjust numerous custom search query length restrictions [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/105 (https://phabricator.wikimedia.org/T407003) [13:28:49] (03merge) 10aklapper: Adjust numerous custom search query length restrictions [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/105 (https://phabricator.wikimedia.org/T407003) [13:29:39] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Project Typeahead queries with less than four letters return random unrelated results - https://phabricator.wikimedia.org/T407003#11272222 (10Aklapper) [14:16:48] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T405679#11272440 (10hashar) Looks good so far, there was a single type of error for `ParserOutput::getImages`Β being deprecated. Filed as... [14:30:14] 10GitLab (CI & Job Runners), 06collaboration-services, 10Toolforge, 06cloud-services-team (FY2025/26-Q1): tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11272529 (10fnegri) I pushed a new Trixie-based image: `docker-registry.svc.toolforge.org/tofu-provisioning:202... [14:35:00] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phabricator/Phorge 2025-10-14 - https://phabricator.wikimedia.org/T407244 (10brennen) 03NEW [14:35:51] (03open) 10brennen: update submodules for 2025-10-14 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/83 (https://phabricator.wikimedia.org/T407244) [14:43:15] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272631 (10dancy) >>! In T407020#11269985, @xcollazo wrote: > A question: there was no bump of CI attributes from our side when this broke on a Friday. Can we add a feature so that `kokkuri`, from... [14:56:17] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272653 (10dancy) Btw the way, the current version of kokkuri is 2.10.1. [15:01:54] (03merge) 10brennen: update submodules for 2025-10-14 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/83 (https://phabricator.wikimedia.org/T407244) [15:02:38] 10Phabricator, 07SecTeam-Processed: Change the dropdown in security ticket dropdown to not include WMF Product and WMF Technology as two separate departments - https://phabricator.wikimedia.org/T384243#11272677 (10sbassett) >>! In T384243#11271999, @Aklapper wrote: > It seems that I could rename one of the two... [15:04:30] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272697 (10xcollazo) Great, will change our CI to adopt `2.10.1`. CC @amastilovic [15:09:54] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Unknown Configuration Option "gitlab.api_key" in Phab Local Config - https://phabricator.wikimedia.org/T407111#11272718 (10Aklapper) [15:18:28] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272737 (10xcollazo) >>! In T407020#11272653, @dancy wrote: > Btw the way, the current version of kokkuri is 2.10.1. Looks like 2.10.1 doesn't include the `kokkuri!151` fix. Can we do a new `kokku... [15:21:34] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272754 (10amastilovic) Oh TIL that we can use kokkuri as a GitLab component! Thumbs up. [15:21:35] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272755 (10dancy) >>! In T407020#11272737, @xcollazo wrote: >>>! In T407020#11272653, @dancy wrote: >> Btw the way, the current version of kokkuri is 2.10.1. > > Looks like 2.10.1 doesn't include... [15:22:29] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/ContentStabilization] Add PDFCreator as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196102 [15:24:19] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272769 (10dancy) Release 2.10.3 has been prepared. [15:25:23] 06Release-Engineering-Team: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272771 (10dancy) >>! In T407020#11272769, @dancy wrote: > Release 2.10.3 has been prepared. Maybe not. Still fumbling with it... [15:26:35] 10Continuous-Integration-Config, 06Reader Experience Team: Add BetaFeatures and MetricsPlatform as CI dependencies for ReadingLists - https://phabricator.wikimedia.org/T407249 (10aude) 03NEW [15:28:42] (03PS1) 10Aude: Add BetaFeatures and MetricsPlatform for ReadingLists [integration/config] - 10https://gerrit.wikimedia.org/r/1196108 (https://phabricator.wikimedia.org/T407249) [15:30:50] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add BlueSpiceFoundation as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196110 [15:32:45] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add BlueSpiceFoundation as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196110 (owner: 10Hslater) [15:33:36] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team, 07Performance Issue, 07Upstream: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#11272818 (10Aklapper) Oh well: * `PhabricatorRepositoryG... [15:34:11] (03open) 10dancy: includes/kokkuri.yaml: bump KOKKURI_IMAGE to 2.10.4 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/152 (https://phabricator.wikimedia.org/T407020) [15:34:12] (03update) 10dancy: includes/kokkuri.yaml: bump KOKKURI_IMAGE to 2.10.4 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/152 (https://phabricator.wikimedia.org/T407020) [15:36:05] (03PS2) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196110 [15:38:12] (03merge) 10dancy: includes/kokkuri.yaml: bump KOKKURI_IMAGE to 2.10.4 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/152 (https://phabricator.wikimedia.org/T407020) [15:38:31] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): Change Maniphest AdvancedSearch default for "Group by" to "None" and for "Order by" to "Updated" - https://phabricator.wikimedia.org/T404843#11272841 (10Aklapper) a:03Aklapper [15:39:18] 10Phabricator, 10Tool-enderbot-dev: Workboard request for #tool-enderbot-dev - https://phabricator.wikimedia.org/T407251 (10Anderlaxe) 03NEW [15:40:05] 06Release-Engineering-Team, 13Patch-For-Review: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11272863 (10dancy) OK. Kokkuri 2.10.4 should be working now. Lemme know how it goes. [15:43:07] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phabricator/Phorge 2025-10-14 - https://phabricator.wikimedia.org/T407244#11272904 (10Aklapper) 05Openβ†’03Resolved Today's deployment included: * Fix for {T407003} * {T404843} * {T399111} (small performa... [15:43:45] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures and MetricsPlatform deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196108 (https://phabricator.wikimedia.org/T407249) (owner: 10Aude) [15:44:04] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): PhabricatorMetaMTAMailableFunctionDatasource does not need to query PhabricatorProjectDatasource - https://phabricator.wikimedia.org/T399111#11272913 (10Aklapper) 05Openβ†’03Resolved This got deployed today. [15:44:55] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): Change Maniphest AdvancedSearch default for "Group by" to "None" and for "Order by" to "Updated" - https://phabricator.wikimedia.org/T404843#11272919 (10Aklapper) 05Openβ†’03Resolved This got deployed today. Verified by going to https://ph... [15:45:08] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures and MetricsPlatform deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196108 (https://phabricator.wikimedia.org/T407249) (owner: 10Aude) [15:46:10] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196108 (https://phabricator.wikimedia.org/T407249) (owner: 10Aude) [15:46:45] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): Project Typeahead queries with less than four letters return random unrelated results - https://phabricator.wikimedia.org/T407003#11272924 (10Aklapper) [15:46:47] 10Phabricator (Search), 06Release-Engineering-Team (Seen): Autocomplete project proposals in Maniphest search form don't always offer "any" and "not" options - https://phabricator.wikimedia.org/T360701#11272925 (10Aklapper) [15:47:55] 06Project-Admins, 10Projects-Cleanup, 10Wikidata: Consider archiving #Data_Import - https://phabricator.wikimedia.org/T404310#11272950 (10Pppery) 05Openβ†’03Resolved a:05conny-kawohl_WMDEβ†’03Leif_WMDE Looks like @Leif_WMDE did the honors [15:48:28] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): Autocomplete project proposals in Maniphest search form don't always offer "any" and "not" options - https://phabricator.wikimedia.org/T360701#11272954 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper This specific issue got resolved in tod... [15:49:43] 10Phabricator (2025-10-14), 06Release-Engineering-Team (Doing 😎): Project Typeahead queries with less than four letters return random unrelated results - https://phabricator.wikimedia.org/T407003#11272970 (10Aklapper) 05Openβ†’03Resolved This got deployed today. Verified the fix by going to https://phabr... [15:54:08] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196108 (https://phabricator.wikimedia.org/T407249) (owner: 10Aude) [15:54:09] 06Project-Admins, 10Tool-enderbot-dev: Workboard request for #tool-enderbot-dev - https://phabricator.wikimedia.org/T407251#11272987 (10A_smart_kitten) 05Openβ†’03Resolved a:03A_smart_kitten done :) [15:54:47] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196110 (owner: 10Hslater) [15:55:14] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ContentStabilization] Add PDFCreator as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196102 (owner: 10Hslater) [15:55:45] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps [integration/config] - 10https://gerrit.wikimedia.org/r/1196108 (https://phabricator.wikimedia.org/T407249) (owner: 10Aude) [15:56:02] !log Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps, for T407249 [15:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:56:04] T407249: Add BetaFeatures and MetricsPlatform as CI dependencies for ReadingLists - https://phabricator.wikimedia.org/T407249 [15:56:34] 10Continuous-Integration-Config, 06Reader Experience Team, 13Patch-For-Review: Add BetaFeatures and MetricsPlatform as CI dependencies for ReadingLists - https://phabricator.wikimedia.org/T407249#11273010 (10Jdforrester-WMF) This is now deployed. Please shout if there are any issues! [15:56:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196110 (owner: 10Hslater) [15:56:48] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ContentStabilization] Add PDFCreator as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196102 (owner: 10Hslater) [15:56:52] !log Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add dependency on BlueSpiceFoundation [15:56:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:09] !log Zuul: [mediawiki/extensions/ContentStabilization] Add PDFCreator as dependency [15:57:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:58:27] rebooting contint servers - for kernel upgrades [15:58:36] Ack. [15:58:37] 10Continuous-Integration-Config, 06Reader Experience Team, 13Patch-For-Review: Add BetaFeatures and MetricsPlatform as CI dependencies for ReadingLists - https://phabricator.wikimedia.org/T407249#11273030 (10aude) @Jdforrester-WMF Thank you for approving and deploying this! [15:58:39] expect like a minute of downtime [15:58:50] for integration.wm.org [16:00:15] we had 496 days of uptime [16:00:28] Alas. [16:00:43] connecting to console now to watch it [16:01:00] 10Gerrit: Searching for users by display name doesn't seem to work - https://phabricator.wikimedia.org/T407232#11273048 (10hashar) The suggestions are provided by the [[ https://gerrit.wikimedia.org/r/Documentation/rest-api-accounts.html#query-account | Query Account API ]] with theΒ `suggest`Β option. The query i... [16:01:26] and it's back up ..phew :) [16:01:42] it always feels a little risky after such long uptimes [16:02:13] and integration website is up.. and bot back.. good [16:03:48] mutante: that wiped the Zuul CI work queue ( https://integration.wikimedia.org/zuul/ ) [16:03:59] for the record Moritz and I usually do them early in the morning :] [16:04:33] so some changes that were on the fly would not get CI test results [16:04:36] it does not seem like that is possible to avoid and it was less than once a year [16:04:52] ticket is https://phabricator.wikimedia.org/T407110 [16:05:30] it was also downtimed and logged [16:05:46] and outside active deployment windows [16:06:06] doesn't change the fact the state got wiped [16:06:23] anyway it doesnot matter, there was barely any change in the queue at the time at least :] [16:06:50] exactly [16:06:52] my point is that it is nice to at least check https://integration.wikimedia.org/zuul/ to gauge how many changes / approval end up affected by vanishing [16:07:42] anyway I am off for diner, thanks for the kernel upgrade [16:07:47] and the answer is: not many [16:09:04] (thanks for kicking my patch, h.ashar! I totally missed the logging, heh) [16:15:41] (03CR) 10Jforrester: [C:03+1] Zuul: [mediawiki/extensions/GrowthExperiments] add EventBus, EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/1195368 (owner: 10Hashar) [16:18:37] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceWikiFarm] Add BlueSpiceDiscovery dep [integration/config] - 10https://gerrit.wikimedia.org/r/1196120 [16:31:34] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.25.1 released - https://phabricator.wikimedia.org/T406772#11273224 (10dancy) 05Openβ†’03Resolved [16:32:44] 10GitLab, 06collaboration-services: Remove gitlab-ce and gitlab-runner bullseye package and component - https://phabricator.wikimedia.org/T406823#11273235 (10Dzahn) Thanks! [16:39:37] 10GitLab (CI & Job Runners), 06collaboration-services, 10Toolforge, 06cloud-services-team (FY2025/26-Q1): tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11273294 (10fnegri) I found something interesting: adding `--net=host` fixes the issue, regardless of which doc... [16:39:48] 10Gerrit, 06Release-Engineering-Team, 06Data-Engineering: Reduce size of analytics/superset/deploy.git Gerrit repo - https://phabricator.wikimedia.org/T406856#11273296 (10BTullis) Agreed. Please feel free to delete it from Gerrit. We no longer need it. Thanks @hashar for checking. The new version in GitLab... [16:50:03] maintenance-disconnect-full-disks build 745661 integration-agent-docker-1042 (/: 36%, /srv: 97%, /var/lib/docker: 27%): OFFLINE due to disk space [16:53:45] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [16:53:59] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [16:55:03] maintenance-disconnect-full-disks build 745662 integration-agent-docker-1042 (/: 36%, /srv: 81%, /var/lib/docker: 26%): RECOVERY disk space OK [17:05:03] maintenance-disconnect-full-disks build 745664 integration-agent-docker-1047 (/: 36%, /srv: 97%, /var/lib/docker: 27%): OFFLINE due to disk space [17:06:04] 06Release-Engineering-Team, 13Patch-For-Review: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11273489 (10xcollazo) >>! In T407020#11272863, @dancy wrote: > OK. Kokkuri 2.10.4 should be working now. Lemme know how it goes. Build succeeded. Thanks for taking care of th... [17:07:18] 06Release-Engineering-Team, 13Patch-For-Review: Kokkuri build failure on airflow-dags repo - https://phabricator.wikimedia.org/T407020#11273504 (10xcollazo) 05Openβ†’03Resolved a:03dancy [17:10:03] maintenance-disconnect-full-disks build 745665 integration-agent-docker-1047 (/: 36%, /srv: 36%, /var/lib/docker: 26%): RECOVERY disk space OK [17:36:09] !log Unblock 166.48.128.0/20 (T407258) [17:36:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:48:41] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11273800 (10bd808) 05In progressβ†’03Resolved With https://gitlab.wikimedia.org/ladsgroup/Phabricator-maintenance-bot/-/merge_requests/31 merged and deployed,... [17:49:52] (03PS1) 10Daimona Eaytoy: Zuul: [mediawiki/extensions/CampaignEvents] Add new dependecies [integration/config] - 10https://gerrit.wikimedia.org/r/1196137 (https://phabricator.wikimedia.org/T404995) [17:49:58] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11273804 (10ssingh) Thanks for working on this! We will try our best to follow up on our end in making sure that Puppet is not broken on the cache hosts in Beta. [18:33:58] (03CR) 10Michael Große: "I haven't looked at this code in depth in a while. Are there GrowthExperiments tests that we are currently skipping because there is no Ev" [integration/config] - 10https://gerrit.wikimedia.org/r/1195368 (owner: 10Hashar) [18:49:55] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [19:48:41] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [19:50:20] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CampaignEvents] Add new dependecies [integration/config] - 10https://gerrit.wikimedia.org/r/1196137 (https://phabricator.wikimedia.org/T404995) (owner: 10Daimona Eaytoy) [19:51:56] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CampaignEvents] Add new dependecies [integration/config] - 10https://gerrit.wikimedia.org/r/1196137 (https://phabricator.wikimedia.org/T404995) (owner: 10Daimona Eaytoy) [19:55:37] (03CR) 10Hashar: [C:03+2] "@daimona.wiki@gmail.com I have reloaded Zuul" [integration/config] - 10https://gerrit.wikimedia.org/r/1196137 (https://phabricator.wikimedia.org/T404995) (owner: 10Daimona Eaytoy) [19:56:39] (03CR) 10Daimona Eaytoy: "Thank you :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1196137 (https://phabricator.wikimedia.org/T404995) (owner: 10Daimona Eaytoy) [20:20:03] maintenance-disconnect-full-disks build 745703 integration-agent-docker-1060 (/: 25%, /srv: 98%, /var/lib/docker: 27%): OFFLINE due to disk space [20:30:03] maintenance-disconnect-full-disks build 745705 integration-agent-docker-1060 (/: 25%, /srv: 100%, /var/lib/docker: 27%): still OFFLINE due to disk space [20:37:28] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [20:45:52] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [20:54:43] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [20:55:03] maintenance-disconnect-full-disks build 745710 integration-agent-docker-1060 (/: 25%, /srv: 66%, /var/lib/docker: 27%): RECOVERY disk space OK [21:09:09] 10GitLab (CI & Job Runners), 06collaboration-services, 13Patch-For-Review: registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11274485 (10cmassaro) We're having more issues today that look related: https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-eval... [21:25:34] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Unknown Configuration Option "gitlab.api_key" in Phab Local Config - https://phabricator.wikimedia.org/T407111#11274524 (10brennen) 05Openβ†’03Resolved a:03brennen As I understand it: `gitlab.api_key` in the `local.json` created by each sc... [21:30:59] (03PS1) 10Daimona Eaytoy: Zuul: [mediawiki/extensions/CampaignEvents] Add CentralAuth dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1196180 (https://phabricator.wikimedia.org/T404995) [21:31:27] (03CR) 10Daimona Eaytoy: Zuul: [mediawiki/extensions/CampaignEvents] Add new dependecies (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1196137 (https://phabricator.wikimedia.org/T404995) (owner: 10Daimona Eaytoy) [21:35:42] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: failed to configure registry cache exporter: invalid reference format error with new kokkuri - https://phabricator.wikimedia.org/T407294 (10dancy) 03NEW [21:35:59] 10GitLab (CI & Job Runners), 06collaboration-services, 13Patch-For-Review: registry.cloud.releng.team returning 503s - https://phabricator.wikimedia.org/T406733#11274564 (10dancy) >>! In T406733#11274485, @cmassaro wrote: > We're having more issues today that look related: https://gitlab.wikimedia.org/repos/... [21:41:01] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: failed to configure registry cache exporter: invalid reference format error with new kokkuri - https://phabricator.wikimedia.org/T407294#11274588 (10dancy) [21:47:14] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: failed to configure registry cache exporter: invalid reference format error with new kokkuri - https://phabricator.wikimedia.org/T407294#11274604 (10dancy) [21:56:44] (03PS1) 10Aude: Revert "Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps" [integration/config] - 10https://gerrit.wikimedia.org/r/1196184 [21:57:48] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: failed to configure registry cache exporter: invalid reference format error with new kokkuri - https://phabricator.wikimedia.org/T407294#11274636 (10dancy) @cmassaro I'm not sure what the exact problem with the newer version of Kokkur... [21:58:07] (03CR) 10CI reject: [V:04-1] Revert "Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps" [integration/config] - 10https://gerrit.wikimedia.org/r/1196184 (owner: 10Aude) [22:08:28] (03CR) 10Aude: [C:04-1] "I don't think this is the issue." [integration/config] - 10https://gerrit.wikimedia.org/r/1196184 (owner: 10Aude) [22:24:32] (03Abandoned) 10Aude: Revert "Zuul: [mediawiki/extensions/ReadingLists] Add BetaFeatures, MetricsPlatform deps" [integration/config] - 10https://gerrit.wikimedia.org/r/1196184 (owner: 10Aude) [22:54:30] 10Continuous-Integration-Config, 06Reader Experience Team (Sprint 7 [Q2 Oct 7-Oct 21 '25]), 07Unplanned-Sprint-Work: Add BetaFeatures and MetricsPlatform as CI dependencies for ReadingLists - https://phabricator.wikimedia.org/T407249#11274730 (10Jdlrobson-WMF) [22:55:36] 10Continuous-Integration-Config, 06Reader Experience Team (Sprint 7 [Q2 Oct 7-Oct 21 '25]), 07Unplanned-Sprint-Work: Add BetaFeatures and MetricsPlatform as CI dependencies for ReadingLists - https://phabricator.wikimedia.org/T407249#11274740 (10Jdlrobson-WMF) p:05Triageβ†’03Low [23:20:26] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07LDAP: Request rename of "d3r1ck01" to "xSavitar" on wikitech/LDAP/Gerrit - https://phabricator.wikimedia.org/T171417#11274776 (10bd808) 05Openβ†’03Declined >>! From ** https://wikitech.wikimedia.org/wiki/SRE/LDAP/Renaming_users **: > We do not rename dev...