[00:12:19] Re: blubber/java image questions about, I think we're good..b-tullis pointed me to another java 17 image and it seems to be working! https://gitlab.wikimedia.org/repos/data-engineering/opensearch/-/merge_requests/4 [02:00:58] (03open) 10bd808: kubernetes: Add service account for nodepool [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/38 (https://phabricator.wikimedia.org/T398367) [02:01:05] (03update) 10bd808: kubernetes: Add service account for nodepool [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/38 (https://phabricator.wikimedia.org/T398367) [02:07:23] (03update) 10bd808: kubernetes: Add service account for nodepool [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/38 (https://phabricator.wikimedia.org/T398367) [03:30:16] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06MediaWiki-Platform-Team, 10PoolCounter, and 2 others: Migrate poolcounter Jenkins job to Blubber - https://phabricator.wikimedia.org/T399075#11022565 (10Krinkle) [03:38:49] 06Release-Engineering-Team, 10Wikidata, 10MW-1.45-notes (1.45.0-wmf.11; 2025-07-22): mwext-node20-rundoc jobs failing with `Please configure a github access token` - https://phabricator.wikimedia.org/T400036#11022573 (10Krinkle) What information is this Wikibase doc script fetching from GitHub? If it is fixe... [07:33:55] 10GitLab, 06collaboration-services: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121 (10ABran-WMF) 03NEW [07:34:27] 10GitLab, 06collaboration-services: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11022704 (10ABran-WMF) 05Open→03In progress p:05Triage→03High [07:34:49] 10GitLab, 06collaboration-services: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11022707 (10ABran-WMF) [07:44:44] (03open) 10arnaudb: Preparing gitlab replica switchover [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/74 (https://phabricator.wikimedia.org/T400121) [07:46:12] (03approved) 10jelto: Preparing gitlab replica switchover [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/74 (https://phabricator.wikimedia.org/T400121) (owner: 10arnaudb) [07:48:51] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11022736 (10ABran-WMF) [07:50:12] (03PS1) 10Jeena Huneidi: Add daily trigger for wikilambda selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) [07:51:43] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11022741 (10ABran-WMF) [07:52:16] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11022742 (10ABran-WMF) [07:52:32] (03CR) 10Jeena Huneidi: "I tried to make a template to be used by the existing job and a new daily job but I wasn't sure how." [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [09:02:54] 10Beta-Cluster-Infrastructure, 06Moderator-Tools-Team, 10PageTriage, 07Chinese-Sites: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378#11022892 (10Stang) 05Open→03Resolved a:03Stang PageTriage has been installed on zhwiki's beta cluster, and some... [09:13:06] 10Continuous-Integration-Infrastructure (Zuul upgrade): Setup RBAC for Cloud VPS hosted kubernetes cluster - https://phabricator.wikimedia.org/T398365#11022900 (10jnuche) 05Open→03Resolved [09:25:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision user in Cloud VPS hosted Kubernetes cluster for use by nodepool - https://phabricator.wikimedia.org/T398367#11022934 (10jnuche) a:05jnuche→03None [09:26:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision user in Cloud VPS hosted Kubernetes cluster for use by nodepool - https://phabricator.wikimedia.org/T398367#11022936 (10jnuche) Removed myself as I'm not working on this at the moment [09:42:34] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11023007 (10Jelto) [09:42:41] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11023008 (10Jelto) [09:54:08] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: upgrade gitlab hosts to bookworm - https://phabricator.wikimedia.org/T399306#11023051 (10Jelto) The estimated downtime for the switchover [without packages](https://gitlab.wikimedia.org/groups/repos/-/packages) is around 60-120 minutes.... [10:51:42] 06Release-Engineering-Team, 10Wikidata, 10MW-1.45-notes (1.45.0-wmf.11; 2025-07-22): mwext-node20-rundoc jobs failing with `Please configure a github access token` - https://phabricator.wikimedia.org/T400036#11023295 (10Lucas_Werkmeister_WMDE) >>! In T400036#11021170, @bd808 wrote: > You could move the activ... [11:12:56] (03approved) 10jnuche: kubernetes: Add service account for nodepool [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/38 (https://phabricator.wikimedia.org/T398367) (owner: 10bd808) [14:20:47] (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Also trigger daily [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [14:29:18] (03CR) 10Jforrester: "I think this should work. I've deployed this as-is; let's see if it works and then tweak?" [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [14:29:39] (03PS3) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Also trigger daily [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [14:29:55] (03CR) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Also trigger daily (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [15:03:01] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11024495 (10PerfektesChaos) You might find a smarter solution for entire ##toolforge.org​## and... [15:12:16] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11024560 (10bd808) >>! In T393487#11024495, @PerfektesChaos wrote: > I am steward of dewiki@BET... [15:27:33] (03merge) 10bd808: kubernetes: Add service account for nodepool [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/38 (https://phabricator.wikimedia.org/T398367) [15:34:42] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11024688 (10PerfektesChaos) Blocking by IP is a good idea for three days while exposed to a DDo... [15:40:42] (03update) 10dancy: Add scap update-next-patches command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/906 (https://phabricator.wikimedia.org/T295925) [15:40:48] (03update) 10dancy: Add scap update-next-patches command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/906 (https://phabricator.wikimedia.org/T295925) [16:00:32] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11024836 (10bd808) @PerfektesChaos I understand your arguments and your frustration. We do not... [16:36:12] (03CR) 10Jforrester: [C:03+2] "| Build periodically > Schedule: @daily" [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [16:37:28] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/WikimediaMessages] Add MetricsPlatform dep [integration/config] - 10https://gerrit.wikimedia.org/r/1170598 (owner: 10Catrope) [16:37:38] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end] Also trigger daily [integration/config] - 10https://gerrit.wikimedia.org/r/1171538 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [16:41:20] (03PS4) 10Catrope: zuul: Add MetricsPlatform as a Phan dependency of WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/1170598 [16:42:04] (03CR) 10Catrope: "After reevaluating I've reverted this to the earlier version that just adds a Phan dependency. We should be able to address the optional s" [integration/config] - 10https://gerrit.wikimedia.org/r/1170598 (owner: 10Catrope) [16:43:15] (03update) 10dancy: Add scap update-next-patches command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/906 (https://phabricator.wikimedia.org/T295925) [16:45:19] (03open) 10bd808: rbac: Remove duplicate "pod" grant from nodepool cluster role [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/39 [16:45:33] (03update) 10bd808: rbac: Remove duplicate "pod" grant from nodepool cluster role [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/39 [16:51:20] (03update) 10bd808: rbac: Remove duplicate "pod" grant from nodepool cluster role [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/39 [16:51:41] (03PS5) 10Jforrester: Zuul: [mediawiki/extensions/WikimediaMessages] Add MetricsPlatform phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1170598 (owner: 10Catrope) [16:51:45] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikimediaMessages] Add MetricsPlatform phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1170598 (owner: 10Catrope) [16:53:14] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaMessages] Add MetricsPlatform phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1170598 (owner: 10Catrope) [16:53:31] !log Zuul: [mediawiki/extensions/WikimediaMessages] Add MetricsPlatform phan dep [16:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:58:12] (03open) 10dancy: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) [16:58:13] (03update) 10dancy: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) [16:58:34] (03update) 10dancy: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) [17:55:32] (03approved) 10thcipriani: Add scap update-next-patches command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/906 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [17:55:42] (03merge) 10thcipriani: Add scap update-next-patches command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/906 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [18:11:21] 10GitLab (Upstream pit of despair 🕳️), 06Release-Engineering-Team: Gitlab "Review changes" button not working - https://phabricator.wikimedia.org/T400091#11025497 (10brennen) [18:52:30] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jrmuizel - https://phabricator.wikimedia.org/T400204 (10Jrmuizel) 03NEW [19:48:54] !log phab/phorge: adding "other assignee" visibility to more forms (T396905) [19:48:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:48:57] T396905: Support "other assignee" field for tasks with "Task" type - https://phabricator.wikimedia.org/T396905 [19:55:24] 10Phabricator, 06Release-Engineering-Team: Support "other assignee" field for tasks with "Task" type - https://phabricator.wikimedia.org/T396905#11025836 (10brennen) a:03brennen This came up in WMF Slack discussion with @egardner today, which prompted me to search and notice this task. For consistency, I mad... [20:30:50] (03open) 10dancy: scap update-next-patches: Add --dry-run flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/910 (https://phabricator.wikimedia.org/T295925) [20:30:52] (03update) 10dancy: scap update-next-patches: Add --dry-run flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/910 (https://phabricator.wikimedia.org/T295925) [21:12:03] (03update) 10thcipriani: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [21:40:48] (03approved) 10thcipriani: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [21:42:46] (03update) 10dancy: scap update-next-patches: Add --dry-run flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/910 (https://phabricator.wikimedia.org/T295925) [21:44:32] (03open) 10dancy: logger.warn -> logger.warning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/911 [21:44:36] (03update) 10dancy: logger.warn -> logger.warning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/911 [21:45:02] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#11026077 (10dduvall) >>! In T398873#10986500, @dancy wrote: > If CI fails (which happens about 10% of the time), we're left with an unusable `wm... [21:48:11] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#11026078 (10dancy) @dduvall I'd like to see `MediaWiki branch and publish WMF single-version image` changed so that instead of destroying and re... [21:49:05] (03merge) 10dancy: logger.warn -> logger.warning [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/911 [21:49:56] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#11026080 (10dduvall) >>! In T398873#11026078, @dancy wrote: > @dduvall I'd like to see `MediaWiki branch and publish WMF single-version image` c... [21:52:59] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#11026091 (10dancy) >>! In T398873#11026080, @dduvall wrote: >>>! In T398873#11026078, @dancy wrote: >> @dduvall I'd like to see `MediaWiki branc... [21:53:41] (03update) 10dancy: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) [21:55:07] (03merge) 10dancy: apply-patches: Skip .dropped patches [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/909 (https://phabricator.wikimedia.org/T295925) [21:57:12] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#11026100 (10dduvall) >>! In T398873#11026091, @dancy wrote: > Yes, but in the meantime `scap prep next` would continue to work correctly (assumi... [22:37:35] we are having trouble with CI - I don't think we can log in to try anything [22:37:35] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-bookworm/1440/console [22:37:36] See these lines [22:37:36] 10:00:00 Failed to read: /src/wikimedia/fundraising/crm/civicrm-buildkit/build/wmf.sh [22:37:36] 10:00:00 [22:37:36] 10:00:00 The metadata for wmf appears incomplete. This suggests that the original download failed. Some possible resolutions: [22:37:37] 10:00:00 [22:37:37] 10:00:00 1. Delete the failed build (/src/wikimedia/fundraising/crm/civicrm-buildkit/build/wmf) and recreate it, or... [22:37:38] 10:00:00 2. Manually generate metadata (/src/wikimedia/fundraising/crm/civicrm-buildkit/build/wmf.sh), or... [22:37:38] 10:00:01 3. Run "civibuild install" with the "--force" option. [22:37:39] affects all our test runs [22:53:44] - OK - am investigating possible theory that is within our control...