[00:01:34] (03open) 10swfrench: Switch all deployments to publish-83 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/171 (https://phabricator.wikimedia.org/T405955) [00:01:50] (03open) 10swfrench: Draft: make-container-image: stop building publish-81 [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/220 (https://phabricator.wikimedia.org/T405955) [00:02:58] (03update) 10swfrench: Switch all deployments to publish-83 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/171 (https://phabricator.wikimedia.org/T405955) [03:10:44] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708#11394760 (10thcipriani) >>! In T306708#11394508, @Aklapper wrote: >> - When did user login last with 2fa (I don't know, offhand ho... [09:42:08] (03PS1) 10Hashar: Zuul: disable recursion when it is a noop [integration/config] - 10https://gerrit.wikimedia.org/r/1208288 (https://phabricator.wikimedia.org/T389998) [10:04:23] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: scap publish-docs job failing - https://phabricator.wikimedia.org/T410680#11395138 (10jnuche) a:03jnuche It seems the problem is a combination of two things: * After the releases instance switchover to codfw, the Jenkins service is being restarted ev... [10:13:33] (03open) 10jnuche: jenkinsrelapi: ensure token is persisted for user [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/108 (https://phabricator.wikimedia.org/T410680) [10:14:20] (03merge) 10jnuche: jenkinsrelapi: ensure token is persisted for user [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/108 (https://phabricator.wikimedia.org/T410680) [10:17:56] 10Continuous-Integration-Config, 10Math, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Allow control over which extra extensions are installed (Math REL1_43 jobs exceed 60min timeout) - https://phabricator.wikimedia.org/T389998#11395163 (10hashar) For the migration I think I will do it in small st... [10:29:51] (03open) 10jnuche: jenkins.yaml: add missing method receiver [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/109 [10:30:11] (03merge) 10jnuche: jenkins.yaml: add missing method receiver [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/109 [10:33:07] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: scap publish-docs job failing - https://phabricator.wikimedia.org/T410680#11395207 (10jnuche) 05Open→03Resolved Configuration seems to survive across restarts now and docpub pipelines are healthy again: https://gitlab.wikimedia.org/repos/releng/s... [10:35:03] maintenance-disconnect-full-disks build 756530 integration-agent-docker-1056 (/: 36%, /srv: 95%, /var/lib/docker: 39%): OFFLINE due to disk space [10:40:03] maintenance-disconnect-full-disks build 756531 integration-agent-docker-1056 (/: 36%, /srv: 57%, /var/lib/docker: 38%): RECOVERY disk space OK [11:41:15] 10Continuous-Integration-Config, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team (Kanban Board): Block the merging of patches to MediaWiki core that would break CentralAuth tests - https://phabricator.wikimedia.org/T406582#11395354 (10OWresch-WMF) [12:16:33] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Doing 😎), 07Essential-Work: scap publish-docs job failing - https://phabricator.wikimedia.org/T410680#11395558 (10jnuche) [12:39:32] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729 (10jnuche) 03NEW [12:50:25] 06Project-Admins, 10VerySmallGLAM: Create project subproject tag for #verysmallglam - https://phabricator.wikimedia.org/T410504#11395725 (10Olea) >>! In T410504#11394189, @A_smart_kitten wrote: > Another possible approach would be to create a new {nav icon=briefcase,name= crm4wb} project independently of the... [13:56:07] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T408274#11395907 (10jnuche) [14:36:40] (03PS2) 10Hashar: Zuul: disable recursion when it is a noop [integration/config] - 10https://gerrit.wikimedia.org/r/1208288 (https://phabricator.wikimedia.org/T389998) [14:51:31] 06Release-Engineering-Team (Radar), 07Epic, 07Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683#11396138 (10zeljkofilipin) Looks like there's no progress on this in years. I'm removing #tp0 projects from the task. Let us k... [15:43:54] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Doing 😎), 07Essential-Work: scap publish-docs job failing - https://phabricator.wikimedia.org/T410680#11396425 (10dancy) Thanks @jnuche ! [15:44:29] (03approved) 10dancy: Switch all deployments to publish-83 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/171 (https://phabricator.wikimedia.org/T405955) (owner: 10swfrench) [15:48:08] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.26.2 released - https://phabricator.wikimedia.org/T410049#11396453 (10dancy) 05Open→03In progress a:03dduvall [15:49:26] 10Continuous-Integration-Infrastructure: Puppet failure on integration-agent-pkgbuilder-1003.integration.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T410653#11396472 (10dancy) 05Open→03Resolved a:03dancy I didn't get an email notification this morning so I checked the host and it looks l... [16:05:02] !log Upgrading gitlab-cloud-runners staging k8s from 1.31.9-do.2 to 1.31.9-do.5 [16:05:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:14:44] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Support "other assignee" field for tasks with "Task" type - https://phabricator.wikimedia.org/T396905#11396548 (10A_smart_kitten) >>! In T396905#11047604, @brennen wrote: > If anyone would like the field visible anywhere it's currently not, pl... [16:17:07] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729#11396550 (10Dzahn) a:03Dzahn [16:27:48] !log Upgraded gitlab-cloud-runners staging k8s from 1.31.9-do.2 to 1.31.9-do.5 [16:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:42:46] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729#11396636 (10Dzahn) I don't think this is particularly new since the switch. It's just "any server that gets a jenkins service also... [16:47:18] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729#11396641 (10Dzahn) wmf_auto_restart_* services/timers are added by infra foundations to ensure security patches can be auto-applied... [16:50:20] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729#11396647 (10Dzahn) I think what changed here is the time when this happened. Because that is randomized. ` # This define can be us... [16:51:44] (03open) 10dancy: staging/staging.tfvars: kubernetes_version = "1.31.9-do.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/535 [16:51:47] (03update) 10dancy: staging/staging.tfvars: kubernetes_version = "1.31.9-do.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/535 [16:52:01] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729#11396650 (10Dzahn) [16:52:49] (03merge) 10dancy: staging/staging.tfvars: kubernetes_version = "1.31.9-do.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/535 [16:54:59] (03open) 10dancy: staging/staging.tfvars: Adjust buildkitd requests/limits [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/536 [16:55:02] (03update) 10dancy: staging/staging.tfvars: Adjust buildkitd requests/limits [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/536 [16:56:05] (03merge) 10dancy: staging/staging.tfvars: Adjust buildkitd requests/limits [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/536 [17:00:32] !log Upgraded gitlab-cloud-runners prod k8s from 1.31.9-do.2 to 1.31.9-do.5 [17:00:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:00:45] (03open) 10dancy: production/prod.tfvars: kubernetes_version = "1.31.9-do.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/537 [17:00:48] (03update) 10dancy: production/prod.tfvars: kubernetes_version = "1.31.9-do.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/537 [17:01:27] (03merge) 10dancy: production/prod.tfvars: kubernetes_version = "1.31.9-do.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/537 [18:22:40] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708#11396900 (10Aklapper) >>! In T306708#11394760, @thcipriani wrote: >> We do not know because https://we.phorge.it/T16310 > > Interes... [18:32:51] 06Release-Engineering-Team (Radar), 07Epic, 07Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683#11396930 (10bd808) This sounds a bit like the idea of "critical user journey tests" that I have talked with both @Lferreira an... [18:37:37] 06Release-Engineering-Team (Radar), 07Epic, 07Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683#11396941 (10SLong-WMF) It is! And one of the draft hypotheses I'm considering for the next fiscal year involves creating and m... [19:38:03] 06Project-Admins: Create project tag for QS (Quality Services) Test Automation - https://phabricator.wikimedia.org/T410776 (10SLong-WMF) 03NEW [20:00:59] to debug this, is it okay if I make some changes by hand to the files in mwdeploy2002 and run the script? [20:01:00] https://phabricator.wikimedia.org/T410738 [20:52:33] 10Scap, 10MediaWiki-Internationalization, 10MediaWiki-Special-pages: Special pages that don't exist should not be autolocalized - https://phabricator.wikimedia.org/T404519#11397518 (10Pppery) [21:29:23] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 13Patch-For-Review: releases host: Jenkins instance periodically restarted - https://phabricator.wikimedia.org/T410729#11397576 (10Dzahn) I stopped the service restarts for now until further discussion. ` [releases2003:~] $ sys... [21:31:06] 06Release-Engineering-Team, 06collaboration-services: Out of date directories on releases.wikimedia.org - https://phabricator.wikimedia.org/T410418#11397580 (10Dzahn) Turns out we have to dig deeper into the sub directories in some cases. Example: we got feedback there are actually current Android app release... [21:41:49] 06Release-Engineering-Team, 06collaboration-services: Out of date directories on releases.wikimedia.org - https://phabricator.wikimedia.org/T410418#11397619 (10Dzahn) Here is the actual data. All files hosted on releases, sorted by last modification date: {P85448} [22:37:08] 10GitLab (Integrations), 10Tool-gitlab-bot, 10Wikibugs: Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940#11397755 (10bd808) >>! In T362940#9784524, @bd808 wrote: > It is on a timer, but that timer is on @thcipriani's laptop today. Let's... [22:49:51] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [22:52:16] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [22:53:12] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [22:57:25] (03merge) 10dancy: Switch all deployments to publish-83 images [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/171 (https://phabricator.wikimedia.org/T405955) (owner: 10swfrench) [22:58:21] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [23:09:23] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [23:10:47] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [23:18:17] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps) [23:27:46] (03update) 10thcipriani: Automate creation of MediaWiki release pages [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/117 (https://phabricator.wikimedia.org/T347207) (owner: 10sandeeps)