[00:05:03] maintenance-disconnect-full-disks build 744021 integration-agent-docker-1051 (/: 36%, /srv: 95%, /var/lib/docker: 36%): OFFLINE due to disk space [00:10:03] maintenance-disconnect-full-disks build 744022 integration-agent-docker-1051 (/: 36%, /srv: 47%, /var/lib/docker: 35%): RECOVERY disk space OK [02:07:55] 10GitLab (Project Migration), 10tool-wscontest: Move wscontest source code to GitLab - https://phabricator.wikimedia.org/T324410#11257752 (10Samwilson) [05:40:50] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774#11257980 (10ABran-WMF) Thanks @Scott_French @hashar for handling this. Making mod_qos LogOnly to workaround this issue was definitely the good ca... [05:40:54] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774#11257981 (10ABran-WMF) [05:40:55] 06Release-Engineering-Team, 06collaboration-services, 07Security: ProbeDown - gerrit1003:443 - https://phabricator.wikimedia.org/T406403#11257982 (10ABran-WMF) [05:49:03] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774#11258014 (10ABran-WMF) the attack pattern that led to some Gerrit unavailability in the recent past is harder to shed with... [07:14:57] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406815 (10jhsoby-WMNO) 03NEW [07:15:01] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406816 (10jhsoby-WMNO) 03NEW [07:15:52] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406816#11258203 (10jhsoby-WMNO) →14Duplicate dup:03T406815 [07:15:55] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406815#11258206 (10jhsoby-WMNO) [07:18:39] GitLab needs a short maintenance break in 30 minutes [07:50:00] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406815#11258320 (10Urbanecm) 05Open→03Resolved a:03Urbanecm Both Phabricator accounts are now disabled. Note I'm also happy to add you to #acl_userdisable if you're intereste... [07:57:34] GitLab maintenance finished [08:22:38] 10GitLab: Remove gitlab-ce and gitlab-runner bullseye package and component - https://phabricator.wikimedia.org/T406823 (10Jelto) 03NEW [08:30:18] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup, 13Patch-For-Review: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11258497 (10Jelto) Today is another scheduled maintenance for Gerrit: T387833. So th... [08:38:11] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Angelkay - https://phabricator.wikimedia.org/T406776#11258532 (10Aklapper) 05Open→03Stalled > Developer account / GitLab username: Angelkay No such account exists in GitLab, and your Phabricator profile at ht... [08:42:42] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11258575 (10Jelto) [08:48:22] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11258584 (10Jelto) 05Open→03Resolved Great, then I'll resolve this task. I opened {T406824} as a... [09:00:28] (03CR) 10Bartosz Wójtowicz: "Thank you Antoine! The changes look good to me, I think we can deploy <3" [integration/config] - 10https://gerrit.wikimedia.org/r/1187752 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [09:15:18] (03PS1) 10Hashar: utils: mw-requires: Flow requires Echo [integration/config] - 10https://gerrit.wikimedia.org/r/1194888 [09:15:33] (03CR) 10Hashar: [C:03+2] utils: mw-requires: Flow requires Echo [integration/config] - 10https://gerrit.wikimedia.org/r/1194888 (owner: 10Hashar) [09:16:59] (03Merged) 10jenkins-bot: utils: mw-requires: Flow requires Echo [integration/config] - 10https://gerrit.wikimedia.org/r/1194888 (owner: 10Hashar) [09:44:10] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406815#11258819 (10Shadabgdg) a:05Urbanecm→03Shadabgdg [09:50:03] maintenance-disconnect-full-disks build 744138 integration-agent-docker-1049 (/: 36%, /srv: 97%, /var/lib/docker: 32%): OFFLINE due to disk space [09:51:57] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/BlueSpicePageTemplates] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194897 (https://phabricator.wikimedia.org/T406376) [09:51:59] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/NSFileRepo] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194898 (https://phabricator.wikimedia.org/T406376) [09:52:01] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/JsonConfig] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194899 (https://phabricator.wikimedia.org/T406376) [09:52:03] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/MathSearch] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194900 (https://phabricator.wikimedia.org/T406376) [09:52:05] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194901 (https://phabricator.wikimedia.org/T406376) [09:55:03] maintenance-disconnect-full-disks build 744139 integration-agent-docker-1049 (/: 36%, /srv: 21%, /var/lib/docker: 31%): RECOVERY disk space OK [10:15:34] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194901 (https://phabricator.wikimedia.org/T406376) [10:15:34] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/BlueSpicePageTemplates] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194897 (https://phabricator.wikimedia.org/T406376) [10:15:34] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/JsonConfig] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194899 (https://phabricator.wikimedia.org/T406376) [10:15:34] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/MathSearch] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194900 (https://phabricator.wikimedia.org/T406376) [10:15:35] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/NSFileRepo] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194898 (https://phabricator.wikimedia.org/T406376) [10:35:21] 10Phabricator, 10WMNO-General: Disable Phabricator accounts Nakeisha_WMNO and anette-WMNO - https://phabricator.wikimedia.org/T406815#11259043 (10jhsoby-WMNO) a:05Shadabgdg→03Urbanecm (@Shadabgdg Why did you claim a task that someone else had already completed?) @Urbanecm: Thanks! No need to add me to... [10:36:09] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194901 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:36:12] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/BlueSpicePageTemplates] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194897 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:36:15] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/JsonConfig] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194899 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:36:19] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/MathSearch] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194900 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:36:22] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/NSFileRepo] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194898 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:37:39] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194901 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:37:49] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/BlueSpicePageTemplates] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194897 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:37:51] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/JsonConfig] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194899 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:38:05] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/MathSearch] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194900 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:38:07] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/NSFileRepo] add VisualEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1194898 (https://phabricator.wikimedia.org/T406376) (owner: 10Hashar) [10:53:07] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T393334#11259140 (10Acrion-dev) Thank you for approving my GitLab Wikimedia account and for the clarification! Yes, a Phabricator project is definitely still wanted. The zelph project has recently received Wikimedia Rapid Grant... [11:17:04] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, and 3 others: ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11259261 (10kostajh) 05Open→03Resolved [11:17:11] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10ConfirmEdit (CAPTCHA extension), 06Trust and Safety Product Team, and 3 others: ConfirmEdit hCaptcha: Get QUnit tests to run in CI - https://phabricator.wikimedia.org/T406231#11259263 (10kostajh) [12:13:01] Project beta-code-update-eqiad build #569161: 04FAILURE in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569161/ [12:18:03] Project mediawiki-core-doxygen build #14225: 04FAILURE in 0.46 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14225/ [12:23:01] Project beta-code-update-eqiad build #569162: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569162/ [12:33:01] Project beta-code-update-eqiad build #569163: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569163/ [12:35:36] 10Gerrit, 06Release-Engineering-Team, 06Data-Engineering: Reduce size of analytics/superset/deploy.git Gerrit repo - https://phabricator.wikimedia.org/T406856 (10hashar) 03NEW [12:43:01] Project beta-code-update-eqiad build #569164: 04STILL FAILING in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569164/ [12:50:45] 10Phabricator, 07Performance Issue: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586#11259630 (10A_smart_kitten) >>! In T108586#7072497, @Bugreporter wrote: > Although I already said before at https://www.mediawiki.org/wiki/Topic:V9yllpjfjpennzcl, let's... [12:55:12] Yippee, build fixed! [12:55:12] Project beta-code-update-eqiad build #569165: 09FIXED in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/569165/ [13:06:40] 10Gerrit, 06collaboration-services: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11259697 (10hashar) Today we had scheduled a switch over of Gerrit from `gerrit1003` to `gerrit2003`. O... [13:18:06] 10Gerrit, 06collaboration-services: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11259763 (10ABran-WMF) [13:31:59] Yippee, build fixed! [13:32:00] Project mediawiki-core-doxygen build #14226: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/14226/ [13:35:21] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting membership in the WMF-NDA group for OKryva-WMF - https://phabricator.wikimedia.org/T406835#11259915 (10Aklapper) 05Open→03Resolved a:03Aklapper I added you. [14:02:00] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Requesting membership in the WMF-NDA group for OKryva-WMF - https://phabricator.wikimedia.org/T406835#11260038 (10Aklapper) @OKryva-WMF: Bonus points for linking your LDAP account via https://phabricator.wikimedia.org/settings/panel/external/ so it... [14:02:34] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Content-Transform-Team, and 20 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11260039 (10Mvolz) [14:02:56] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Content-Transform-Team, and 20 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11260040 (10Mvolz) [14:03:44] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Content-Transform-Team, and 20 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11260042 (10Mvolz) [14:04:01] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Grant NDA access to CLo (WMF) - https://phabricator.wikimedia.org/T406867#11260055 (10Aklapper) 05Open→03Resolved a:03Aklapper I added you. (In theory there was a workflow to add new staff to the WMF-NDA Phabricator project when new staff was... [14:14:21] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T405678#11260113 (10jnuche) 05Open→03Resolved Things have been quiet for a while. Resolving [14:19:21] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Data-Engineering, and 21 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11260122 (10MSantos) [14:19:44] 06Release-Engineering-Team, 10ChangeProp, 10Citoid, 06Data-Engineering, and 21 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#11260124 (10MSantos) [14:20:30] (03CR) 10Hashar: [C:04-1] "-1 to ack the remarks mentioned by Tacsipacsi. I'll take them in account and amend, but not today :]" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1194221 (owner: 10Hashar) [14:23:45] 10Gerrit, 06Release-Engineering-Team, 06Data-Engineering: Reduce size of analytics/superset/deploy.git Gerrit repo - https://phabricator.wikimedia.org/T406856#11260143 (10ABran-WMF) [15:10:48] (03open) 10dancy: staging.tfvars: buildkitd_tag = "wmf-v0.25.1" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/510 (https://phabricator.wikimedia.org/T406772) [15:10:50] (03update) 10dancy: staging.tfvars: buildkitd_tag = "wmf-v0.25.1" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/510 (https://phabricator.wikimedia.org/T406772) [15:12:01] (03merge) 10dancy: staging.tfvars: buildkitd_tag = "wmf-v0.25.1" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/510 (https://phabricator.wikimedia.org/T406772) [15:13:13] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [15:17:49] !log Upgraded gitlab-cloud-runners staging buildkit to v0.25.1 (T406772) [15:17:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:52] T406772: buildkit v0.25.1 released - https://phabricator.wikimedia.org/T406772 [15:24:34] (03open) 10dancy: prod.tfvars: buildkitd_tag = "wmf-v0.25.1" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/511 (https://phabricator.wikimedia.org/T406772) [15:24:36] (03update) 10dancy: prod.tfvars: buildkitd_tag = "wmf-v0.25.1" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/511 (https://phabricator.wikimedia.org/T406772) [15:25:42] (03merge) 10dancy: prod.tfvars: buildkitd_tag = "wmf-v0.25.1" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/511 (https://phabricator.wikimedia.org/T406772) [15:32:22] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup, 13Patch-For-Review: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11260702 (10jcrespo) Is it ok to deploy now? [15:35:04] !log Upgraded gitlab-cloud-runners production buildkit to v0.25.1 (T406772) [15:35:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:35:11] T406772: buildkit v0.25.1 released - https://phabricator.wikimedia.org/T406772 [15:36:32] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.25.1 released - https://phabricator.wikimedia.org/T406772#11260723 (10dancy) p:05Triage→03Medium a:03dancy [15:37:22] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.25.0 released - https://phabricator.wikimedia.org/T406144#11260729 (10dancy) 05In progress→03Declined Superseded by T406772 [15:54:06] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup, 13Patch-For-Review: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11260857 (10Jelto) >>! In T403946#11260702, @jcrespo wrote: > Is it ok to deploy now... [15:55:00] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [15:55:01] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [15:55:05] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [15:57:44] (03update) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [15:58:20] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [15:59:11] (03approved) 10dancy: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 (owner: 10dduvall) [16:00:17] (03merge) 10dduvall: variables: `kokkuri variables env` command [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/145 [16:29:56] (03open) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:29:56] (03update) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:29:57] (03update) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:31:02] (03update) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:31:07] (03update) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:31:09] (03update) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:36:41] (03approved) 10dancy: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 (owner: 10dduvall) [16:39:40] (03merge) 10dduvall: image: Use variable for image cache repo [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/146 [16:43:43] 10Beta-Cluster-Infrastructure, 06Traffic: Copy the Traffic team on alerts for deployment-cache* hosts - https://phabricator.wikimedia.org/T406650#11261114 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [16:51:58] (03open) 10dduvall: version: 2.10.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/147 [16:53:17] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [16:55:12] (03merge) 10dduvall: version: 2.10.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/147 [16:59:23] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [17:02:02] (03open) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:04:33] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:04:33] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:04:36] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:05:27] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:05:30] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:05:33] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:06:03] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:06:04] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:06:07] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [17:16:24] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [17:25:09] (03open) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [17:25:10] (03update) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [17:25:11] (03update) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [17:30:17] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [17:52:52] 06Project-Admins, 06LPL Hypothesis, 10Projects-Cleanup, 10Wikipedia Subscriptions: Archive Wikipedia Subscriptions - https://phabricator.wikimedia.org/T404331#11261465 (10SBisson) [17:53:30] 06Project-Admins, 06LPL Hypothesis, 10Projects-Cleanup: Archive Wikipedia Subscriptions - https://phabricator.wikimedia.org/T404331#11261468 (10SBisson) [17:53:39] 06Project-Admins, 06LPL Hypothesis, 10Projects-Cleanup: Archive Wikipedia Subscriptions - https://phabricator.wikimedia.org/T404331#11261469 (10SBisson) 05Open→03Resolved a:03SBisson [18:08:48] (03update) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [18:08:49] (03update) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [18:08:51] (03update) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [18:21:36] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T405677#11261637 (10Jdforrester-WMF) [18:21:38] 06Release-Engineering-Team (Doing 😎), 06Abstract Wikipedia team, 06DBA, 10Wikifunctions, and 2 others: Wikimedia\Rdbms\DBQueryError: Error 1146: Table '[wiktionary].wikifunctionsclient_usage' doesn't exist - https://phabricator.wikimedia.org/T406185#11261636 (10Jdforrester-WMF) [19:20:01] 10GitLab, 10bacula, 06collaboration-services, 10Data-Persistence-Backup: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons - https://phabricator.wikimedia.org/T403946#11261826 (10jcrespo) 05Open→03Resolved This should be done now, with probably some tuning required... [19:58:39] (03approved) 10dancy: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 (owner: 10dduvall) [20:05:03] maintenance-disconnect-full-disks build 744261 integration-agent-docker-1064 (/: 25%, /srv: 98%, /var/lib/docker: 39%): OFFLINE due to disk space [20:07:39] (03merge) 10dduvall: variables: Do not quote values [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/149 [20:10:03] maintenance-disconnect-full-disks build 744262 integration-agent-docker-1064 (/: 25%, /srv: 40%, /var/lib/docker: 38%): RECOVERY disk space OK [20:14:12] (03open) 10dduvall: version: 2.10.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/150 [20:17:57] (03merge) 10dduvall: version: 2.10.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/150 [20:18:58] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [20:21:32] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [20:21:34] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [20:29:11] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Gerrit connection troubles and ERR_CONNECTION_RESET - https://phabricator.wikimedia.org/T406774#11262136 (10Dzahn) I deployed Arnaud's change above that re-enables QoS (not logging only anymore) but with the previously... [20:29:54] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025) [20:32:52] (03update) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [20:36:30] (03approved) 10dancy: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 (owner: 10dduvall) [20:36:35] (03update) 10dancy: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 (owner: 10dduvall) [20:37:02] (03merge) 10dduvall: images: Use `KOKKURI_JWT_PROJECT_PATH` over `CI_PROJECT_PATH` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/148 [20:37:03] (03PS1) 10Subramanya Sastry: Remove fonts package no longer available in ubuntu [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1195046 [21:54:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11262577 (10Dzahn) ` zuul1001:/etc/apache2/sites-enabled] $ telnet localhost 9000 Trying ::1... Connected to localh... [22:03:48] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11262588 (10Dzahn) mostly works..just: ` curl --resolve 'zuul.wikimedia.org:80:10.64.32.104' http://zuul.wikimedia... [22:10:36] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11262619 (10Dzahn) [22:58:40] (03update) 10dancy: backport.py: Revise relevant dependency selection algorithm [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1010 (https://phabricator.wikimedia.org/T365146 https://phabricator.wikimedia.org/T371611 https://phabricator.wikimedia.org/T388025)