[00:01:41] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:41] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [03:27:12] Project mwcore-phpunit-coverage-master build #3745: 04STILL FAILING in 27 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/3745/ [07:45:38] grr [07:45:45] moar things broken [07:48:58] PHP Fatal error: Cannot declare class BlockUsers, because the name is already in use in /workspace/src/maintenance/blockUsers.php on line 27 [07:48:59] fun [08:21:09] ah that is https://phabricator.wikimedia.org/T371188 ;) [08:50:46] 10Phabricator: Disable personal Herald rules H124, H213? - https://phabricator.wikimedia.org/T371219 (10Aklapper) 03NEW p:05Triage→03Low [08:58:09] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10021889 (10Jelto) I deleted a lot of very old GitLab logs (mostly migration and upgrade logs from 2023), freeing up some disk space. Now the disk usage fluctuates between 60% and 80%. I hav... [09:05:25] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222 (10Jelto) 03NEW [09:06:13] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10021914 (10Jelto) [09:06:15] 10GitLab (Infrastructure), 06collaboration-services: Increase disk size for GitLab test instance - https://phabricator.wikimedia.org/T369837#10021917 (10Jelto) →14Duplicate dup:03T371066 [09:47:58] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222#10022121 (10Jelto) p:05Triage→03Medium a:03Jelto Manually triggering a backup with ` sudo /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE=yes SKIP=builds,art... [10:14:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 06collaboration-services: Decommission integration.mediawiki.org - https://phabricator.wikimedia.org/T361250#10022241 (10hashar) Thank you for the cleanup @Dzahn ! [10:47:28] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10022456 (10Volans) [11:04:52] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10022490 (10Clement_Goubert) [12:05:27] 10Continuous-Integration-Infrastructure: Add WMDE staff working on development of Wikibase software - https://phabricator.wikimedia.org/T370766#10022613 (10hashar) 05Open→03Resolved a:03hashar I have added everyone as members of the `integration` project. Any user can sudo as `jenkins-deploy` which sho... [12:13:28] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10022661 (10Jgiannelos) [12:14:18] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Rebuild or delete deployment-docker-mobileapps01 - https://phabricator.wikimedia.org/T369915#10022657 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos Yeah i just deleted it. [12:29:17] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#10022695 (10Jgiannelos) Hi, i think this was an attempt to quickly ass... [12:36:44] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222#10022726 (10Jelto) The backup works again with reduced `max_storage_concurrency`, but it still generated a lot of load, making the host almost unusable and triggering probe down alert... [12:54:03] 10Continuous-Integration-Infrastructure: Add WMDE staff working on development of Wikibase software - https://phabricator.wikimedia.org/T370766#10022772 (10Lucas_Werkmeister_WMDE) SSH and sudo seems to work :) (I only `echo`ed this time) `lang=shell-session lucaswerkmeister-wmde@integration-castor05:~$ sudo... [12:59:48] 10Beta-Cluster-Infrastructure, 10Math: Make MathML default rendering in Labs - https://phabricator.wikimedia.org/T371254 (10Physikerwelt) 03NEW [13:04:50] 10GitLab, 06collaboration-services: Backups are failing on the GitLab test instance - https://phabricator.wikimedia.org/T371222#10022827 (10Jelto) a:05Jelto→03eoghan After scaling the gitlab test instance to `g4.cores4.ram8.disk20` backups are working fine again and don't slow down the system significantly... [13:06:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10Castor: Castor should aggregate wmf/* cache - https://phabricator.wikimedia.org/T303836#10022831 (10hashar) 05Open→03Declined [13:10:49] 10Scap: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255 (10Lucas_Werkmeister_WMDE) 03NEW [13:10:55] 10Continuous-Integration-Infrastructure: Add WMDE staff working on development of Wikibase software - https://phabricator.wikimedia.org/T370766#10022845 (10hashar) @Lucas_Werkmeister_WMDE thank you for the verification! [13:19:07] 10Scap: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10022861 (10Lucas_Werkmeister_WMDE) According to @akosiaris in IRC: > it's passing --recursive to the wrong git subcommand, it should be passing --recursive to git submodule foreach [13:24:13] 10Scap: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10022881 (10Lucas_Werkmeister_WMDE) Hm, but the code looks to me like it’s supposed to do the right thing? `lang=py def list_submodules_paths_urls(repo, args): """Return a list of the paths and URLs o... [13:26:24] 10Scap: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10022892 (10Lucas_Werkmeister_WMDE) Okay, it’s Git behaving weirdly (IMHO): `lang=shell-session lucaswerkmeister-wmde@deploy1003 /srv/mediawiki-staging $ git submodule foreach --recursive 'echo a && echo... [13:29:42] (03open) 10jnuche: backport: pass argument to correct git command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/391 (https://phabricator.wikimedia.org/T371255) [13:30:54] (03update) 10jnuche: backport: pass argument to correct git command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/391 (https://phabricator.wikimedia.org/T371255) [13:30:59] (03update) 10jnuche: backport: pass argument to correct git command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/391 (https://phabricator.wikimedia.org/T371255) [13:32:47] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10022915 (10Lucas_Werkmeister_WMDE) (Unfortunately I can’t easily repeat the above test on deploy1002 because `/src/mediawiki-staging` no longer exists as a git repo, and I couldn’t f... [13:38:31] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10022928 (10Lucas_Werkmeister_WMDE) Scratch that, I was just accidentally on `mwdebug1002` instead of `deploy1002` 😅 we can see Git’s old behavior: `lang=shell-session lucaswerkmeist... [13:43:04] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10022937 (10Lucas_Werkmeister_WMDE) Git used to interpret the `--recursive` wherever it was in the argv: `lang=shell-session lucaswerkmeister-wmde@deploy1002 /srv/mediawiki-staging $... [13:44:12] (03approved) 10thcipriani: backport: pass argument to correct git command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/391 (https://phabricator.wikimedia.org/T371255) (owner: 10jnuche) [13:46:39] (03merge) 10jnuche: backport: pass argument to correct git command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/391 (https://phabricator.wikimedia.org/T371255) [13:47:09] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: package_builder python-all conflicts with base::standard_packages python2.7 removal - https://phabricator.wikimedia.org/T370337#10022948 (10hashar) 05Open→03Resolved a:03hashar I have solved... [13:47:57] (03open) 10jnuche: Release 4.94.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/392 [13:49:48] (03merge) 10jnuche: Release 4.94.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/392 [13:51:26] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Rebuild integration-agent-pkgbuilder-1001 and integration-agent-pkgbuilder-1002 to get rid of Debian Buster - https://phabricator.wikimedia.org/T360786#10022971 (10hashar) > Should we delete the old ones to close this out? integration-ag... [13:55:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10observability, 06SRE, 13Patch-For-Review: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089#10022995 (10hashar) I must have declined this as part of a task triage since I usually leave a comment when... [13:57:14] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10023015 (10thcipriani) I think the update to from git v2.20.1 to v.2.30.2 explains this. [[https://github.com/git/git/commit/a282f5a90613de5f4b449749ea8738ac20872271#diff-7bd0801cbd4... [13:57:26] (03PS1) 10Ssingh: Revert "Archive operations/debs/trafficserver" [integration/config] - 10https://gerrit.wikimedia.org/r/1057881 [14:03:05] (03CR) 10Hashar: "The project got archived from Zuul CI and Gerrit cause it has been migrated to GitLab at https://gitlab.wikimedia.org/repos/sre/trafficser" [integration/config] - 10https://gerrit.wikimedia.org/r/1057881 (owner: 10Ssingh) [14:05:36] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10023089 (10Lucas_Werkmeister_WMDE) Seems to be working again with scap 4.94.0 ([SAL](https://sal.toolforge.org/production?q="4.94.0")), thanks all! [14:10:41] (03CR) 10Ssingh: "That is indeed true. However, we want to migrate this back to Gerrit as the Gitlab workflow is not working for us. I want to wait for more" [integration/config] - 10https://gerrit.wikimedia.org/r/1057881 (owner: 10Ssingh) [14:13:35] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10023148 (10Lucas_Werkmeister_WMDE) Though this feels concerning, partway through the `scap backport`: ` 14:11:07 Started sync-masters... [14:17:19] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261 (10Lucas_Werkmeister_WMDE) 03NEW [14:18:31] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261#10023170 (10Lucas_Werkmeister_WMDE) Note that this currently shows up in deployments as errors in the `sync-masters` step, see T371255#10023148. [14:18:47] 10Scap, 13Patch-For-Review: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10023177 (10Lucas_Werkmeister_WMDE) >>! In T371255#10023148, @Lucas_Werkmeister_WMDE wrote: > Though this feels concerning, partway through the `scap backport`: Made a separate task... [14:20:38] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261#10023188 (10Lucas_Werkmeister_WMDE) `lang=shell-session lucaswerkmeister-wmde@deploy1002 ~ $ dpkg -L scap dpkg-query: package 'scap' is not installed Use dpkg --contents (= dpkg-deb --contents) to list archive fil... [14:29:11] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261#10023242 (10Lucas_Werkmeister_WMDE) Aha, scap is now trying to use Python 3.9 (the Python 3.7 dir in the venv is just a little leftover stub apparently): `lang=shell-session lucaswerkmeister-wmde@deploy1002 ~ $ l... [14:42:13] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261#10023297 (10Lucas_Werkmeister_WMDE) >>! In T371261#10023170, @Lucas_Werkmeister_WMDE wrote: > Note that this currently shows up in deployments as errors in the `sync-masters` step, see T371255#10023148. It also m... [14:42:18] (03CR) 10BCornwall: [C:03+1] Revert "Archive operations/debs/trafficserver" [integration/config] - 10https://gerrit.wikimedia.org/r/1057881 (owner: 10Ssingh) [14:43:11] (03CR) 10Ssingh: "@hashar@free.fr: ready for your review." [integration/config] - 10https://gerrit.wikimedia.org/r/1057881 (owner: 10Ssingh) [14:45:58] 10Scap: scap backport broken on deploy1003 (bullseye, Git 2.30) - https://phabricator.wikimedia.org/T371255#10023316 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03jnuche [14:52:27] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#10023394 (10hashar) [15:04:51] 06Release-Engineering-Team, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10023460 (10Milimetric) Just for the record, we met and discussed @Joe's proposal (this task'... [15:08:15] 10Continuous-Integration-Config: Gate-and-submit-1_39 fails for CampaignEvents because it tries to install WikimediaCampaignEvents (which does not have a 1_39 branch) - https://phabricator.wikimedia.org/T369279#10023466 (10hashar) Since `WikimediaCampaignEvents` does not have a `REL1_39` branch, Quibble falls ba... [15:26:14] 10GitLab (Integrations), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: Container image reports in debmonitor are broken - https://phabricator.wikimedia.org/T348876#10023573 (10elukey) To keep archives happy - the code is fully deployed on build200... [15:27:08] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10023539 (10Clement_Goubert) 05In progress→03Resolved [15:28:36] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10023584 (10LSobanski) a:03Dzahn [15:28:38] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10023586 (10LSobanski) p:05Triage→03Medium [15:30:35] Project mwcore-phpunit-coverage-master build #3746: 04STILL FAILING in 30 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/3746/ [15:32:51] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Update SQL output for Phabricator WMF QLS report mails - https://phabricator.wikimedia.org/T370947#10023605 (10LSobanski) [16:02:48] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261#10023717 (10akosiaris) Yes this is Python virtualenv related. I 've tried some simple fixes already but didn't work However: * I am gonna be decommissioning deploy1002 this week * I 'll be reimaging deploy2002 (... [16:14:00] 10Scap: scap broken on deploy1002 / deploy2002 (buster) - https://phabricator.wikimedia.org/T371261#10023775 (10Lucas_Werkmeister_WMDE) That works for me; deployers should just be aware that these messages will show up until then (two yellow errors in sync-masters, and a red nonzero exit at the very end). [16:45:31] 10Scap, 06serviceops: Reimage deploy2002 as bullseye - https://phabricator.wikimedia.org/T371282 (10akosiaris) 03NEW [16:48:55] (03update) 10lwatson: Codex 1.10.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/27 [16:56:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#10024079 (10hashar) [17:14:06] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#10024160 (10jijiki) [17:14:35] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#10024163 (10hashar) [18:15:34] (03approved) 10egardner: Codex 1.10.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/27 (owner: 10lwatson) [18:15:47] (03merge) 10egardner: Codex 1.10.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/27 (owner: 10lwatson) [20:23:09] Project beta-update-databases-eqiad build #77807: 04FAILURE in 3 min 8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/77807/ [21:32:21] Yippee, build fixed! [21:32:21] Project beta-update-databases-eqiad build #77808: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/77808/ [22:08:49] 10Gerrit, 06collaboration-services, 07git-lfs: Gerrit LFS objects lack an automatic sync to gerrit replicas - https://phabricator.wikimedia.org/T257741#10025627 (10Dzahn) 05Resolved→03Open a:05eoghan→03Dzahn I have to follow-up on this after merging https://gerrit.wikimedia.org/r/c/operations/puppet/... [22:09:09] 10GitLab, 06collaboration-services: gitlab/devtools: send logs to a new disk - https://phabricator.wikimedia.org/T371066#10025636 (10Dzahn) [22:23:39] 10Beta-Cluster-Infrastructure, 10Add-Link, 10Growth-Team (FY2024-25 Q1 Sprint 2): refreshLinkRecommendation script fails in Beta cluster with FileNotFoundError - https://phabricator.wikimedia.org/T370792#10025672 (10Urbanecm_WMF) p:05Triage→03Low [22:23:41] 10Beta-Cluster-Infrastructure, 10Add-Link, 10Growth-Team (FY2024-25 Q1 Sprint 2): refreshLinkRecommendation script fails in Beta cluster with FileNotFoundError - https://phabricator.wikimedia.org/T370792#10025657 (10Urbanecm_WMF) a:03Urbanecm_WMF [23:01:56] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T366962#10025733 (10thcipriani) p:05Triage→03Medium a:03jnuche [23:02:25] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T366963#10025738 (10thcipriani) p:05Triage→03Medium a:03jeena [23:02:55] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T366964#10025743 (10thcipriani) p:05Triage→03Medium a:03Aklapper [23:03:33] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T366965#10025748 (10thcipriani) p:05Triage→03Medium a:03hashar