[00:00:37] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:37] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [01:02:45] (03PS1) 10Arlolra: Add shn to the wikipedias list [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1059953 [01:04:54] (03CR) 10Arlolra: [C:03+2] "Merging to restart visual testing on shnwikivoyage" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1059953 (owner: 10Arlolra) [01:05:28] (03Merged) 10jenkins-bot: Add shn to the wikipedias list [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1059953 (owner: 10Arlolra) [03:28:24] Project mwcore-phpunit-coverage-master build #3761: 04STILL FAILING in 28 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/3761/ [05:39:12] 10Deployments, 10Release-Engineering-Team (Priority Backlog šŸ“„), 07Kubernetes: Build and publish multiple MediaWiki production images for a given set of PHP versions - https://phabricator.wikimedia.org/T370934#10044050 (10Joe) I think we need to be able to pass to the release script a set of base images for m... [07:05:31] (03CR) 10Hashar: "There is no need to add every single contributors to the allow list. A Code-Review +1 from a trusted person would trigger tests :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1059926 (owner: 10Pppery) [07:12:22] 10Continuous-Integration-Config: Migrate docker-registry.wikimedia.org/releng/release-notes from Buster to Bookworm - https://phabricator.wikimedia.org/T371511#10044133 (10jnuche) The modified job succeeded šŸŽ‰ - https://integration.wikimedia.org/ci/job/train-deploy-notes/4413/ - https://www.mediawiki.org/wiki... [07:12:24] 10Continuous-Integration-Config: Migrate docker-registry.wikimedia.org/releng/release-notes from Buster to Bookworm - https://phabricator.wikimedia.org/T371511#10044134 (10jnuche) [07:12:25] 10Release-Engineering-Team (Priority Backlog šŸ“„), 05Release, 05Train Deployments: 1.43.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T366962#10044135 (10jnuche) [09:03:36] 10Gerrit: Gerrit - Too many concurrent connections (8) - max. allowed: 8 - https://phabricator.wikimedia.org/T371749#10044358 (10xSavitar) > Please change you remote configuration to use https for fetching: > ` > git remote set-url origin https://gerrit.wikimedia.org/r/mediawiki/core.git > git remote set-url... [09:12:50] (03PS1) 10Hashar: dockerfiles: ci-src-setup to Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1060077 (https://phabricator.wikimedia.org/T335765) [09:14:51] (03PS1) 10Hashar: jjb: migrate ci-src-setup-simple to Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1060078 (https://phabricator.wikimedia.org/T335765) [09:15:31] (03CR) 10Hashar: [C:03+2] dockerfiles: ci-src-setup to Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1060077 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [09:17:20] (03Merged) 10jenkins-bot: dockerfiles: ci-src-setup to Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1060077 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [09:19:28] 10Gerrit: Gerrit - Too many concurrent connections (8) - max. allowed: 8 - https://phabricator.wikimedia.org/T371749#10044403 (10hashar) > On the side, Iā€™m also suspecting (without any logical proof) that a PHPStorm plugin - GitToolbox that I recently installed was doing multiple connections to Gerrit to do... [10:13:03] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: (some) Gitlab builds hanging - https://phabricator.wikimedia.org/T371620#10044507 (10Jelto) As far as I can tell, all of the linked jobs used the Digital Ocean Kubernetes runners, and all jobs fail because of system errors wit... [10:25:41] 10Deployments: Consider using JSON content model for deployment calendar - https://phabricator.wikimedia.org/T366880#10044584 (10Wargo) Yes, module solution seems to be enough. No need for extensions. Something stops us? [10:33:22] 10Release-Engineering-Team (Seen), 06collaboration-services, 10Data Pipelines, 06Data-Engineering, and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10044599 (10Stevemunene) Deployed the new version on the test cluster, with ` sudo dpkg -i airflow-2.9.3-py3.10-20240715_amd... [10:37:15] !log Updating Jenkins jobs to migrate ci-src-setup-simple to Bookworm | https://gerrit.wikimedia.org/r/1060078 | T335765 [10:37:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:37:18] T335765: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765 [10:44:33] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#10044609 (10hashar) [10:48:50] (03CR) 10Hashar: "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1060078 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [10:51:56] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: (some) Gitlab builds hanging - https://phabricator.wikimedia.org/T371620#10044610 (10MatthewVernon) Thanks for that suggestion - I tried a wmcs build, and it [[ https://gitlab.wikimedia.org/repos/sre/trafficserver/-/jobs/33409... [11:15:56] (03CR) 10Hashar: [C:03+2] jjb: migrate ci-src-setup-simple to Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1060078 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [11:17:44] (03Merged) 10jenkins-bot: jjb: migrate ci-src-setup-simple to Bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1060078 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [11:38:05] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: (some) Gitlab builds hanging - https://phabricator.wikimedia.org/T371620#10044722 (10MatthewVernon) Build [[ https://gitlab.wikimedia.org/repos/sre/trafficserver/-/jobs/334109 | 334109 ]] has hung again. So either I'm really u... [11:56:35] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: (some) Gitlab builds hanging - https://phabricator.wikimedia.org/T371620#10044738 (10MatthewVernon) [it did go on to fail `ERROR: Job failed (system failure): pods "runner-9e2abdumz-project-1859-concurrent-0-wmwlr4zv" not foun... [12:13:10] 10GitLab (Account Approval): Requesting GitLab account activation for Arthur taylor - https://phabricator.wikimedia.org/T371888 (10ArthurTaylor) 03NEW [12:13:44] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Arthur taylor - https://phabricator.wikimedia.org/T371888#10044780 (10ArthurTaylor) [12:38:54] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Arthur taylor - https://phabricator.wikimedia.org/T371888#10044840 (10Aklapper) 05Openā†’03Invalid Per banner on GitLab no action is required as you have been a member of #Trusted-Contributors since December... [15:18:03] 06Release-Engineering-Team: Rewrite remaining make-container-image code in Python - https://phabricator.wikimedia.org/T371904 (10dancy) 03NEW [15:27:07] Project mwcore-phpunit-coverage-master build #3762: 04STILL FAILING in 27 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/3762/ [15:29:45] (03Abandoned) 10Pppery: Add Bodhisattwa to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1059927 (owner: 10Pppery) [15:29:49] (03Abandoned) 10Pppery: Add WgevaertWikiBase to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1059926 (owner: 10Pppery) [15:40:59] (03update) 10dancy: Convert remaining image building code to Python [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 (https://phabricator.wikimedia.org/T371904) [15:41:03] (03update) 10dancy: Convert remaining image building code to Python [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 (https://phabricator.wikimedia.org/T371904) [15:41:13] (03update) 10dancy: Convert remaining image building code to Python [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 (https://phabricator.wikimedia.org/T371904) [16:09:47] (03PS1) 10Hashar: jjb: migrate python tox jobs to python-all image [integration/config] - 10https://gerrit.wikimedia.org/r/1060142 (https://phabricator.wikimedia.org/T342019) [16:18:13] (03CR) 10CI reject: [V:04-1] jjb: migrate python tox jobs to python-all image [integration/config] - 10https://gerrit.wikimedia.org/r/1060142 (https://phabricator.wikimedia.org/T342019) (owner: 10Hashar) [16:18:45] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#10045861 (10hashar) [16:53:47] 10Deployments, 10Release-Engineering-Team (Priority Backlog šŸ“„), 07Kubernetes: Build and publish multiple MediaWiki production images for a given set of PHP versions - https://phabricator.wikimedia.org/T370934#10046051 (10Scott_French) Agreed with @Joe's assessment above: for each image type (e.g., mediawiki)... [17:02:55] (03open) 10dancy: make-release/mwrelease/branch.py: Rename a do_core_work parameter [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/101 [17:02:57] (03update) 10dancy: make-release/mwrelease/branch.py: Rename a do_core_work parameter [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/101 [17:06:07] (03update) 10dancy: make-release/mwrelease/branch.py: Rename a do_core_work parameter [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/101 [17:23:02] 10Beta-Cluster-Infrastructure, 10Add-Link, 10Growth-Team (FY2024-25 Q1 Sprint 3): refreshLinkRecommendation script fails in Beta cluster with FileNotFoundError - https://phabricator.wikimedia.org/T370792#10046159 (10Urbanecm_WMF) [17:31:03] 10Deployments: Consider using JSON content model for deployment calendar - https://phabricator.wikimedia.org/T366880#10046244 (10bd808) >>! In T366880#10044584, @Wargo wrote: > Yes, module solution seems to be enough. No need for extensions. Something stops us? Technically, nothing other than time and coordinat... [17:31:29] (03approved) 10brennen: deploy.py: Handle missing keyholder key with clear error logging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/328 (https://phabricator.wikimedia.org/T313624) (owner: 10sandeeps) [18:01:28] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Seen), 06collaboration-services, and 2 others: Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#10046298 (10Dzahn) @hashar The new private key has been added to the jenkins credentials store, twic... [18:10:15] (03update) 10dancy: Convert remaining image building code to Python [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 (https://phabricator.wikimedia.org/T371904) [18:10:51] (03update) 10dancy: Convert remaining image building code to Python [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 (https://phabricator.wikimedia.org/T371904) [18:11:17] (03merge) 10dancy: Convert remaining image building code to Python [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/99 (https://phabricator.wikimedia.org/T371904) [18:13:01] 06Release-Engineering-Team, 13Patch-For-Review: Rewrite remaining make-container-image code in Python - https://phabricator.wikimedia.org/T371904#10046366 (10dancy) 05Openā†’03In progress p:05Triageā†’03Medium [18:53:50] 06Release-Engineering-Team, 13Patch-For-Review: Rewrite remaining make-container-image code in Python - https://phabricator.wikimedia.org/T371904#10046508 (10dancy) 05In progressā†’03Resolved Deployed to prod and tested. [19:26:29] !log extloc: experimenting with running from Procfile (T365665) [19:26:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:26:31] T365665: extloc: Move to Toolforge Build Service - https://phabricator.wikimedia.org/T365665 [19:29:52] !log Deleted deployment-deploy03 agent from the CI Jenkins, that got replaced by deployment-deploy04 by thcipriani in July as part of migrating deployment-prep instances out of Buster [19:29:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:37:11] brennen: how is the build service stuff working for you? Docs ok for the things you needed to know so far? [19:44:27] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Seen), 06collaboration-services, 06SRE: Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#10046567 (10hashar) I have changed the key of the contint1002 agent (via the [[ https://integration.wikim... [19:50:54] bd808: yeah, so far so good [19:57:23] i can see the appeal of a push-to-build workflow here. [20:06:06] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 06cloud-services-team: Wikitech system account and SUL for Jenkins agents? - https://phabricator.wikimedia.org/T371930 (10hashar) 03NEW [20:06:28] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 06cloud-services-team: Wikitech system account and SUL for Jenkins agents? - https://phabricator.wikimedia.org/T371930#10046604 (10hashar) [20:06:31] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Seen), 06collaboration-services, 06SRE: Upgrade CI Jenkins ssh key to ecdsa - https://phabricator.wikimedia.org/T177826#10046603 (10hashar) [20:06:55] brennen: once we close the loop on making it an automated action to build and deploy the new container image I think folks will generally like it. David has things on the roadmap to enable deployments to be multi-container too for more complicated tools. [20:11:41] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 06cloud-services-team: Wikitech system account and SUL for Jenkins agents? - https://phabricator.wikimedia.org/T371930#10046610 (10bd808) The login for https://idm.wikimedia.org is via https://idp.wikimedia.org. The IdP service uses the... [20:12:03] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 06cloud-services-team: Wikitech system account and SUL for Jenkins agents? - https://phabricator.wikimedia.org/T371930#10046614 (10hashar) [20:17:12] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Bitu, 10CAS-SSO, and 2 others: Wikitech system account and SUL for Jenkins agents? - https://phabricator.wikimedia.org/T371930#10046623 (10bd808) Here is the developer account record: ` $ ldap uid=jenkins-deploy dn: uid=jenkins-deplo... [20:41:31] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Bitu, 10CAS-SSO, and 2 others: Wikitech system account and SUL for Jenkins agents? - https://phabricator.wikimedia.org/T371930#10046726 (10Dzahn) Given that users are always supposed to use different keys for prod vs cloud, should th... [21:51:37] (03open) 10dancy: Use debian:11 as the base image [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/81 [21:51:41] (03update) 10dancy: Use debian:11 as the base image [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/81 [21:52:41] (03merge) 10dancy: Use debian:11 as the base image [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/81 [22:03:44] 10Release-Engineering-Team (Yak Shaving šŸƒšŸŖ’), 10Tool-extloc, 13Patch-For-Review: extloc: Move to Toolforge Build Service - https://phabricator.wikimedia.org/T365665#10046945 (10brennen) 05Openā†’03Resolved