[06:33:44] (03open) 10phedenskog: Add zuul_pipeline column to jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/4 (https://phabricator.wikimedia.org/T424515) [06:40:01] Project mediawiki-core-doxygen build #20073: 04FAILURE in 7 min 31 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/20073/ [06:52:07] Yippee, build fixed! [06:52:07] Project mediawiki-core-doxygen build #20074: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/20074/ [07:12:59] (03PS1) 10Kosta Harlan: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 [07:14:34] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [07:20:19] (03PS2) 10Kosta Harlan: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 [07:21:34] (03open) 10phedenskog: Add zuul_pipeline column to jobs (and update the queries) [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/5 (https://phabricator.wikimedia.org/T424515) [07:21:57] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [07:22:37] (03PS3) 10Kosta Harlan: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 [07:23:41] (03close) 10phedenskog: Add zuul_pipeline column to jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/4 (https://phabricator.wikimedia.org/T424515) [07:27:01] (03CR) 10Dreamy Jazz: [C:03+1] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [07:39:56] (03CR) 10Phuedx: [C:03+1] "Regardless of whether we make WikimediaEvents explicitly depend on TestKitchen _now_, WikimediaCustomizations should load TestKitchen so t" [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [07:47:54] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 10Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11874235 (10phuedx) Here's another instance of the lack of an explicit dependency causing issues: https://wikimedia.slack.com/archive... [07:59:34] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11874262 (10Jelto) [08:04:37] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11874275 (10Jelto) [08:06:04] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11874278 (10Jelto) 05Openβ†’03Resolved All hosts are migrated and cleanup has happened on all hosts, I'll resolve th... [08:08:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 10Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11874287 (10phuedx) >>! In T419679#11730635, @Bhsd wrote: > I am seeing [CI failure](https://integration.wikimedia.org/ci/job/quibble... [08:10:17] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 10Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11874289 (10Bhsd) >>! In T419679#11874287, @phuedx wrote: > Is the CI failure still happening? It has been fixed and the patch has b... [08:22:46] (03CR) 10Hashar: [C:03+2] wm-checks-api: add tag for PostgreSQL jobs [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1266965 (owner: 10Hashar) [08:23:35] (03Merged) 10jenkins-bot: wm-checks-api: add tag for PostgreSQL jobs [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1266965 (owner: 10Hashar) [08:25:02] (03CR) 10Hashar: [C:03+2] jjb: Drop a handful of unused PHP 8.1 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1278620 (owner: 10Jforrester) [08:26:52] (03Merged) 10jenkins-bot: jjb: Drop a handful of unused PHP 8.1 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1278620 (owner: 10Jforrester) [08:27:06] (03CR) 10Hashar: "Note in Zuul release pipelines, we still refer to `mwext-phan-php81`:" [integration/config] - 10https://gerrit.wikimedia.org/r/1278620 (owner: 10Jforrester) [08:30:10] (03CR) 10Hashar: [C:03+2] "I have deleted the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/1278620 (owner: 10Jforrester) [08:38:02] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [08:39:39] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add TestKitchen dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [08:42:19] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [08:53:34] RESOLVED: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:22:36] (03CR) 10Hashar: [C:03+2] "This is now deployed, sorry I forgot to do it immediately since I was multi tasking!" [integration/config] - 10https://gerrit.wikimedia.org/r/1280074 (owner: 10Kosta Harlan) [09:23:44] (03CR) 10Hashar: [C:03+2] phpunit: Add test plan for core phpunit tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1271080 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [09:28:37] (03PS3) 10Hashar: Add support to spawn Memcached as a backend [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) [09:28:37] (03CR) 10Hashar: "Hi Peter, the ultimate goal is to remove supervisord from the Quibble CI images to simplify the stack. Not sure whether you have the envy " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [09:33:37] (03open) 10jnuche: update_jenkins.sh: harden local host key detection [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/115 [09:33:44] (03merge) 10jnuche: update_jenkins.sh: harden local host key detection [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/115 [09:40:57] (03Merged) 10jenkins-bot: phpunit: Add test plan for core phpunit tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1271080 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [09:52:00] (03CR) 10Phedenskog: [C:03+1] Fix semantic for Quibble injected PHP constants [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277462 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [10:01:26] (03PS10) 10Pwangai: phpunit: Support parallel execution for core default suite [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) [10:16:04] (03PS1) 10Hashar: dockerfiles: rebuild Quibble images for supervisord [integration/config] - 10https://gerrit.wikimedia.org/r/1280257 (https://phabricator.wikimedia.org/T397810) [10:16:17] (03PS3) 10Hashar: dockerfiles: quibble-bullseye: improve supervisord configuration [integration/config] - 10https://gerrit.wikimedia.org/r/1251122 (https://phabricator.wikimedia.org/T397810) [10:17:49] (03PS2) 10Hashar: dockerfiles: quibble-bullseye: fix run as different user [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) [10:17:49] (03CR) 10Hashar: "I have removed the `changelog` entries from this change since there is another tightly related one I6159edb0085abad26a26475176a1febc0415a4" [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [11:28:41] (03CR) 10Phedenskog: [C:03+1] "Looks good, what is the main end goal here, we are in control of memcached instead of the container?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [11:41:04] (03open) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/116 (https://phabricator.wikimedia.org/T424850) [11:41:44] (03merge) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/116 (https://phabricator.wikimedia.org/T424850) [12:04:38] 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Update scap3-dev - https://phabricator.wikimedia.org/T424954 (10jnuche) 03NEW [12:04:44] 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Update scap3-dev - https://phabricator.wikimedia.org/T424954#11874987 (10jnuche) [12:41:48] 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975 (10Urbanecm_WMF) 03NEW [12:43:56] (03CR) 10Hashar: "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:44:07] (03CR) 10Hashar: [C:03+2] dockerfiles: rebuild Quibble images for supervisord [integration/config] - 10https://gerrit.wikimedia.org/r/1280257 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:44:12] (03CR) 10Hashar: [C:03+2] dockerfiles: quibble-bullseye: improve supervisord configuration [integration/config] - 10https://gerrit.wikimedia.org/r/1251122 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:44:18] (03CR) 10Hashar: [C:03+2] dockerfiles: quibble-bullseye: fix run as different user [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:44:43] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11875286 (10Urbanecm_WMF) p:05Triageβ†’03High Triaging as High, as this makes deployers fly blind. CC @thcipriani. [12:46:07] (03Merged) 10jenkins-bot: dockerfiles: quibble-bullseye: fix run as different user [integration/config] - 10https://gerrit.wikimedia.org/r/1251084 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:46:10] (03Merged) 10jenkins-bot: dockerfiles: quibble-bullseye: improve supervisord configuration [integration/config] - 10https://gerrit.wikimedia.org/r/1251122 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:46:12] (03Merged) 10jenkins-bot: dockerfiles: rebuild Quibble images for supervisord [integration/config] - 10https://gerrit.wikimedia.org/r/1280257 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [12:47:54] (03open) 10jnuche: housekeeping [repos/releng/scap3-dev] - 10https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/46 (https://phabricator.wikimedia.org/T424954) [12:48:33] (03merge) 10jnuche: housekeeping [repos/releng/scap3-dev] - 10https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/46 (https://phabricator.wikimedia.org/T424954) [12:50:32] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review: Update scap3-dev - https://phabricator.wikimedia.org/T424954#11875313 (10jnuche) 05Openβ†’03Resolved [13:20:07] (03PS1) 10Hashar: jjb: update Quibble jobs for supervisord improvements [integration/config] - 10https://gerrit.wikimedia.org/r/1280408 (https://phabricator.wikimedia.org/T397810) [13:21:22] (03CR) 10Hashar: [C:03+2] "I ran the new version of the image on contint using:" [integration/config] - 10https://gerrit.wikimedia.org/r/1280408 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [13:23:08] (03Merged) 10jenkins-bot: jjb: update Quibble jobs for supervisord improvements [integration/config] - 10https://gerrit.wikimedia.org/r/1280408 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [14:05:37] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Quibble, 07ci-test-error (WMF-deployed Build Failure): Implement a retry policy for network errors in CI - https://phabricator.wikimedia.org/T424990 (10ABran-WMF) 03NEW [14:07:22] (03PS2) 10Arnaudb: zuul: retry policy on network errors [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T424990) [14:07:26] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Quibble, and 3 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11875668 (10ABran-WMF) 05Stalledβ†’03Resolved Closing that task as the network error rate is now more under contro... [14:22:15] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11875738 (10dancy) Looking into this. [14:31:40] (03PS11) 10Hashar: phpunit: Support parallel execution for core default suite [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [14:40:24] (03CR) 10Hashar: [C:03+2] "Paired with Peter & Peter 😊" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [14:41:24] (03CR) 10Hashar: "The ultimate goal is to remove supervisord to manage backend services and have that directly in Quibble :-]" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [14:41:48] (03CR) 10Hashar: [C:03+2] Fix semantic for Quibble injected PHP constants [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277462 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [14:41:50] (03CR) 10Hashar: [C:03+2] Add support to spawn Memcached as a backend [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [14:46:21] (03Merged) 10jenkins-bot: phpunit: Support parallel execution for core default suite [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [14:47:31] (03Merged) 10jenkins-bot: Fix semantic for Quibble injected PHP constants [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277462 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [14:47:33] (03CR) 10CI reject: [V:04-1] Add support to spawn Memcached as a backend [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [14:55:54] (03PS1) 10Pmiazga: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add EventStreamConfig dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280466 [14:56:28] (03CR) 10Pmiazga: "A sample change when this fails: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaCustomizations/+/1280374" [integration/config] - 10https://gerrit.wikimedia.org/r/1280466 (owner: 10Pmiazga) [14:59:37] (03merge) 10thcipriani: Add job run time deviation and spread over one day [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/3 (https://phabricator.wikimedia.org/T420700) (owner: 10phedenskog) [14:59:54] (03merge) 10thcipriani: Add zuul_pipeline column to jobs (and update the queries) [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/5 (https://phabricator.wikimedia.org/T424515) (owner: 10phedenskog) [15:05:40] 06Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11875998 (10thcipriani) [15:11:09] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879#11876017 (10Dzahn) restarted zuul-executor on zuul1002 after deploying version upgrade change: ` Apr 30 15:09:19 zuul1002 docker[1886062]: 5... [15:11:58] 10GitLab (Project Migration), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 10Fundraising analytics stack, and 2 others: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#11876024 (10greg) [15:14:18] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879#11876037 (10Dzahn) ` [zuul1002:~] $ sudo docker ps CONTAINER ID IMAGE... [15:15:06] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879#11876044 (10Dzahn) [15:17:13] did train log triage happen in an earlier slot? asking for a "friend" :-P [15:19:10] !log upgrading zuul to 14.2.0-1 on "new zuul" machines (T424879) [15:19:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:19:13] T424879: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879 [15:23:15] (03open) 10dancy: JobLog.vue: Improve performance of job log [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1149 (https://phabricator.wikimedia.org/T424975) [15:23:20] (03update) 10dancy: JobLog.vue: Improve performance of job log [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1149 (https://phabricator.wikimedia.org/T424975) [15:24:19] (03PS4) 10Hashar: Add support to spawn Memcached as a backend [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) [15:25:52] (03PS5) 10Hashar: Add support to spawn Memcached as a backend [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) [15:26:19] (03merge) 10dancy: JobLog.vue: Improve performance of job log [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1149 (https://phabricator.wikimedia.org/T424975) [15:26:39] (03CR) 10Hashar: [C:03+2] "Rebased/solved conflict with `tests/plans/010_phpunit_core.yaml` which has been introduced recently." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [15:27:15] (03update) 10dancy: Release 4.256.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1150 [15:27:16] (03open) 10dancy: Release 4.256.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1150 [15:29:54] (03merge) 10dancy: Release 4.256.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1150 [15:33:10] 06Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11876187 (10dancy) The offending job output: P92090 [15:42:16] (03Merged) 10jenkins-bot: Add support to spawn Memcached as a backend [integration/quibble] - 10https://gerrit.wikimedia.org/r/1277526 (https://phabricator.wikimedia.org/T397810) (owner: 10Hashar) [15:51:58] 06Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11876251 (10dancy) Fixed in scap 4.256.0. The job log for https://spiderpig.wikimedia.org/jobs/1862 now renders without freezing the browser. I... [16:01:52] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add EventStreamConfig dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280466 (owner: 10Pmiazga) [16:03:33] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add EventStreamConfig dep [integration/config] - 10https://gerrit.wikimedia.org/r/1280466 (owner: 10Pmiazga) [16:05:33] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Tmullapu] - https://phabricator.wikimedia.org/T424682#11876337 (10Tmullapu) I have signed into GitLab using my Wikimedia developer account (username: Tmullapu). The account shows as pending approval. I have an ac... [16:09:51] (03CR) 10Hashar: [C:03+2] "deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1280466 (owner: 10Pmiazga) [16:29:41] 06Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig πŸ•ΈοΈ): Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11876476 (10Scott_French) Under the hood, `helmfile apply` will call `helm diff upgrade`, which (1) communicates to `helmfile` whether there is i... [16:30:33] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Allow wikis to be named, instead of using a hash - https://phabricator.wikimedia.org/T372964#11876480 (10thcipriani) [16:48:29] 06Release-Engineering-Team, 10dev-images, 10Catalyst (Luka Ijo Pimeja Jan): Upgrade Patchdemo/Catalyst PHP images to use PHP 8.3, ensure the node matches what's used in CI - https://phabricator.wikimedia.org/T423359#11876511 (10thcipriani) p:05Triageβ†’03Medium a:03brennen [16:49:05] (03open) 10dancy: kubernetes.py: Pass --context 5 to helmfile apply [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1151 (https://phabricator.wikimedia.org/T424975) [16:49:12] (03update) 10dancy: kubernetes.py: Pass --context 5 to helmfile apply [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1151 (https://phabricator.wikimedia.org/T424975) [16:53:57] (03update) 10dancy: kubernetes.py: Pass --context 5 to helmfile apply [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1151 (https://phabricator.wikimedia.org/T424975) [16:55:34] (03update) 10dancy: kubernetes.py: Pass --context 5 to helmfile apply [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1151 (https://phabricator.wikimedia.org/T424975) [16:56:22] (03approved) 10swfrench: kubernetes.py: Pass --context 5 to helmfile apply [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1151 (https://phabricator.wikimedia.org/T424975) (owner: 10dancy) [16:56:36] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Tmullapu] - https://phabricator.wikimedia.org/T424682#11876542 (10brennen) 05Openβ†’03Resolved a:03brennen Account approved. Happy hacking! [16:56:38] (03merge) 10dancy: kubernetes.py: Pass --context 5 to helmfile apply [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1151 (https://phabricator.wikimedia.org/T424975) [16:56:54] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Tmullapu] - https://phabricator.wikimedia.org/T424682#11876546 (10brennen) [16:57:24] (03update) 10dancy: Release 4.257.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1152 [16:57:25] (03open) 10dancy: Release 4.257.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1152 [16:59:15] 06Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11876565 (10dancy) >>! In T424975#11876476, @Scott_French wrote: > I am ... mildly(?) ... confident the `helm diff upgrade`... [16:59:39] (03merge) 10dancy: Release 4.257.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1152 [17:07:39] 06Release-Engineering-Team (Doing 😎), 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: Certain deployment logs cause Spiderpig to crash the browser - https://phabricator.wikimedia.org/T424975#11876582 (10dancy) 05Openβ†’03Resolved a:03dancy `--context 5` fix deployed in scap 4.257.0. [17:09:28] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879#11876587 (10dduvall) [17:11:04] (03open) 10dancy: gitlab-runner-values.yaml.tftpl: Bump to docker-registry.wikimedia.org/bullseye:20260426 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/589 (https://phabricator.wikimedia.org/T423971) [17:11:12] (03update) 10dancy: gitlab-runner-values.yaml.tftpl: Bump to docker-registry.wikimedia.org/bullseye:20260426 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/589 (https://phabricator.wikimedia.org/T423971) [17:12:08] (03update) 10dancy: gitlab-runner-values.yaml.tftpl: Bump to docker-registry.wikimedia.org/bullseye:20260426 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/589 (https://phabricator.wikimedia.org/T423971) [17:12:22] (03update) 10dancy: gitlab-runner-values.yaml.tftpl: Bump to docker-registry.wikimedia.org/bullseye:20260426 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/589 (https://phabricator.wikimedia.org/T423971) [17:13:52] (03merge) 10dancy: gitlab-runner-values.yaml.tftpl: Bump to docker-registry.wikimedia.org/bullseye:20260426 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/589 (https://phabricator.wikimedia.org/T423971) [17:16:45] !log wikibugs: most maintainers at hackathon, so go release-engineering added as a maintainer while looking to debug error at https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/810904 [17:16:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:23:50] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Update default GitLab runner image to a base image without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423971#11876616 (10dancy) 05Openβ†’03Resolved... [18:41:44] Krinkle: https://github.com/wikimedia/eslint-plugin-mediawiki/pull/130 as a starter for 10. [19:18:05] (03CR) 10Jforrester: "Yeah, I intend to argue with @Daimona about this at the Hackathon. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/1278620 (owner: 10Jforrester) [19:35:02] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600#11877023 (10brennen) 05Openβ†’03Stalled a:03brennen Hi @Krrish_Sehgal. Sorry you haven't gotten a response here. Can you confirm that you have logged i... [19:35:41] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600#11877030 (10brennen) [19:38:46] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for xaoseric - https://phabricator.wikimedia.org/T416092#11877035 (10brennen) 05Openβ†’03Resolved a:03brennen [19:40:05] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for xaoseric - https://phabricator.wikimedia.org/T416092#11877042 (10brennen) Account activated; apologies for the delay, I think this one slipped through the cracks early in the year. Happy hacking! [20:08:13] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WebAuthn] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1280653 (https://phabricator.wikimedia.org/T303495) [20:08:50] (03Abandoned) 10Jforrester: Zuul: Turn off extension-(coverage|codehealth) for WebAuthn [integration/config] - 10https://gerrit.wikimedia.org/r/1234503 (https://phabricator.wikimedia.org/T415832) (owner: 10Reedy) [20:09:06] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/WebAuthn] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1234505 (https://phabricator.wikimedia.org/T415832) [20:09:12] (03Abandoned) 10Jforrester: Zuul: [mediawiki/extensions/WebAuthn] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1280653 (https://phabricator.wikimedia.org/T303495) (owner: 10Jforrester) [20:09:20] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/WebAuthn] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1234505 (https://phabricator.wikimedia.org/T415832) [20:11:12] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WebAuthn] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1234505 (https://phabricator.wikimedia.org/T415832) (owner: 10Jforrester) [20:12:54] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WebAuthn] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/1234505 (https://phabricator.wikimedia.org/T415832) (owner: 10Jforrester) [20:14:27] !log Zuul: [mediawiki/extensions/WebAuthn] Archive, for T415832 / T303495 [20:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:14:30] T415832: Remove https://doc.wikimedia.org/cover-extensions/WebAuthn/ - https://phabricator.wikimedia.org/T415832 [20:14:31] T303495: Merge WebAuthn extension into OATHAuth - https://phabricator.wikimedia.org/T303495 [20:16:32] !log jforrester@doc1004:~$ # sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn/ # T415832 [20:16:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:17:08] 10Continuous-Integration-Config, 10MediaWiki-extensions-OATHAuth, 06Product Safety and Integrity, 13Patch-For-Review: Remove https://doc.wikimedia.org/cover-extensions/WebAuthn/ - https://phabricator.wikimedia.org/T415832#11877121 (10Jdforrester-WMF) 05Openβ†’03Resolved [20:24:25] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting [integration/config] - 10https://gerrit.wikimedia.org/r/1280665 [20:24:34] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting [integration/config] - 10https://gerrit.wikimedia.org/r/1280665 (owner: 10Jforrester) [20:26:11] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting [integration/config] - 10https://gerrit.wikimedia.org/r/1280665 (owner: 10Jforrester) [20:26:25] !log Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting [20:26:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:45:49] 10GitLab (Upstream pit of despair πŸ•³οΈ), 06Release-Engineering-Team (Radar): GitLab MR reviews throw 500 errors when commenting on some specific lines - https://phabricator.wikimedia.org/T337514#11877231 (10brennen) 05Openβ†’03Resolved a:03brennen Upstream, this was marked a duplicate of the (now closed)... [20:47:25] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for mroik - https://phabricator.wikimedia.org/T425037 (10Mroik) 03NEW [20:47:41] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06collaboration-services: Ocassionally pushing to GitLab takes very long - https://phabricator.wikimedia.org/T332967#11877253 (10brennen) 05Openβ†’03Resolved a:03brennen I haven't seen this behavior any time recently that I can think of.... [20:55:09] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for tfmorris - https://phabricator.wikimedia.org/T424730#11877274 (10brennen) 05Openβ†’03Resolved a:03brennen I've added you to #trusted-contributors, the bot should approve your GitLab account shortly. Happy... [20:58:12] (03open) 10dancy: canary checks: Remove support for baremetal targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1153 (https://phabricator.wikimedia.org/T424901) [20:58:15] (03update) 10dancy: canary checks: Remove support for baremetal targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1153 (https://phabricator.wikimedia.org/T424901) [20:59:40] (03merge) 10dancy: canary checks: Remove support for baremetal targets [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1153 (https://phabricator.wikimedia.org/T424901) [21:04:55] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879#11877306 (10Dzahn) [21:05:28] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11877308 (10Dzahn) zuul-nodepool has been removed from zuul::main machines. (config, systemd unit, docker container not running, etc) [21:05:30] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Upgrade to Zuul 14.x - https://phabricator.wikimedia.org/T424879#11877309 (10Dzahn) zuul-nodepool has been removed from zuul::main machines. (config, systemd unit, docker container not running, etc) [21:09:30] (03open) 10dancy: logstash_checker.py: Support arbitrary stage error rate checking [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154 [21:09:34] (03update) 10dancy: logstash_checker.py: Support arbitrary stage error rate checking [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154 [21:10:36] (03update) 10dancy: logstash_checker.py: Support arbitrary stage error rate checking [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154 (https://phabricator.wikimedia.org/T317405) [21:10:40] (03update) 10dancy: logstash_checker.py: Support arbitrary stage error rate checking [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154 (https://phabricator.wikimedia.org/T317405) [21:13:04] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for mroik - https://phabricator.wikimedia.org/T425037#11877315 (10brennen) a:03brennen Hi @Mroik - can you let us know what projects you plan to work on? [21:13:30] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for mroik - https://phabricator.wikimedia.org/T425037#11877319 (10brennen) [21:14:42] (03update) 10dancy: logstash_checker.py: Support arbitrary stage error rate checking [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154 (https://phabricator.wikimedia.org/T317405) [21:20:40] (03update) 10dancy: logstash_checker.py: Support arbitrary stage error rate checking [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1154 (https://phabricator.wikimedia.org/T317405) [21:25:37] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for mroik - https://phabricator.wikimedia.org/T425037#11877340 (10Mroik) I'm honestly not sure. I was supposed to look over at the list of open tasks for the wikimedia hackathon this week but didn'... [21:28:28] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for mroik - https://phabricator.wikimedia.org/T425037#11877343 (10brennen) 05Openβ†’03Resolved Cool cool - this is mostly just a basic filtering process to stem the tide of spambot accounts w... [21:28:53] 06Release-Engineering-Team, 10Scap, 07Developer Productivity: Consider reducing output in pre-sync phase of scap-backport - https://phabricator.wikimedia.org/T323325#11877348 (10dancy) 05Declinedβ†’03Open Reconsidering this. I'm open to trimming the terminal output. [21:29:21] (03update) 10brennen: Hide events from non-public projects [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 [21:29:33] (03update) 10brennen: Hide events from non-public projects [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 [21:32:42] 06Release-Engineering-Team, 10Scap, 07Developer Productivity: Consider reducing output in pre-sync phase of scap-backport - https://phabricator.wikimedia.org/T323325#11877357 (10dancy) This is the output from a recent config backport (I couldn't handle the pictures of text). @Krinkle, if you can make some c... [21:34:02] (03open) 10dancy: config.py: Add k8s_helmfile_diff_context_lines setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1155 [21:34:11] (03update) 10dancy: config.py: Add k8s_helmfile_diff_context_lines setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1155 [21:35:29] (03merge) 10brennen: Hide events from non-public projects [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 [21:36:13] (03update) 10dancy: config.py: Add k8s_helmfile_diff_context_lines setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1155 [21:36:34] !log gitlab-webhooks: building & restarting to deploy https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 [21:36:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:46:22] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06Infrastructure-Foundations: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm - https://phabricator.wikimedia.org/T407557#11877400 (10brennen) [21:47:11] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: GitLab ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T408416#11877403 (10brennen) [21:47:54] (03approved) 10swfrench: config.py: Add k8s_helmfile_diff_context_lines setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1155 (owner: 10dancy) [21:55:46] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Radar): Retire /cloudvps-repos, merge to /repos - https://phabricator.wikimedia.org/T387062#11877419 (10brennen) [21:55:55] 10GitLab (Support): Can't login to gitlab.wikimedia.org - https://phabricator.wikimedia.org/T422105#11877421 (10brennen) [22:01:53] 10GitLab, 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: Fundraising-ML - https://phabricator.wikimedia.org/T417305#11877431 (10brennen) 05Openβ†’03Resolved a:03brennen Sorry for the delay on this one. Just realized a few private repo requests have fallen through... [22:04:10] 10GitLab, 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: CSP Research - https://phabricator.wikimedia.org/T420485#11877436 (10brennen) 05Openβ†’03In progress a:03brennen I note that https://gitlab.wikimedia.org/repos/security/csp-research has already been created. Is... [22:08:17] 10GitLab, 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: production access meta data - https://phabricator.wikimedia.org/T411642#11877459 (10brennen) 05Openβ†’03In progress Apologies nobody attended to this request sooner. Is this one still needed, and if so what group... [22:27:50] (03merge) 10dancy: config.py: Add k8s_helmfile_diff_context_lines setting [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1155 [22:39:06] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600#11877514 (10Pppery) The account would have been auto-rejected due to not being approved within 90 days. See T384264 [22:58:41] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600#11877535 (10brennen) > The account would have been auto-rejected due to not being approved within 90 days. Ah, good call. I'd misremembered that it would... [22:58:50] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600#11877536 (10brennen) p:05Triageβ†’03Low [23:19:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-eventgate-4 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:29:35] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-eventgate-4 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources