[00:34:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11802632 (10DLynch) Definitely still happening -- just saw it on https://integration.wikimedia.org/ci/job/quibbl... [05:49:20] 10GitLab (Integrations), 06Release-Engineering-Team, 06cloud-services-team, 10Toolforge: gitlab-webhooks build fails on "The runtime.txt file isn't supported" - https://phabricator.wikimedia.org/T422734#11802933 (10dcaro) Made a note in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_contai... [06:02:45] (03CR) 10Phedenskog: "What about splitting this into two changes. One first that adds a test plan for phpunit and then the actual change and an updated plan?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [06:55:13] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11802993 (10ArthurTaylor) Same - https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php82/36243/console [07:21:57] (03PS1) 10Phedenskog: docs: Use correct variable name for the release tag [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269255 [07:26:10] (03PS2) 10Phedenskog: docs: Use correct variable name for the release tag [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269255 [07:33:09] (03PS3) 10Hashar: releasing: Use correct variable name for the release tag [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269255 (owner: 10Phedenskog) [07:33:17] (03CR) 10Hashar: [C:03+2] releasing: Use correct variable name for the release tag [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269255 (owner: 10Phedenskog) [07:37:55] (03PS3) 10Phedenskog: releasing: Prepare changelog for 1.17.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268704 [07:44:11] (03CR) 10Hashar: [C:03+2] "Lets fly!" [integration/config] - 10https://gerrit.wikimedia.org/r/1268559 (https://phabricator.wikimedia.org/T417412) (owner: 10Zfilipin) [07:46:03] (03Merged) 10jenkins-bot: Zuul: use standalone jobs for GrowthExperiments Cypress tests [integration/config] - 10https://gerrit.wikimedia.org/r/1268559 (https://phabricator.wikimedia.org/T417412) (owner: 10Zfilipin) [07:48:45] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1268559 "Zuul: use standalone jobs for GrowthExperiments Cypress tests" | T417412 [07:48:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:48:47] T417412: Configure the CI so the new GrowthExperiments standalone job runs in all appropriate pipelines - https://phabricator.wikimedia.org/T417412 [07:52:07] (03Merged) 10jenkins-bot: releasing: Use correct variable name for the release tag [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269255 (owner: 10Phedenskog) [08:33:50] 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments, 10Wikidata: Run @selenium-test npm script in release branches - https://phabricator.wikimedia.org/T422786 (10zeljkofilipin) 03NEW [08:33:54] 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments, 10Wikidata: Run @selenium-test npm script in release branches - https://phabricator.wikimedia.org/T422786#11803211 (10zeljkofilipin) [08:34:32] 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments, 10Wikidata: Run @selenium-test npm script in release branches - https://phabricator.wikimedia.org/T422786#11803214 (10zeljkofilipin) [09:00:51] 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments, 10Wikidata: Run @selenium-test npm script in release branches - https://phabricator.wikimedia.org/T422786#11803348 (10hashar) [09:01:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11803365 (10ABran-WMF) Thanks @DLynch @ArthurTaylor for reporting these builds. The error rate seems steady: `... [09:27:53] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11803469 (10ABran-WMF) A first manual test on gerrit-spare shows no breakage, I will now try to apply that chang... [09:35:36] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: patchdemo staging: Creation of new demos with catalyst backend fails on connecting to db - https://phabricator.wikimedia.org/T421181#11803509 (10jnuche) 05Openβ†’03Resolved I can't reproduce the issue on staging a... [09:52:31] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11803552 (10ABran-WMF) Envoy now stops reusing connections to httpd on our Gerrit primary instance. I'll monitor... [12:38:08] 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments, 10Wikidata: Run @selenium-test npm script in release branches - https://phabricator.wikimedia.org/T422786#11804119 (10hashar) The already cut release branches still have `selenium-test` and have the selenium tests running. A more recent e... [13:15:51] (03CR) 10Hashar: "I thought maybe Phan would be able to detect the `TypeError`, it might not be that smart. The repository does not have much PHPUnit test" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1267982 (owner: 10Umherirrender) [13:28:57] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11804390 (10ABran-WMF) No impact from that change over the past few hours: ` root@contint1002:/srv/jenkins/build... [13:30:51] 10Scap: scap backport fails too fast when trying to re-merge a patch that had failed CI - https://phabricator.wikimedia.org/T420501#11804398 (10Lucas_Werkmeister_WMDE) I think something similar happened in [SpiderPig #1723](https://spiderpig.wikimedia.org/jobs/1723) (cc @hashar), where it complained that β€œThe ch... [13:35:32] 10Scap: scap backport fails too fast when trying to re-merge a patch that had failed CI - https://phabricator.wikimedia.org/T420501#11804427 (10hashar) This happened while backporting https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1269334 . The sequence was: * patch got rebased triggering test * scap backpor... [13:50:10] 10Continuous-Integration-Config, 06Growth-Team, 10GrowthExperiments, 10Wikidata: Run @selenium-test npm script in *future* release branches - https://phabricator.wikimedia.org/T422786#11804479 (10hashar) [14:01:29] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Upgrade K3s cluster to most recent stable version - https://phabricator.wikimedia.org/T400077#11804535 (10jnuche) [14:01:41] (03PS4) 10Hashar: releasing: Quibble 1.17.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268704 (owner: 10Phedenskog) [14:02:10] (03CR) 10Hashar: [C:03+2] releasing: Quibble 1.17.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268704 (owner: 10Phedenskog) [14:02:49] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Upgrade K3s cluster to most recent stable version - https://phabricator.wikimedia.org/T400077#11804561 (10jnuche) As a first step, I'll test locally and on `catalyst-dev` how risky the v1.28 -> v1.31 jump is [14:03:41] (03PS1) 10Hashar: release: Start 1.17.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269472 [14:03:55] (03CR) 10Hashar: [C:03+2] release: Start 1.17.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269472 (owner: 10Hashar) [14:06:00] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Upgrade K3s cluster to most recent stable version - https://phabricator.wikimedia.org/T400077#11804578 (10jnuche) [14:19:45] (03Merged) 10jenkins-bot: releasing: Quibble 1.17.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268704 (owner: 10Phedenskog) [14:22:00] (03Merged) 10jenkins-bot: release: Start 1.17.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269472 (owner: 10Hashar) [14:23:20] !log Tagged Quibble 1.17.0 @ 864381c6b63bdbcd8c74a3162c406fffcaaf8694 [14:23:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:25:28] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.17.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1269476 [14:26:06] (03CR) 10Jforrester: "Is there a task? What's new in Quibble 1.17?" [integration/config] - 10https://gerrit.wikimedia.org/r/1269476 (owner: 10Hashar) [14:26:48] James_F: I am pairing the releease with Peter & Zeljko :] feel free to join https://meet.google.com/tps-orzu-kzc ! [14:26:53] but good point yeah there is no changelog [14:27:02] outside of the quibble changelog.md [14:27:10] hashar: Sadly it's a Thursday so I'm in back-to-back meetings for 5 hours. :-( [14:27:21] yeah no worries :-) [14:27:35] Ihave skipped adding changelog to the commit messages / tag in interest of tag [14:27:43] but CHANGELOG.md has them phedenskog updated it this morning [14:27:44] \o/ [14:28:09] Of /course/ https://changelog.md/ is an actual site. And indeed, a product. [14:28:36] (03CR) 10Hashar: [C:03+2] dockerfiles: update Quibble to 1.17.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1269476 (owner: 10Hashar) [14:29:34] Looking at https://gerrit.wikimedia.org/r/c/integration/quibble/+/1268704 I think T419683 and T418743 are the ones I'm most happy about. [14:29:34] T419683: Quibble should report individual durations for browser tests npm install/test run - https://phabricator.wikimedia.org/T419683 [14:29:35] T418743: npm install run twice for core - https://phabricator.wikimedia.org/T418743 [14:30:21] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.17.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1269476 (owner: 10Hashar) [14:30:48] (03PS2) 10Ahmon Dancy: jjb: Bump quibble-coverage to 1.16.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1269070 (https://phabricator.wikimedia.org/T421689) [14:32:25] (03CR) 10Pwangai: "Sounds okay, working on it." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [14:33:07] (03CR) 10Ahmon Dancy: [C:03+2] jjb: Bump quibble-coverage to 1.16.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1269070 (https://phabricator.wikimedia.org/T421689) (owner: 10Ahmon Dancy) [14:34:48] (03Merged) 10jenkins-bot: jjb: Bump quibble-coverage to 1.16.0-s3 [integration/config] - 10https://gerrit.wikimedia.org/r/1269070 (https://phabricator.wikimedia.org/T421689) (owner: 10Ahmon Dancy) [14:35:09] dancy: have you fixed the php coverage script? :-]]] [14:35:30] that is quite great [14:36:21] Yeah [14:36:25] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 13Patch-For-Review: phalcongelist/php-diff has been deleted from GitHub, breaking builds of quibble-coverage - https://phabricator.wikimedia.org/T421689#11804783 (10dancy) 05Openβ†’03Resolved a:03dancy Old appearance: {F75418688} New appeara... [14:37:07] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 13Patch-For-Review: phalcongelist/php-diff has been deleted from GitHub, breaking builds of quibble-coverage - https://phabricator.wikimedia.org/T421689#11804796 (10hashar) That is fantastic! Thank you so much Ahmon. [14:39:06] (03PS1) 10Hashar: jjb: Upgrade Quibble jobs to 1.17.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1269482 [14:40:00] Quibble 1.17.0 images are being published to the registry [14:41:59] 10GitLab: GitLab Private Repository Request for: UA compliance Python Job Repo for Airflow-based pipeline - https://phabricator.wikimedia.org/T422183#11804827 (10KCVelaga_WMF) [14:54:41] 10Continuous-Integration-Config, 10phpunit-patch-coverage, 13Patch-For-Review: phalcongelist/php-diff has been deleted from GitHub, breaking builds of quibble-coverage - https://phabricator.wikimedia.org/T421689#11804910 (10Jdforrester-WMF) Thank you! [14:58:29] !log Upgrading Quibble jobs to 1.17.0 [14:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:12:50] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842 (10Daimona) 03NEW [15:13:07] !log Rolling back Quibble jobs to 1.16.0 (api-testing stage fails due to missing npm install step` [15:13:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:16] (03CR) 10Hashar: [C:03+2] jjb: Upgrade Quibble jobs to 1.17.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1269482 (owner: 10Hashar) [15:21:56] (03Merged) 10jenkins-bot: jjb: Upgrade Quibble jobs to 1.17.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1269482 (owner: 10Hashar) [15:23:32] (03PS1) 10Hashar: Revert "jjb: Upgrade Quibble jobs to 1.17.0" [integration/config] - 10https://gerrit.wikimedia.org/r/1269500 (https://phabricator.wikimedia.org/T422843) [15:24:04] (03CR) 10Hashar: [C:03+2] Revert "jjb: Upgrade Quibble jobs to 1.17.0" [integration/config] - 10https://gerrit.wikimedia.org/r/1269500 (https://phabricator.wikimedia.org/T422843) (owner: 10Hashar) [15:25:45] (03Merged) 10jenkins-bot: Revert "jjb: Upgrade Quibble jobs to 1.17.0" [integration/config] - 10https://gerrit.wikimedia.org/r/1269500 (https://phabricator.wikimedia.org/T422843) (owner: 10Hashar) [15:38:30] (03PS1) 10Phedenskog: api-testing: Make sure npm install runs for api-testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269507 (https://phabricator.wikimedia.org/T422843) [15:45:12] (03PS1) 10Phedenskog: api-testing: Add test plan for api testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269508 (https://phabricator.wikimedia.org/T422843) [15:53:20] (03Abandoned) 10Phedenskog: api-testing: Add test plan for api testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269508 (https://phabricator.wikimedia.org/T422843) (owner: 10Phedenskog) [15:55:50] so yeah I upgraded Quibble to 1.17.0 . That broke and thus rolled back [15:56:00] Peter is writing the patches above and we will revisit tomorrow morning (EU time) [15:56:03] \o/ [15:57:07] (03PS1) 10Phedenskog: api-testing: Add test plan for api testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269513 (https://phabricator.wikimedia.org/T422843) [15:58:15] (03PS2) 10Phedenskog: api-testing: Make sure npm install runs for api-testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269507 (https://phabricator.wikimedia.org/T422843) [16:01:56] (03CR) 10CI reject: [V:04-1] api-testing: Make sure npm install runs for api-testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269507 (https://phabricator.wikimedia.org/T422843) (owner: 10Phedenskog) [16:06:56] (03PS3) 10Phedenskog: api-testing: Make sure npm install runs for api-testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269507 (https://phabricator.wikimedia.org/T422843) [16:12:48] (03PS4) 10Phedenskog: api-testing: Make sure npm install runs for api-testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269507 (https://phabricator.wikimedia.org/T422843) [17:00:22] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform: Kafka-topics broken in beta: "zookeeper is not a recognized option" - https://phabricator.wikimedia.org/T422842#11805798 (10BPirkle) [17:10:20] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11805833 (10neriah) [17:12:09] (03update) 10dancy: backport.py: Process scap backport --revert changes in right order [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1115 (https://phabricator.wikimedia.org/T388057) [17:15:33] (03merge) 10dancy: backport.py: Process scap backport --revert changes in right order [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1115 (https://phabricator.wikimedia.org/T388057) [17:19:58] (03open) 10dancy: Release 4.246.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1123 [17:22:37] (03merge) 10dancy: Release 4.246.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1123 [18:07:38] 10Scap: scap backport --revert can't handle stacked commits - https://phabricator.wikimedia.org/T388057#11805999 (10dancy) 05Openβ†’03Resolved a:03dancy Deployed via scap 4.246.0 [18:31:13] (03open) 10dancy: ErrorLogsPage.vue: Handle some error message links better [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1124 [18:31:17] (03update) 10dancy: ErrorLogsPage.vue: Handle some error message links better [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1124 [18:34:46] (03merge) 10dancy: ErrorLogsPage.vue: Handle some error message links better [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1124 [18:35:19] (03update) 10dancy: Release 4.247.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1125 [18:35:21] (03open) 10dancy: Release 4.247.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1125 [18:38:36] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#11806072 (10Jdforrester-WMF) [18:41:31] (03merge) 10dancy: Release 4.247.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1125 [18:51:45] 10Phabricator: Add sortable headers for Related Changes in Gerrit - https://phabricator.wikimedia.org/T422867 (10Reedy) 03NEW [19:13:15] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Ghost4Man - https://phabricator.wikimedia.org/T422871 (10Ghost4Man) 03NEW [19:34:46] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Ghost4Man - https://phabricator.wikimedia.org/T422871#11806327 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Account approved. VΓ­tejte! [19:47:39] 10Continuous-Integration-Config, 10phan, 10phan-taint-check-plugin: Research use of pecl package phan_helpers for CI runs of phan - https://phabricator.wikimedia.org/T422876 (10Umherirrender) 03NEW [20:52:31] (03PS4) 10Pwangai: phpunit: Support parallel execution for core default suite [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) [20:57:08] (03PS1) 10Pwangai: Add test plan for core phpunit parallel tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269656 [20:59:53] (03CR) 10Pwangai: "Done" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1268686 (https://phabricator.wikimedia.org/T422108) (owner: 10Pwangai) [21:07:23] (03open) 10dancy: ErrorLogsPage.vue: Use rison-node library to encode the dashboard query [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1126 [21:07:26] (03update) 10dancy: ErrorLogsPage.vue: Use rison-node library to encode the dashboard query [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1126 [21:10:19] (03update) 10dancy: ErrorLogsPage.vue: Use rison-node library to encode the dashboard query [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1126 [21:21:11] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Automated Supervision test automation for CI - https://phabricator.wikimedia.org/T422555#11806638 (10SLong-WMF) [23:17:30] (03PS2) 10Pwangai: Add test plan for core phpunit parallel tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1269656 (https://phabricator.wikimedia.org/T422108)