[00:01:55] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:55] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:33:52] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Tweak requests for the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T373952#10151281 (10Dzahn) >>! In T373952#10140525, @Aklapper wrote: > That does not mean it'll work in production..... [01:25:26] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 06collaboration-services: releases1003 file system over 90% full - https://phabricator.wikimedia.org/T368239#10151370 (10Dzahn) I will soon respond to the other points, but for now let me just say: currently we only use 46% of... [02:31:41] 10Gerrit, 06Release-Engineering-Team: Link with ampersand is broken in commit messages - https://phabricator.wikimedia.org/T374772#10151442 (10Krinkle) Thanks @Paladox. I see the backports have landed at: https://gerrit-review.googlesource.com/q/I5d3fb84f5533a59ba041eef806d148a6f960c5c7 Nice! [07:48:48] (03approved) 10jnuche: releasing/plugins.txt: Remove deprecated greenballs plugin [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/82 (https://phabricator.wikimedia.org/T328687) (owner: 10dancy) [07:58:53] 10Deployments, 06Release-Engineering-Team, 06serviceops: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907 (10hashar) 03NEW [08:06:00] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10observability, 07Developer Productivity, and 2 others: Consider enabling all MW log channels by default for WMF - https://phabricator.wikimedia.org/T228838#10151732 (10hashar) Yesterday I have modified the [[ https://grafana.wikimedia.org/d/00... [08:06:38] (03CR) 10Jaime Nuche: [C:03+2] scap/checks.yaml: Combine two installation step checks into one [releng/phatality] - 10https://gerrit.wikimedia.org/r/1073286 (https://phabricator.wikimedia.org/T374880) (owner: 10Ahmon Dancy) [08:06:58] (03Merged) 10jenkins-bot: scap/checks.yaml: Combine two installation step checks into one [releng/phatality] - 10https://gerrit.wikimedia.org/r/1073286 (https://phabricator.wikimedia.org/T374880) (owner: 10Ahmon Dancy) [09:19:41] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Tweak requests for the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T373952#10152018 (10Aklapper) 05Open→03Resolved p:05Triage→03Medium a:03Aklapper @dzahn: Yay, nice surprise, thanks a lot for... [09:31:23] (03approved) 10jnuche: gerrit.py: Handle plain change number in GerritSession.change_number_from_url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/446 (owner: 10dancy) [10:11:31] 10Gerrit, 06Release-Engineering-Team, 07Upstream: Link with ampersand is broken in commit messages - https://phabricator.wikimedia.org/T374772#10152312 (10Aklapper) [10:45:20] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [10:52:50] 10Continuous-Integration-Config, 10Diffusion, 10Phabricator: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921#10152475 (10Lucas_Werkmeister_WMDE) [10:55:54] (03PS1) 10Elukey: zuul: move chartmuseum and helm3 to debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 [10:57:26] (03PS2) 10Elukey: zuul: move chartmuseum and helm3 to debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 [10:58:12] (03PS3) 10Elukey: zuul: move chartmuseum and helm3 to debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 [10:58:44] (03PS4) 10Elukey: zuul: move chartmuseum and helm3 to debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 [11:00:07] (03CR) 10Effie Mouzeli: "Thank you for taking care of this!" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) (owner: 10Effie Mouzeli) [11:17:55] (03CR) 10Muehlenhoff: [C:03+1] "LGTM (and in addition the current images are using buster)" [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 (owner: 10Elukey) [11:18:39] 06Project-Admins: Create project tag for Metrics Platform - https://phabricator.wikimedia.org/T372368#10152582 (10Aklapper) 05Open→03Stalled @cjming, @VirginiaPoundstone: Could someone please answer the last comment? Thanks in advance! It also remains unclear to me what should happen with tasks on the curren... [11:24:07] 06Project-Admins: umbrella project for Wikimedia Ukraine - https://phabricator.wikimedia.org/T374503#10152587 (10Aklapper) > as its subproject already There is no way to "move" projects but we can create a new tech subproject under the new umbrella, and then move tasks from the old tech project and then archiv... [11:25:50] (03CR) 10Kosta Harlan: [C:03+2] Add support for core's composer `phpunit:...parallel:...` commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [11:39:09] 06Project-Admins: umbrella project for Wikimedia Ukraine - https://phabricator.wikimedia.org/T374503#10152638 (10Aklapper) p:05Triage→03Medium a:03Aklapper [11:43:40] (03Merged) 10jenkins-bot: Add support for core's composer `phpunit:...parallel:...` commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1039669 (https://phabricator.wikimedia.org/T365976) (owner: 10Arthur taylor) [11:51:09] (03PS1) 10Kosta Harlan: release: Quibble 1.10.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1073438 (https://phabricator.wikimedia.org/T365976) [11:52:35] (03PS1) 10Kosta Harlan: release: Start 1.10.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1073439 [11:56:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Speed up git clone operation in wmf-quibble job - https://phabricator.wikimedia.org/T374717#10152685 (10kostajh) @ArthurTaylor and I had noticed this and were discussing. In addition to pruning branches, is it possible that using clone depth... [12:01:49] (03open) 10jnuche: allow deletion of environments when in read-only mode [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/631 (https://phabricator.wikimedia.org/T374285) [12:02:04] (03update) 10jnuche: allow deletion of environments when in read-only mode [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/631 (https://phabricator.wikimedia.org/T374285) [12:07:01] (03update) 10jnuche: allow deletion of environments when in read-only mode [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/631 (https://phabricator.wikimedia.org/T374285) [12:17:06] 10Deployments, 06Release-Engineering-Team, 06serviceops: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907#10152792 (10akosiaris) Looking [back in time a bit](https://logstash.wikimedia.org/goto/65381bdae8cad296c81a7ccd9b453e46) this isn... [13:11:53] 10Deployments, 06Release-Engineering-Team, 06serviceops: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907#10153052 (10hashar) I filed the task after the first deploy of the day, so maybe something was invalidated over night such as the... [13:35:09] 10Gerrit, 10Catalyst (PatchDemo): Update gerrit bot for PatchDemo - https://phabricator.wikimedia.org/T374954 (10Esanders) 03NEW [14:27:24] (03PS2) 10Hashar: Stop flagging a conflict for same file modifications [mediawiki-config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1073461 [14:29:48] Anyone is aware of any ongoing issues with gitlab? I have posted the issues I'm seeing on the -gitlab channel but got no reply in >1h [14:45:12] discussed in -gitlab and private already. it's only about DO runners, using our own infra runners would not be affected. [14:50:11] 10Scap, 10Phatality: scap phatality deployment problem - https://phabricator.wikimedia.org/T374880#10153585 (10dancy) a:03dancy [14:51:08] (03CR) 10Hashar: [C:04-2] "I have raised the topic on our internal `ops` mailing list. Lets wait next week before enabling content merge." [mediawiki-config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1073461 (owner: 10Hashar) [14:54:01] (03CR) 10Hashar: [C:03+2] zuul: move chartmuseum and helm3 to debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 (owner: 10Elukey) [14:54:38] 10Scap, 10Phatality: scap phatality deployment problem - https://phabricator.wikimedia.org/T374880#10153611 (10dancy) 05Open→03Resolved Considering this resolved after https://gerrit.wikimedia.org/r/1073286. However I do have some remaining suspicions about the rollback behavior should `scap deploy` f... [14:55:18] (03Merged) 10jenkins-bot: zuul: move chartmuseum and helm3 to debian-glue-non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 (owner: 10Elukey) [15:04:47] (03CR) 10Hashar: [C:03+2] "> LGTM (and in addition the current images are using buster)" [integration/config] - 10https://gerrit.wikimedia.org/r/1073426 (owner: 10Elukey) [15:05:36] (03CR) 10Lucas Werkmeister (WMDE): "gate-and-submit will run against the rebased version of the change, right?" [mediawiki-config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1073461 (owner: 10Hashar) [15:11:24] (03merge) 10dancy: gerrit.py: Handle plain change number in GerritSession.change_number_from_url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/446 [15:19:15] (03merge) 10dancy: releasing/plugins.txt: Remove deprecated greenballs plugin [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/82 (https://phabricator.wikimedia.org/T328687) [15:27:07] Project mediawiki-core-doxygen build #4304: 04FAILURE in 9 min 5 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/4304/ [15:27:07] Project mwcore-phpunit-coverage-master build #3851: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/3851/ [15:27:09] Project beta-code-update-eqiad build #513575: 04FAILURE in 4 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/513575/ [15:27:09] Project beta-update-databases-eqiad build #78974: 04FAILURE in 7 min 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/78974/ [15:28:17] !log Manually reconnecting Jenkins agents that got disconnected and can't reconnect for X reason [15:29:37] ah [15:29:43] well I guess the network failed on WMCS :] [15:31:35] (03update) 10dancy: test-requirements.txt: Add pytest-subtests [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/445 [15:31:37] (03CR) 10Kosta Harlan: [C:03+1] dockerfiles: update Quibble to 1.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1073476 (https://phabricator.wikimedia.org/T365976) (owner: 10Hashar) [15:31:39] (03update) 10dancy: test-requirements.txt: Add pytest-subtests [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/445 [15:31:50] (03PS1) 10Subramanya Sastry: Add jawikivoyage specific CSS hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1073477 [15:32:17] hashar: it did, hopefully working now [15:32:44] yup :) [15:34:52] 10Deployments, 06Release-Engineering-Team, 06serviceops: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907#10153807 (10akosiaris) >>! In T374907#10153052, @hashar wrote: > For the OCI image being pulled, scap has a step described as //d... [15:35:35] Yippee, build fixed! [15:35:36] Project beta-code-update-eqiad build #513576: 09FIXED in 2 min 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/513576/ [15:35:57] happy hacking! [15:36:40] (03merge) 10dancy: test-requirements.txt: Add pytest-subtests [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/445 [15:36:52] Thanks for that mediawiki-config Gerrit config change Hashar! Good idea. [15:39:57] dancy: the alternative is to split InitializeSettings.php and CommonSettings.php in small php snippets [15:40:01] and compile them to a single file [15:40:04] but well hmm [15:40:26] so yeah less conflicts, hopefully [15:50:25] (03Merged) 10jenkins-bot: release: Quibble 1.10.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1073438 (https://phabricator.wikimedia.org/T365976) (owner: 10Kosta Harlan) [15:50:27] (03Merged) 10jenkins-bot: release: Start 1.10.1 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1073439 (owner: 10Kosta Harlan) [15:56:57] !log Tag Quibble 1.10.0 @ 2dd562aa75aca65413d009e3e382a0a67ed328d7 # T50217 T365976 [15:57:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:01] T50217: Speed up MediaWiki PHPUnit build by running integration tests in parallel - https://phabricator.wikimedia.org/T50217 [15:57:01] T365976: [REPO][CLIENT][SW] Create a `composer phpunit:prepare-parallel-test-suites` command to split suite into smaller groups - https://phabricator.wikimedia.org/T365976 [15:57:37] (03CR) 10Hashar: [C:03+2] "I have tagged Quibble 1.10.0" [integration/config] - 10https://gerrit.wikimedia.org/r/1073476 (https://phabricator.wikimedia.org/T365976) (owner: 10Hashar) [15:59:01] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1073476 (https://phabricator.wikimedia.org/T365976) (owner: 10Hashar) [16:00:14] kostajh: I guess I will flip the jobs to Quibble 1.10.0 tomorrow [16:05:15] (03CR) 10Kosta Harlan: [C:03+1] "thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/1073476 (https://phabricator.wikimedia.org/T365976) (owner: 10Hashar) [16:16:54] images are uploading :) [16:28:44] Yippee, build fixed! [16:28:44] Project mediawiki-core-doxygen build #4305: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/4305/ [16:28:46] 06Release-Engineering-Team, 10Scap: Separate scap client and server distributions - https://phabricator.wikimedia.org/T374983 (10dancy) 03NEW [16:29:25] 06Release-Engineering-Team, 10Scap: Separate scap client and server distributions - https://phabricator.wikimedia.org/T374983#10154131 (10dancy) [16:32:33] Yippee, build fixed! [16:32:33] Project beta-update-databases-eqiad build #78975: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/78975/ [16:36:41] (03CR) 10Ahmon Dancy: [C:03+2] "I've submitted the updated WikimediaDebug.zip to the Chrome Web Store. Pending approval. Still researching the Firefox side of things." [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1070275 (https://phabricator.wikimedia.org/T371537) (owner: 10Effie Mouzeli) [16:46:19] 06Gerrit-Privilege-Requests, 13Patch-For-Review: Give Edesol +2 on Extension:Add_HTML_Meta_and_Title in Gerrit - https://phabricator.wikimedia.org/T372073#10154191 (10Pppery) The consequence of that is that you will lose automatic libraryupgrader commits and have to repoint translatewiki, and I'll have to proc... [17:01:18] 06Release-Engineering-Team, 10MediaWiki-extensions-StopForumSpam, 07ci-test-error, 10MW-1.43-notes (1.43.0-wmf.24; 2024-09-24): api.php?format=json (FormatJson.php) Allowed memory size fatal error in quibble-vendor-mysql-php74-selenium-docker - https://phabricator.wikimedia.org/T353001#10154269 (10sbass... [17:31:00] (03open) 10dduvall: jobs: Refactor `publishMWSingleVersion` to use `scap build-images` [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (https://phabricator.wikimedia.org/T369115) [17:31:02] (03update) 10dduvall: jobs: Refactor `publishMWSingleVersion` to use `scap build-images` [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (https://phabricator.wikimedia.org/T369115) [17:31:16] (03update) 10dduvall: jobs: Refactor `publishMWSingleVersion` to use `scap build-images` [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (https://phabricator.wikimedia.org/T369115) [17:31:18] (03update) 10dduvall: jobs: Refactor `publishMWSingleVersion` to use `scap build-images` [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (https://phabricator.wikimedia.org/T369115) [17:35:21] 06Gerrit-Privilege-Requests, 13Patch-For-Review: Give Edesol +2 on Extension:Add_HTML_Meta_and_Title in Gerrit - https://phabricator.wikimedia.org/T372073#10154428 (10Edesol) @Pppery i am good on the repo ownership front. rly a none-issue. but why is the Patchset still dangling? the tests run successful now.... [17:41:45] (03approved) 10dancy: jobs: Refactor `publishMWSingleVersion` to use `scap build-images` [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (https://phabricator.wikimedia.org/T369115) (owner: 10dduvall) [17:42:14] (03merge) 10dduvall: jobs: Refactor `publishMWSingleVersion` to use `scap build-images` [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/83 (https://phabricator.wikimedia.org/T369115) [18:11:50] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319#10154565 (10Krinkle) Seen again during several recent changes including: * https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Kartographer/+/107168... [18:23:44] 06Gerrit-Privilege-Requests, 13Patch-For-Review: Give Edesol +2 on Extension:Add_HTML_Meta_and_Title in Gerrit - https://phabricator.wikimedia.org/T372073#10154619 (10Pppery) Someone would need to approve the patch by giving it a Code-Review +2 for it to be merged, Anyone in the [[ https://gerrit.wikimedia.org... [18:25:40] 06Release-Engineering-Team, 10WikimediaDebug, 13Patch-For-Review: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10154620 (10dancy) [18:39:26] 06Release-Engineering-Team, 10WikimediaDebug, 13Patch-For-Review: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10154696 (10dancy) [19:05:28] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10observability, 07Developer Productivity, and 2 others: Consider enabling all MW log channels by default for WMF - https://phabricator.wikimedia.org/T228838#10154823 (10matmarex) Thanks for deploying all these patches @hashar! I had another lo... [19:33:24] 06Gerrit-Privilege-Requests, 13Patch-For-Review: Give Edesol +2 on Extension:Add_HTML_Meta_and_Title in Gerrit - https://phabricator.wikimedia.org/T372073#10154872 (10Edesol) any suggestion on how to raise interest in the patch set? do i need to open a new task? [19:41:10] (03approved) 10kindrobot: allow deletion of environments when in read-only mode [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/631 (https://phabricator.wikimedia.org/T374285) (owner: 10jnuche) [20:11:02] (03PS1) 10Hashar: jjb: switch jobs to Quibble 1.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1073542 (https://phabricator.wikimedia.org/T365976) [20:12:49] (03CR) 10CI reject: [V:04-1] jjb: switch jobs to Quibble 1.10.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1073542 (https://phabricator.wikimedia.org/T365976) (owner: 10Hashar) [20:29:09] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10observability, 07Developer Productivity, and 2 others: Consider enabling all MW log channels by default for WMF - https://phabricator.wikimedia.org/T228838#10155066 (10matmarex) [20:57:51] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10observability, 07Developer Productivity, and 2 others: Consider enabling all MW log channels by default for WMF - https://phabricator.wikimedia.org/T228838#10155276 (10hashar) 05Open→03Resolved I think that concludes the five years adv... [21:43:21] 10Release-Engineering-Team (Seen), 06Infrastructure-Foundations, 06SRE, 10SRE-tools: Support running puppet Beaker on CI - https://phabricator.wikimedia.org/T253635#10155446 (10hashar) Acceptance tests run by CI would be quite nice to have, specially for non SRE since that helps build confidence a given pa...