[07:39:14] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11377644 (10Izubiaurre) > ...our current config, it says "Wikimedia sync by RepOSSync" updated... [07:46:01] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 06MediaWiki-Platform-Team, 10Projects-Cleanup, 05MW-1.39-notes: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11377674 (10A_smart_kitten) This may have also caused {T410243} [09:01:13] 10Phabricator, 06Data-Engineering: Data Engineering's Herald rule (H126) might not currently react to any tasks - https://phabricator.wikimedia.org/T410248 (10A_smart_kitten) 03NEW [09:04:34] 10Phabricator, 06Data-Engineering: Data Engineering's Herald rule (H126) might not currently react to any tasks - https://phabricator.wikimedia.org/T410248#11377811 (10Aklapper) Thanks for finding this. Indeed `Project tags incl their milestones` is a hack from T144041 and can only work with "include none of" [10:45:48] (03open) 10brouberol: Remove the repos/data-engineering/airflow repository [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/134 (https://phabricator.wikimedia.org/T408711) [10:58:43] (03update) 10brouberol: Remove the repos/data-engineering/airflow repository [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/134 (https://phabricator.wikimedia.org/T408711) [12:01:17] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 07Essential-Work: FR Herald rule H455 failed - https://phabricator.wikimedia.org/T399987#11378573 (10A_smart_kitten) > Should we rephrase to `Project tags incl their milestones (only use with Include None Of)`? Yeah, something like that might be sligh... [12:56:38] (03merge) 10hashar: DeploymentCalendar: add arnaudb to sre-collab [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/215 (owner: 10arnaudb) [12:57:44] (03update) 10hashar: DeploymentCalendar: add arnaudb to sre-collab [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/215 (owner: 10arnaudb) [13:01:57] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WandaScore] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205934 (owner: 10TechieNK) [13:01:59] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WandaScribe] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205933 (owner: 10TechieNK) [13:04:04] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WandaScore] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205934 (owner: 10TechieNK) [13:04:04] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/WandaScribe] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205933 (owner: 10TechieNK) [13:27:02] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/DrawioEditor] Add VisualEditorPlus as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1206374 [13:30:03] maintenance-disconnect-full-disks build 755413 integration-agent-docker-1045 (/: 36%, /srv: 95%, /var/lib/docker: 28%): OFFLINE due to disk space [13:35:03] maintenance-disconnect-full-disks build 755414 integration-agent-docker-1045 (/: 36%, /srv: 94%, /var/lib/docker: 27%): RECOVERY disk space OK [13:42:31] (03PS1) 10Robert Vogel: Add CI dependency to Extension:DrawioEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1206376 [13:57:52] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/WandaScribe] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205933 (owner: 10TechieNK) [13:59:57] (03Abandoned) 10Robert Vogel: Add CI dependency to Extension:DrawioEditor [integration/config] - 10https://gerrit.wikimedia.org/r/1206376 (owner: 10Robert Vogel) [14:02:15] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WandaScribe] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205933 (owner: 10TechieNK) [14:03:56] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WandaScribe] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/1205933 (owner: 10TechieNK) [14:09:23] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/DrawioEditor] Add VisualEditorPlus as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1206374 (owner: 10Hslater) [14:11:08] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DrawioEditor] Add VisualEditorPlus as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1206374 (owner: 10Hslater) [14:12:15] !log Zuul: [mediawiki/extensions/DrawioEditor] Add VisualEditorPlus as dependency [14:12:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:19:48] alright, let's see why we're getting so many duplicate donation message failmails [14:22:11] ugh, paypal failmail is still happening? [14:24:28] oops, talking to myself in the releng channel. I'll continue the monologue over in -fundraising :) [14:48:42] 06Project-Admins, 06Test-Platform: Create project for end-to-end tests - https://phabricator.wikimedia.org/T404950#11379132 (10zeljkofilipin) a:03SDunlap [14:49:22] 06Project-Admins, 06Test-Platform: Create project for end-to-end tests - https://phabricator.wikimedia.org/T404950#11379133 (10zeljkofilipin) Agreed to assign to @SDunlap during #tp0 meeting. [14:55:03] maintenance-disconnect-full-disks build 755430 integration-agent-docker-1056 (/: 36%, /srv: 99%, /var/lib/docker: 34%): OFFLINE due to disk space [14:58:03] hello! wikibase CI has been broken the whole day due to T410269 which was caused by a patch in CirrusSearch. I'm pretty sure that in order to fix it, we need to release a new version of the quibble bullseye image to include the latest version of the opensearch plugins. [14:58:03] T410269: Wikibase CI broken: Unknown filter type [truncate_norm] for [truncate_keyword] - https://phabricator.wikimedia.org/T410269 [14:58:20] dcausse or ebernhardson if you're around, could you take a quick look at the ticket ^ to confirm this? [14:58:32] sure [14:58:36] thank you! [15:00:03] maintenance-disconnect-full-disks build 755431 integration-agent-docker-1056 (/: 36%, /srv: 27%, /var/lib/docker: 34%): RECOVERY disk space OK [15:47:10] dcausse: thanks for the comment on the ticket. looks like hashar is not around right now to help with the new quibble image and I don't know how else to unblock CI for the two teams working on wikibase. would reverting https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1198219 be an option for you until we have the new quibble image available? [15:47:44] (03merge) 10dancy: Remove the repos/data-engineering/airflow repository [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/134 (https://phabricator.wikimedia.org/T408711) (owner: 10brouberol) [15:49:32] jakob_WMDE: we haven't started to reindex yet, so I think we can revert if there's no option on your side to silence the test [15:52:51] it's the UpdateSearchIndexConfig maintenance script that errors before any tests are even run so I don't think there's much we can do on our end :/ [15:53:31] :( [15:54:13] sent a revert [15:54:54] thank you! I'll put a depends-on to it on a wikibase patch to confirm that it would fix it [16:33:17] dcausse, jakob_WMDE: Is integration/quibble.git going to need a specific patch to bump to the new version? Or does it just float and any new image will work? [16:33:57] Looks like just a bump to the main CI images? [16:35:03] I think so, yes. any new image should work as long as the OpenSearch plugin is already on the apt registry (which dcausse probably knokws) [16:36:03] (03PS1) 10Jforrester: Docker: [quibble-bullseye] Bump for wmf-opensearch-search-plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1206410 (https://phabricator.wikimedia.org/T410269) [16:36:03] haven't seen any version being pin so I suppose it'll take the latest when rebuilding? [16:36:13] Ack. ^^ Does that look OK to you? [16:37:07] James_F: sounds good to me [16:37:12] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-bullseye] Bump for wmf-opensearch-search-plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1206410 (https://phabricator.wikimedia.org/T410269) (owner: 10Jforrester) [16:37:15] WFM. [16:37:35] thanks! [16:37:43] thanks James_F! [16:37:55] Of course. But thank me if it works. :-) [16:38:01] :) [16:38:03] hehe [16:38:11] for context, we just started mirroring the upstream opensearch deb repos last week in T407123 [16:38:12] T407123: Mirror OpenSearch repos from upstream - https://phabricator.wikimedia.org/T407123 [16:38:56] (03Merged) 10jenkins-bot: Docker: [quibble-bullseye] Bump for wmf-opensearch-search-plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1206410 (https://phabricator.wikimedia.org/T410269) (owner: 10Jforrester) [16:39:32] !log Docker: [quibble-bullseye] Bump for wmf-opensearch-search-plugins, for T410269 [16:39:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:39:35] T410269: Wikibase CI broken: Unknown filter type [truncate_norm] for [truncate_keyword] - https://phabricator.wikimedia.org/T410269 [16:41:01] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11379758 (10LSobanski) a:03Dzahn [16:41:27] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11379760 (10LSobanski) p:05Medium→03Low [16:43:03] (03PS1) 10Jforrester: jjb: Update Quibble-based jobs for new wmf-opensearch-search-plugins release [integration/config] - 10https://gerrit.wikimedia.org/r/1206414 (https://phabricator.wikimedia.org/T410269) [16:58:19] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update Magnum cluster config to use higher iops volume type and flavor - https://phabricator.wikimedia.org/T410290 (10bd808) 03NEW [17:00:31] (03CR) 10Jforrester: [C:03+2] jjb: Update Quibble-based jobs for new wmf-opensearch-search-plugins release [integration/config] - 10https://gerrit.wikimedia.org/r/1206414 (https://phabricator.wikimedia.org/T410269) (owner: 10Jforrester) [17:00:33] dcausse, jakob_WMDE: OK, now fully deployed. [17:00:44] So a re-apply should work? [17:00:56] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11379918 (10thcipriani) Hi all! I provided the list of projects in gitlab back in June of 202... [17:02:33] (03Merged) 10jenkins-bot: jjb: Update Quibble-based jobs for new wmf-opensearch-search-plugins release [integration/config] - 10https://gerrit.wikimedia.org/r/1206414 (https://phabricator.wikimedia.org/T410269) (owner: 10Jforrester) [17:03:33] James_F: I reverted the culprit patch in the meantime but will test with a depends-on [17:03:37] Ack. [17:06:18] ah Jakob is on it with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1206418 [17:06:39] yes, but I made it a few mins too early :| [17:07:05] jakob_WMDE: I'll abort the current CI runs. [17:07:34] Aha, you rebased. [17:07:44] s/rebase/edited the commit message/ [17:07:48] yup :) [17:12:12] yayy, it's working! thanks again James_F and dcausse! [17:13:14] np! will update our docs to update quibble next time, completely forgot that you had setup opensearch there couple month ago [17:13:52] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#11380009 (10Aklapper) I now disabled all IO on any non-observe or non-mirror repository URIs in Diff... [17:15:44] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 06MediaWiki-Platform-Team, 10Projects-Cleanup, 05MW-1.39-notes: Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11380029 (10Krinkle) >>! In T401155#11377674, @A_smart_kitten wrote: > This may have also caused {T41... [17:55:22] (03open) 10swfrench: Draft: Switch mediawiki_runtime_image to PHP 8.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1044 (https://phabricator.wikimedia.org/T405955) [17:56:24] !log Unblock 197.14.32.0/20 (T410218) [17:56:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:02:15] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11380420 (10Dzahn) I am thinking the ideal outcome would be if we stop maintaining a manual li... [18:04:48] 10GitLab, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10wikimedia.biterg.io: Bitergia crawler bot runs into errors collection GitLab data - https://phabricator.wikimedia.org/T409903#11380438 (10Izubiaurre) > It sounds like, many of the repos you're unable to access are delete... [18:47:52] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.26.1 released - https://phabricator.wikimedia.org/T410049#11380565 (10dancy) [21:12:36] (03update) 10dancy: Drop php_fpm_unsafe_restart_script support [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/864 (https://phabricator.wikimedia.org/T396166) [21:17:03] 06Release-Engineering-Team (Doing 😎), 10Scap, 07Essential-Work, 13Patch-For-Review: Are `php_fpm`/`php_version` inside `scap.cfg` used anymore? - https://phabricator.wikimedia.org/T396166#11381224 (10dancy) >>! In T396166#10952265, @CodeReviewBot wrote: > dancy **merged** https://gitlab.wikimedia.org/r... [21:20:48] (03open) 10dancy: changelog: Fix misleading changelog entry [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1045 (https://phabricator.wikimedia.org/T396166) [21:20:50] (03update) 10dancy: changelog: Fix misleading changelog entry [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1045 (https://phabricator.wikimedia.org/T396166) [21:23:01] (03merge) 10dancy: changelog: Fix misleading changelog entry [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1045 (https://phabricator.wikimedia.org/T396166) [21:38:18] 10Continuous-Integration-Infrastructure, 10Browser Test Platform, 10Testing Support, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#11381289 (10SDunlap) [21:38:35] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10Browser Test Platform, 10Testing Support: Global errors in JavaScript should fail Selenium Jenkins jobs - https://phabricator.wikimedia.org/T318902#11381299 (10SDunlap) [21:38:39] 10Beta-Cluster-Infrastructure, 10Browser Test Platform, 10Testing Support, 07OKR-Work: Make selenium users use botflags at beta-cluster by setting the bot flag (passing "bot=1" parameter) when saving edits - https://phabricator.wikimedia.org/T116027#11381302 (10SDunlap) [21:39:21] 10Continuous-Integration-Config, 10Browser Test Platform, 10Testing Support: Jenkins selenium job should fail when all tests are skipped - https://phabricator.wikimedia.org/T324480#11381311 (10SDunlap) [21:40:09] 10Beta-Cluster-Infrastructure, 10Browser Test Platform, 10Testing Support, 07Epic: Run Selenium tests targeting Beta cluster - https://phabricator.wikimedia.org/T373680#11381326 (10SDunlap) [21:40:19] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Browser Test Platform, and 4 others: CI jobs failing with various timeouts (March 2025) - https://phabricator.wikimedia.org/T388416#11381324 (10SDunlap) [21:41:44] 06Release-Engineering-Team (Seen), 10Scap, 10Browser Test Platform, 10Testing Support: Running smoke tests during deployment - https://phabricator.wikimedia.org/T187733#11381344 (10SDunlap) [21:41:52] 06Release-Engineering-Team (Radar), 10Browser Test Platform, 10Testing Support, 07Epic, 07Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683#11381350 (10SDunlap) [21:42:14] 10Continuous-Integration-Config, 10Browser Test Platform, 06Machine-Learning-Team, 10ORES, and 2 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939#11381346 (10SDunlap) [21:52:01] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Setup service name for Beta Cluster access to logstash service in logging project - https://phabricator.wikimedia.org/T409363#11381394 (10bd808) 05Stalled→03Open [21:54:43] 10Beta-Cluster-Infrastructure, 10Scap, 10Observability-Logging: Setup service name for Beta Cluster access to logstash service in logging project - https://phabricator.wikimedia.org/T409363#11381403 (10bd808) @colewhite, I have the technical rights needed to do this, but maybe it would work better in the lon... [22:38:35] (03PS1) 10Aude: Zuul: [mediawiki/extensions/WP25EasterEggs] Add CommunityConfiguration dep [integration/config] - 10https://gerrit.wikimedia.org/r/1206469 (https://phabricator.wikimedia.org/T408435) [23:36:44] (03open) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72 [23:36:49] (03update) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72 [23:40:34] (03update) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72 [23:46:28] (03update) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72 [23:49:45] (03update) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72 [23:53:49] (03update) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72 [23:55:43] (03update) 10dduvall: Refactor to use `.kokkuri:bake` [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/72