[00:00:34] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:33] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:18:06] 10Beta-Cluster-Infrastructure: error.log is not rotated in beta - https://phabricator.wikimedia.org/T345566 (10Daimona) FTR, @hashar and I talked about this earlier today in #wikimedia-releng. If the guess above is correct, then it might be easier to just migrate the host to bullseye, which is part of T327742. [00:18:10] 10Beta-Cluster-Infrastructure, 10Cassandra, 10Beta-Cluster-reproducible, 10User-zeljkofilipin: Can not log in, log out, or save edits to the beta cluster (session failures) - https://phabricator.wikimedia.org/T324128 (10Daimona) 05Open→03Resolved Logins are currently working, hence resolving this task. [06:48:53] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-LoginNotify, 10Wikimedia-Site-requests, 10Patch-For-Review: Clarify LoginNotify emails are from beta - https://phabricator.wikimedia.org/T181908 (10tstarling) 05Open→03Resolved a:03tstarling Confirmed fixed. ` Subject: Multiple failed attempts to... [07:07:48] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T345590 (10LSobanski) p:05Triage→03Medium [07:15:41] 10Beta-Cluster-Infrastructure: error.log is not rotated in beta - https://phabricator.wikimedia.org/T345566 (10fgiunchedi) >>! In T345566#9140462, @Daimona wrote: > @herron @fgiunchedi This seems to be a consequence of T338127, woud you have any ideas as to what's going on here? My ineducated guess at the momen... [08:06:05] We need to make sure that deployers are available for backport/config changes during window. We can update if someone is not available. [09:32:36] (03CR) 10Hashar: [C: 03+2] jjb: update tox-mysqld jobs to Buster (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/954725 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [09:33:41] (03Merged) 10jenkins-bot: jjb: update tox-mysqld jobs to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954725 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [09:37:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: tox 4 no more support [tox:jenkins] configuration section - https://phabricator.wikimedia.org/T345607 (10hashar) [09:38:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: tox 4 no more support [tox:jenkins] configuration section - https://phabricator.wikimedia.org/T345607 (10hashar) [09:38:48] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) > labs-tools-wikinity-tox-docker to be investigated. That failure is an aftermath of the tox 4 upgrade (T3451... [09:40:50] (03PS3) 10Hashar: jjb: sample-workspace-sizes job from Stretch to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/954714 (https://phabricator.wikimedia.org/T278203) [09:42:18] (03CR) 10Hashar: [C: 03+2] jjb: sample-workspace-sizes job from Stretch to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/954714 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [09:43:22] (03Merged) 10jenkins-bot: jjb: sample-workspace-sizes job from Stretch to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/954714 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [10:36:36] (03PS2) 10Hashar: dockerfiles: remove tox image, replaced by tox-buster [integration/config] - 10https://gerrit.wikimedia.org/r/954713 (https://phabricator.wikimedia.org/T278203) [10:36:38] (03PS1) 10Hashar: dockerfiles: tox-poolcounter to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954895 (https://phabricator.wikimedia.org/T278203) [10:47:16] 10GitLab (CI & Job Runners), 10Data Engineering and Event Platform Team: Allow a shared, protected runner for the data-engineering group in GitLab - https://phabricator.wikimedia.org/T295045 (10BTullis) 05Open→03Declined This is no longer necessary, due to the work undertaken by the #release-engineering-te... [10:47:19] 10GitLab (Pipeline Services Migration🐤), 10Data Products, 10Metrics Platform Backlog: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10WDoranWMF) [11:54:45] (03PS3) 10Hashar: dockerfiles: remove tox image, replaced by tox-buster [integration/config] - 10https://gerrit.wikimedia.org/r/954713 (https://phabricator.wikimedia.org/T278203) [11:54:59] (03CR) 10Hashar: [C: 03+2] dockerfiles: tox-poolcounter to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954895 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [11:56:25] (03Merged) 10jenkins-bot: dockerfiles: tox-poolcounter to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954895 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [11:57:55] (03PS1) 10Hashar: jjb: update poolcounter tox job to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954906 (https://phabricator.wikimedia.org/T278203) [11:59:31] (03CR) 10Hashar: [C: 03+2] jjb: update poolcounter tox job to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954906 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [12:00:37] (03Merged) 10jenkins-bot: jjb: update poolcounter tox job to Buster [integration/config] - 10https://gerrit.wikimedia.org/r/954906 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [12:03:41] (03PS4) 10Hashar: dockerfiles: remove tox image, replaced by tox-buster [integration/config] - 10https://gerrit.wikimedia.org/r/954713 (https://phabricator.wikimedia.org/T278203) [12:04:28] (03PS3) 10Hashar: dockerfiles: remove ci-stretch, no more used [integration/config] - 10https://gerrit.wikimedia.org/r/954716 (https://phabricator.wikimedia.org/T278203) [12:04:38] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab (gitlab1004 -> gitlab2002) - September 2023 - https://phabricator.wikimedia.org/T345531 (10LSobanski) [12:04:47] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - September 2023 - https://phabricator.wikimedia.org/T345590 (10LSobanski) [12:05:48] (03CR) 10Hashar: [C: 03+2] "Pfff finally" [integration/config] - 10https://gerrit.wikimedia.org/r/954713 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [12:06:12] (03CR) 10Hashar: [C: 03+2] dockerfiles: remove ci-stretch, no more used [integration/config] - 10https://gerrit.wikimedia.org/r/954716 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [12:07:10] (03Merged) 10jenkins-bot: dockerfiles: remove tox image, replaced by tox-buster [integration/config] - 10https://gerrit.wikimedia.org/r/954713 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [12:07:14] (03Merged) 10jenkins-bot: dockerfiles: remove ci-stretch, no more used [integration/config] - 10https://gerrit.wikimedia.org/r/954716 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [12:09:02] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) 05Open→03Resolved a:03hashar Stretch images are gone from `integration/config.git`. [12:10:24] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab (gitlab1004 -> gitlab2002) - September 2023 - https://phabricator.wikimedia.org/T345531 (10LSobanski) [12:11:14] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - September 2023 - https://phabricator.wikimedia.org/T345590 (10LSobanski) [12:59:36] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Priority Backlog 📥), 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Move Toolforge PipelineLib repositories to GitLab - https://phabricator.wikimedia.org/T334399 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos... [13:26:59] 10GitLab (Pipeline Services Migration🐤), 10Research, 10collaboration-services: Move research webpages to gitlab - https://phabricator.wikimedia.org/T334511 (10fkaelin) 05In progress→03Resolved Closing this as resolved, created to update the documentation T345630 The repos are - https://gitlab.wikimedia... [13:27:05] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10fkaelin) [13:33:23] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for USER[S] - https://phabricator.wikimedia.org/T345631 (10valerio.bozzolan) [13:33:42] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for USER[S] - https://phabricator.wikimedia.org/T345631 (10valerio.bozzolan) [13:34:01] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for MauroBianchiInmagik - https://phabricator.wikimedia.org/T345631 (10valerio.bozzolan) [13:40:21] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Radar), 10Research: GitLab Issues experiment with Research - https://phabricator.wikimedia.org/T304614 (10fkaelin) 05Open→03Resolved A couple observations: - while initially I liked the idea of using gitlab issues to track work, i... [14:21:18] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10xcollazo) > Could you re-run the job and see what happens? Certainly. Will report in a few. [14:24:21] (03CR) 10Hashar: [C: 03+2] "Thank you! I have deployed them!" [integration/config] - 10https://gerrit.wikimedia.org/r/954333 (owner: 10Jforrester) [14:25:15] (03PS2) 10Hashar: jjb: Create basic node18 and node20 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/954333 (owner: 10Jforrester) [14:25:33] (03CR) 10Hashar: [C: 03+2] jjb: Create basic node18 and node20 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/954333 (owner: 10Jforrester) [14:27:02] (03Merged) 10jenkins-bot: jjb: Create basic node18 and node20 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/954333 (owner: 10Jforrester) [15:13:40] 10GitLab (Pipeline Services Migration🐤), 10Data Products, 10Metrics Platform Backlog: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10thcipriani) p:05Triage→03Medium Looks like we're running a series of test entrypoints (npm, composer, maven, and swift). #together c... [15:18:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10xcollazo) Ok, it seems like we had a transient issue indeed when I reproed a couple days ago. Both my jobs w... [15:20:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10xcollazo) Also, should we document at https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner ? [15:23:30] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10CodeReviewBot) xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-d... [16:00:19] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T343727 (10thcipriani) a:05brennen→03hashar [16:07:12] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for MauroBianchiInmagik - https://phabricator.wikimedia.org/T345631 (10thcipriani) [16:07:44] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for MauroBianchiInmagik - https://phabricator.wikimedia.org/T345631 (10thcipriani) 05Open→03Resolved a:03thcipriani Should be approved now. Please reopen this task if you have any issues logging in/accessing Git... [16:24:15] 10Phabricator, 10Patch-For-Review: API: `user.search` should output MediaWiki User - https://phabricator.wikimedia.org/T344303 (10jrbs) Thanks for the explainer :) Appreciate the patch! [16:48:29] 10WikimediaDebug, 10Observability-Logging, 10Performance-Team, 10Patch-For-Review: Ingest php syslog from Excimer UI (webperf host) into Logstash - https://phabricator.wikimedia.org/T339137 (10Krinkle) [17:52:45] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10CodeReviewBot) xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-du... [18:03:09] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10sbassett) [18:03:20] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Address issues within certain Gitlab CI security templates - https://phabricator.wikimedia.org/T338034 (10sbassett) [18:11:31] 10Phabricator (Upstream), 10Upstream: Allowing changing tags and move task on workboard of newly added tag in one go within the 'Edit Task' screen - https://phabricator.wikimedia.org/T161064 (10Aklapper) See also old upstream for this design decision at https://secure.phabricator.com/T6027 [18:12:39] (03PS1) 10Jforrester: Zuul: Add Mormegil to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/954977 [18:13:05] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Mormegil to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/954977 (owner: 10Jforrester) [18:14:11] (03Merged) 10jenkins-bot: Zuul: Add Mormegil to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/954977 (owner: 10Jforrester) [18:15:29] !log Zuul: Add Mormegil to CI allow list [18:15:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:38:58] (03PS2) 10Jforrester: jjb: Update node jobs to node16 images with npm 8.x not 7.x [integration/config] - 10https://gerrit.wikimedia.org/r/954331 (https://phabricator.wikimedia.org/T314470) [18:39:00] (03PS2) 10Jforrester: jjb: Update quibble jobs to node16 images with npm 8.x not 7.x [integration/config] - 10https://gerrit.wikimedia.org/r/954332 (https://phabricator.wikimedia.org/T314470) [18:39:02] (03PS2) 10Jforrester: Dockerfiles: Drop node14 images, unused [integration/config] - 10https://gerrit.wikimedia.org/r/954334 [18:39:04] (03PS2) 10Jforrester: [DNM] dockerfiles: [php-ast] Build a PHP 8.3 AST [integration/config] - 10https://gerrit.wikimedia.org/r/954336 [18:39:06] (03PS2) 10Jforrester: [DNM] dockerfiles: Provide php83 test image [integration/config] - 10https://gerrit.wikimedia.org/r/954337 [18:39:20] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/954331 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [18:39:27] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/954332 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [18:39:32] (03CR) 10Jforrester: [C: 03+2] Dockerfiles: Drop node14 images, unused [integration/config] - 10https://gerrit.wikimedia.org/r/954334 (owner: 10Jforrester) [18:41:01] (03Merged) 10jenkins-bot: jjb: Update node jobs to node16 images with npm 8.x not 7.x [integration/config] - 10https://gerrit.wikimedia.org/r/954331 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [18:41:04] (03Merged) 10jenkins-bot: jjb: Update quibble jobs to node16 images with npm 8.x not 7.x [integration/config] - 10https://gerrit.wikimedia.org/r/954332 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [18:41:06] (03Merged) 10jenkins-bot: Dockerfiles: Drop node14 images, unused [integration/config] - 10https://gerrit.wikimedia.org/r/954334 (owner: 10Jforrester) [20:22:55] James_F: please see -operations [20:26:19] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdlrobson) This may have caused T345660 [20:40:12] (03PS1) 10Jforrester: jjb: Temporarily hold back wmf-quibble jobs to npm7 not npm8 for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/954991 (https://phabricator.wikimedia.org/T345660) [20:40:54] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/954991 (https://phabricator.wikimedia.org/T345660) (owner: 10Jforrester) [20:42:01] (03Merged) 10jenkins-bot: jjb: Temporarily hold back wmf-quibble jobs to npm7 not npm8 for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/954991 (https://phabricator.wikimedia.org/T345660) (owner: 10Jforrester) [20:47:38] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T343727 (10matmarex) [21:04:59] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release Pipeline (Blubber): Refactor Blubber's BuildKit frontend gateway to use LLB directly - https://phabricator.wikimedia.org/T345458 (10CodeReviewBot) dduvall merged https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/5... [21:24:59] James_F: once Lexeme depends-on wikibase passes CI, I'll force land the stack [21:25:29] (03PS3) 10Jforrester: [DNM] dockerfiles: [php-ast] Build a PHP 8.3 AST [integration/config] - 10https://gerrit.wikimedia.org/r/954336 [21:25:32] how come this is only failing now? [21:26:07] Krinkle: Because npm7 was failing silently but npm8 fails with exit(1), it seems. [21:26:10] Sigh etc. [21:26:37] sure but didn't we roll out npm8 and node16 shortly after one another last month? [21:26:43] (Also boo that php-ast isn't ready for 8.3 yet so I can't push out an rc.0 image.) [21:26:58] The image, yes, the jobs, no, not until just a few hours ago. [21:27:04] I see. okay [21:27:11] Foolishly I only tested in 30 repos and not Wikibase. [21:27:44] Hmm, 954994 is failing in CI even with the dependency. [21:27:52] seems still red on 954994,PS2 [21:28:03] Yeah. [21:28:08] Different error this time. [21:28:25] This really needs WMDE to take ownership, I fear. [21:29:18] I'm going to bed. :-) [21:29:18] (03CR) 10CI reject: [V: 04-1] [DNM] dockerfiles: [php-ast] Build a PHP 8.3 AST [integration/config] - 10https://gerrit.wikimedia.org/r/954336 (owner: 10Jforrester) [21:29:53] ok, well, glad it was "only" the npm8 upgrade from today, so we should be green again on most repos [21:29:59] we can try agian once WMDE come through [21:30:12] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release Pipeline (Blubber): Refactor Blubber's BuildKit frontend gateway to use LLB directly - https://phabricator.wikimedia.org/T345458 (10CodeReviewBot) dduvall opened https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/5... [21:36:22] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release Pipeline (Blubber): Refactor Blubber's BuildKit frontend gateway to use LLB directly - https://phabricator.wikimedia.org/T345458 (10CodeReviewBot) dduvall closed https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/5... [22:41:59] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release Pipeline (Blubber): Refactor Blubber's BuildKit frontend gateway to use LLB directly - https://phabricator.wikimedia.org/T345458 (10CodeReviewBot) dduvall merged https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/5... [22:44:03] 10Release-Engineering-Team: Rethinking Code Review and Deployment of MediaWiki - https://phabricator.wikimedia.org/T345670 (10thcipriani) [22:49:30] 10Release-Engineering-Team: Rethinking Code Review and Deployment of MediaWiki - https://phabricator.wikimedia.org/T345670 (10jeena) [22:49:50] 10Release-Engineering-Team: Rethinking Code Review and Deployment of MediaWiki - https://phabricator.wikimedia.org/T345670 (10jeena) [22:50:14] 10Release-Engineering-Team: Rethinking Code Review and Deployment of MediaWiki - https://phabricator.wikimedia.org/T345670 (10thcipriani) Related to {T89945}—long-lived per-group deployments (rather than new branches each week). [22:51:14] 10Release-Engineering-Team: Rethinking Code Review and Deployment of MediaWiki - https://phabricator.wikimedia.org/T345670 (10jeena) [23:30:21] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [23:30:43] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [23:31:44] 10Release-Engineering-Team (Priority Backlog 📥), 10Release Pipeline (Blubber): Refactor Blubber's BuildKit frontend gateway to use LLB directly - https://phabricator.wikimedia.org/T345458 (10dduvall) An experimental native-LLB build of Blubber's BuildKit frontend has been published as `docker-registry.wikimedi... [23:31:52] 10Release-Engineering-Team (Priority Backlog 📥), 10Release Pipeline (Blubber): Refactor Blubber's BuildKit frontend gateway to use LLB directly - https://phabricator.wikimedia.org/T345458 (10dduvall) 05Open→03Resolved [23:31:54] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) [23:33:44] 10Release-Engineering-Team (Priority Backlog 📥), 10Epic, 10Release Pipeline (Blubber): Deprecate Blubber's CLI and microservice (blubberoid) interfaces - https://phabricator.wikimedia.org/T318289 (10dduvall) >>! In T318289#8404147, @bd808 wrote: > Another way to attempt to break the cycle would be to impleme...