[00:21:35] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work, and 2 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10741851 (10Scott_French) As a reminder to myself for later: there are a couple of spots in mediawiki/services/parsoid with `systemctl... [03:58:20] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 3 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10742054 (10Ryasmeen) [07:08:23] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10742372 (10elukey) >>! In T390251#10741315, @bd808 wrote: >>>! In T390251#10737321, @elukey wrote: >> @dancy what do you think about adding a configurable sleep in the use case that Scott pointed out?... [08:21:44] hi, I have a patch to allow releng to look at log for the weekly train presync and clean tasks and grant us the ability to start them. I am not sure whom in your team to ask for review :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1130947 [08:27:22] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935 (10hashar) 03NEW [08:27:50] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742603 (10hashar) [08:30:24] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742612 (10hashar) [08:30:28] hashar: that's actually a new sudo group, which used to require a review in the biweekly SRE meeting, IIRC it's I/F now that does that. I 'll add moritzm to the task. [08:30:43] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742619 (10hashar) p:05Triage→03Unbreak! That is blocking the train, thus setting up {nav Unbreak Now!} [08:31:04] akosiaris: ahh great, thank you for the triage! [08:31:48] and the overnight image building has failed to push the image to the registry so I have filed a task blocking the train: https://phabricator.wikimedia.org/T391935 [08:31:49] ah, wait, the group already exists, it's release-engineering. it's extra privileges we are talking about (sudo::group threw me off). I take that back, it's easier then [08:32:32] I am happy for the change to follow whatever process as long as it eventually get done :-] There is no rush [09:21:20] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742697 (10jnuche) I tried running the presync again earlier today. The same problem seemed to happen, with the image push stuck for around 14m before I aborted ma... [09:41:39] 06serviceops, 06Release-Engineering-Team: scap train-presync failed to push image: blob upload unknown - https://phabricator.wikimedia.org/T391935#10742760 (10Clement_Goubert) This is {T390251}, will take a look [11:41:08] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10743098 (10Clement_Goubert) I tried a `scap sync-world` (just to get an image push) and it did manage to push all images in around 15 minutes, but then failed pulling the image to the testserver-stage... [12:29:23] 06serviceops, 13Patch-For-Review: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10743379 (10Clement_Goubert) I took it upon myself to add that sleep as we are currently unable to deploy. [14:07:07] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10743894 (10MatthewVernon) >>! In T390251#10741708, @Scott_French wrote: > @MatthewVernon - When you get a chance, has anything changed recently - e.g., around the week of 24th of March - that might ha... [14:56:11] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744106 (10elukey) >>! In T390251#10743894, @MatthewVernon wrote: > > [apropos the deprecation of swift, we are bringing some production users to the apus Ceph cluster (which does S3), but that has a... [15:09:50] 06serviceops, 06Abstract Wikipedia team, 10function-orchestrator: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986 (10cmassaro) 03NEW [15:21:03] 06serviceops, 13Patch-For-Review: Move Wikikube services to Istio ingress (where possible) - https://phabricator.wikimedia.org/T391457#10744239 (10elukey) [15:28:58] 06serviceops, 10envoy, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10744276 (10akosiaris) After chatting on IRC with @fgiunchedi, the current understanding is that the default histogram buckets are at htt... [15:34:10] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744315 (10bd808) >>! In T390251#10742372, @elukey wrote: > Hi @bd808, I choose to read the sequence of questions as genuine interest and not as a passive aggressive comment, even if it very much look... [15:47:58] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744390 (10dancy) At this point we have two problems. * Large image pushes are now unreliable (this seems new for mediawiki deployments). No workaround proposed yet. * Large image pulls are unreliabl... [16:04:17] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work, 03Web Team Essential Work 2025: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10744450 (10Scott_French) Alright, parsoidtest1001 is now running PHP 8.1. I've verified that: 1. The test-specifi... [16:17:56] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744545 (10dduvall) >>! In T390251#10744390, @dancy wrote: > At this point we have two problems. > * Large image pushes are now unreliable (this seems new for mediawiki deployments). No workaround pr... [16:38:24] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744640 (10dancy) >>! In T390251#10744545, @dduvall wrote: >>>! In T390251#10744390, @dancy wrote: >> At this point we have two problems. >> * Large image pushes are now unreliable (this seems new for... [17:04:15] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744698 (10dduvall) >>! In T390251#10744640, @dancy wrote: > > In the case of uploads, here is the bad sequence: > > * The client (e.g. dockerd) issues `POST /v2//blob/uploads/` to initiate an... [17:06:07] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744699 (10dduvall) Also, I wonder if there's a way we can force monolithic uploads? [17:22:30] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744746 (10dduvall) >>! In T390251#10744699, @dduvall wrote: > Also, I wonder if there's a way we can force monolithic uploads? It seems that others have had this idea to solve this same issue and th... [17:24:39] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10744759 (10dduvall) Other possibly relevant discussions around this issue. https://github.com/distribution/distribution/issues/1013 https://github.com/distribution/distribution/pull/2441 [18:55:58] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10745119 (10Scott_French) Following up on discussion out of band about remaining hosts that install PHP 7.4...