[07:19:11] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work, 03Web Team Essential Work 2025: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10737220 (10MoritzMuehlenhoff) >>! In T380485#10731980, @Scott_French wrote: > Thus, in the very near term, we may... [07:54:47] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10737321 (10elukey) @dancy what do you think about adding a configurable sleep in the use case that Scott pointed out? We could try with 300s and 600s in my opinion, very painful but probably needed at... [08:41:48] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10737484 (10kostajh) Starting with Slack seems fine, we may want to switch to Phab later. Thanks for your work on this. [09:06:18] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, and 2 others: Create a cookbook to automate gerrit's switchover - https://phabricator.wikimedia.org/T260666#10737609 (10ABran-WMF) This first iteration is still fairly manual but will give us a stepping stone to build upon. I'll... [10:26:36] 06serviceops, 10WMF-JobQueue: Create mw-jobrunner and general job running documentation - https://phabricator.wikimedia.org/T391809 (10hnowlan) 03NEW [10:26:44] 06serviceops, 10WMF-JobQueue: Create mw-jobrunner and general job running documentation - https://phabricator.wikimedia.org/T391809#10737963 (10hnowlan) p:05Triage→03Medium [10:54:44] 06serviceops, 10MW-on-K8s, 06SRE Observability, 13Patch-For-Review: Deploy statsd-exporter to mw-cron - https://phabricator.wikimedia.org/T391672#10738050 (10Clement_Goubert) 05In progress→03Resolved {F59060683} Metrics look like they're being sent through the statsd-exporter correctly (at least t... [11:26:17] 06serviceops, 06MediaWiki-Platform-Team: Migrate "startupregistrystats" maintenance script to k8s-mw-cron (mediawiki-platform-team) - https://phabricator.wikimedia.org/T388540#10738168 (10Clement_Goubert) >>! In T388540#10735954, @Krinkle wrote: >>>! In T388540#10694039, @Clement_Goubert wrote: >> […] [[ https... [12:13:13] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: [SW] [WBQC] shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#10738323 (10karapayneWMDE) a:03JMeybohm In https://phabricator.wikimedia.org/T380751, we updated format constr... [12:46:47] 06serviceops, 10envoy, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10738561 (10fgiunchedi) [12:53:41] 06serviceops, 10MW-on-K8s, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10738603 (10jijiki) [13:02:44] 06serviceops, 10envoy, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10738648 (10fgiunchedi) [13:56:49] 06serviceops, 10MW-on-K8s, 06SRE, 13Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791#10739003 (10hnowlan) 05Open→03Resolved All jobrunner hardware decommissioned or reclaimed, services torn down, puppet cleaned up. [16:47:37] 06serviceops, 06MW-Interfaces-Team, 10WMF-JobQueue, 07Documentation: Create mw-jobrunner and general job running documentation - https://phabricator.wikimedia.org/T391809#10740051 (10A_smart_kitten) [18:13:11] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work, and 2 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10740652 (10Scott_French) Thanks, @MoritzMuehlenhoff - I actually hadn't even checked yet and just assumed a reimage would be necessar... [19:00:19] 06serviceops, 13Patch-For-Review: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#10740878 (10Scott_French) 05Open→03In progress Alright, we're now free to rename and refactor `mw-php-migration.lua` et al. [19:00:33] 06serviceops, 13Patch-For-Review: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#10740882 (10Scott_French) [20:26:38] 06serviceops, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, and 2 others: Gradually isolate mediawiki authentication code and infrastructure - https://phabricator.wikimedia.org/T391784#10741283 (10Tgr) > Thanks to SUL3, now login mostly happens on auth.wikimed... [20:36:33] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10741310 (10Scott_French) One additional point of note that was puzzling me before: When a blob is committed during the terminating `PUT` of the upload, I've not seen any cases where verification of t... [20:39:00] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10741315 (10bd808) >>! In T390251#10737321, @elukey wrote: > @dancy what do you think about adding a configurable sleep in the use case that Scott pointed out? We could try with 300s and 600s in my opi... [20:42:07] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10741332 (10bd808) Can we test the delay to reach consensus hypothesis in a controlled experiment rather than as part of the live train and backports system by repeatedly publishing and pulling a large... [21:07:15] 06serviceops, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, and 2 others: Gradually isolate mediawiki authentication code and infrastructure - https://phabricator.wikimedia.org/T391784#10741436 (10Tgr) The short-term and medium-term proposal seems swapped to m... [21:09:11] 06serviceops, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, and 3 others: Gradually isolate mediawiki authentication code and infrastructure - https://phabricator.wikimedia.org/T391784#10741454 (10sbassett) [21:54:30] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10741610 (10dancy) I'm working on something that will attempt to validate that an image is downloadable. If that works, I'll insert it after the build process so that it imparts the minimum necessary... [23:02:34] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10741709 (10Scott_French) +1 to @dancy's idea to poll for a usable / valid image, as a vastly preferable workaround to adding a sleep. Following up on my footnote in T390251#10741310 about `blob inval...