[00:23:21] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581161 (10Scott_French) Thanks, @tstarling! So, if I'm understanding the changes in https://gerrit.wikimedia.org/r/1122588 correctly, it looks like `quibble... [00:29:37] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581178 (10Scott_French) Yes, it looks like https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/dockerfiles/quibble-bullseye... [01:02:49] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581278 (10Scott_French) Alright, I believe merging https://gerrit.wikimedia.org/r/1122700, rebuilding the images, and then merging https://gerrit.wikimedia.o... [01:17:50] 06serviceops, 10MediaWiki-Core-HTTP-Cache, 10MW-on-K8s: mwscript-k8s purgeList does not reliably purge cached URLs - https://phabricator.wikimedia.org/T387127#10581295 (10Pppery) [01:21:45] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581298 (10Scott_French) Confirmed that `docker-registry.wikimedia.org/releng/quibble-bullseye-php81:1.12.0-s3` now has the expected PCRE2 (10.42). [02:12:07] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10581357 (10Scott_French) [02:24:23] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581361 (10Scott_French) Alright, many thanks to @Reedy for the image rebuilds following https://gerrit.wikimedia.org/r/1122700 and @bd808 for updating the jo... [02:56:00] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581371 (10Reedy) >>! In T386006#10581278, @Scott_French wrote: > However, that's about the limit of my knowledge in terms of how CI image are built, and am s... [03:02:53] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581376 (10Reedy) And as a followup from the conversation in -releng as this was all going on xref {T320818} https://gerrit.wikimedia.org/r/c/integration/co... [03:42:28] 06serviceops, 10MW-on-K8s: Kubernetes maintenance script status UI - https://phabricator.wikimedia.org/T387268 (10RLazarus) 03NEW [03:43:05] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10581444 (10RLazarus) >>! In T341553#10574262, @Urbanecm_WMF wrote: > Hello! Can we please please find something better to view running jobs? Moved this to T387268. [08:15:58] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 07Code-Health-Objective: hewiki: Use backing node service instead of RESTBase on pregeneration changeprop rules - https://phabricator.wikimedia.org/T372749#10581705 (10Jgiannelos) Metrics overall look OK. Using... [08:45:42] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581763 (10hashar) >>! In T386006#10581278, @Scott_French wrote: > Alright, I believe merging https://gerrit.wikimedia.org/r/1122700, rebuilding the images, a... [08:52:02] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Pregeneration performance optimizations for PCS - https://phabricator.wikimedia.org/T386919#10581773 (10Jgiannelos) After discussing with @Ottomata and @Joe it sounds like a good idea to add more fields if we need them. Using meta though... [09:14:04] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581830 (10hashar) We have the same issue on #beta-cluster-infrastructure: > Warning: PHP Startup: ^(text/|application/xhtml\+xml) (offset=0): unrecognised co... [09:17:22] 06serviceops, 10Page Content Service: Rollout more wikis after week 1 of testing with production traffic - https://phabricator.wikimedia.org/T387277 (10Jgiannelos) 03NEW [09:23:33] 06serviceops, 10Page Content Service, 13Patch-For-Review: Rollout more wikis after week 1 of testing with production traffic - https://phabricator.wikimedia.org/T387277#10581858 (10Jgiannelos) @Seddon Any objections on the list? I wasn't sure if there is a language specific experiment happening that could be... [09:51:22] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10581911 (10MoritzMuehlenhoff) >>! In T386006#10577357, @Scott_French wrote: > The motivation behind using a standalone component for PCRE2, vs. including the... [10:55:50] 06serviceops, 10MW-on-K8s, 06SRE, 10Release-Engineering-Team (Priority Backlog 📥): Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10582102 (10JMeybohm) I stumbled upon this again recently and I think the current configuration does not allow pod creation at... [10:57:50] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429#10582122 (10JMeybohm) Indeed we can't use the same cert for both kubeconfig files given the different permissions required. Unfortunately the second parameter to `... [11:00:30] 06serviceops: Install prometheus-bird-exporter on k8s clusters - https://phabricator.wikimedia.org/T387288 (10ayounsi) 03NEW [11:01:35] 06serviceops: Install prometheus-bird-exporter on k8s clusters - https://phabricator.wikimedia.org/T387288#10582149 (10ayounsi) [11:02:25] 06serviceops: Add prometheus-bird-exporter sidecar to calico-node pods - https://phabricator.wikimedia.org/T387288#10582154 (10JMeybohm) [13:25:56] 06serviceops: New upstream release for imposm3 - https://phabricator.wikimedia.org/T363034#10582778 (10TheDJ) Adding serviceops. Some of these changes seem like they could improve efficiency of thumbnailing. [13:27:33] 06serviceops, 10Maps (Maps-data): New upstream release for imposm3 - https://phabricator.wikimedia.org/T363034#10582785 (10TheDJ) [13:33:45] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 05Wikifunctions Improve performance: increase CPU and Node heap limit? - https://phabricator.wikimedia.org/T385859#10582834 (10akosiaris) > Increasing the Orchestrator's CPU limit to 600ms rather than the current 400ms? The... [13:42:34] 06serviceops, 10CX-cxserver, 10LPL Essential (LPL Essential 2025 Feb-Mar), 13Patch-For-Review, 07Technical-Debt: Use openapi compliant examples in swagger spec - https://phabricator.wikimedia.org/T382294#10582873 (10akosiaris) [13:51:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Add prometheus-bird-exporter sidecar to calico-node pods - https://phabricator.wikimedia.org/T387288#10582930 (10JMeybohm) [15:22:39] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10583449 (10Scott_French) Thanks, all, for the follow-up. @hashar - Many thanks for addressing the issue in Beta. My understanding - which of course might be... [15:35:39] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10583494 (10MatthewVernon) Staging now has PCRE2 - https://apt-staging.wikimedia.org/wikimedia-staging/pool/main/p/pcre2/ I've also put in a MR to make noudeb... [16:04:09] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Ensure tls-proxy container is started before launching main container - https://phabricator.wikimedia.org/T387208#10583636 (10Clement_Goubert) p:05Triage→03High [16:53:16] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10583883 (10Jhancock.wm) [16:59:00] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10583913 (10thcipriani) [17:01:48] 06serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Pushing mediawiki-multiversion Docker image from deploy server takes 4 minutes - https://phabricator.wikimedia.org/T341441#10583934 (10thcipriani) [17:09:21] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 07Epic: Pregeneration performance optimizations for PCS - https://phabricator.wikimedia.org/T386919#10583984 (10Ottomata) In [[ https://wikimedia.slack.com/archives/CSV483812/p1740588321416919 | Slack ]], @Jgiannelos asked: > More specifically... [17:10:35] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10583993 (10Jhancock.wm) @Scott_French would you be able to (or know who) could help me move conf2005 to cle... [17:16:07] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10584031 (10Scott_French) @Jhancock.wm - Thanks for flagging! Yes, I can help you with that. I'll open a tas... [17:21:38] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10584067 (10Aklapper) [17:22:29] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Respect kubeVersion constraints in deployment-charts CI - https://phabricator.wikimedia.org/T387376 (10JMeybohm) 03NEW [17:49:40] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10584289 (10akosiaris) To add some more information on the `rp_filter` setting, apparently starting with calico version `3.23.0` (we r... [21:03:16] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10584765 (10Scott_French) [21:04:18] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10584766 (10Scott_French) [21:50:56] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10584869 (10Scott_French) Roughly correlated with the switch to 8.1, we're now seeing a trickle of `ShellboxError`s like the following: `name=normalized_message,lines=10 [{reqId}] {exc... [22:09:50] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10584981 (10Scott_French) [22:19:42] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10585014 (10Scott_French) Last known error was at 22:01:19.025 UTC. I'm going to hold for a bit to see whether the errors have indeed stopped (which may take a little while to state con... [22:56:38] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10585220 (10Scott_French) I'm going to hold where we are now until my morning tomorrow (Thursday, some time after 15:00 UTC), and as long as no new issues are discovered, merge https://... [23:09:20] 06serviceops, 06MediaWiki-Engineering, 06Traffic, 13Patch-For-Review, and 2 others: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10585251 (10Scott_French) As of earlier today (Wednesday) we're back to serving a portion of traffic on PHP 8.1. While we're not fu... [23:27:21] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10585311 (10Scott_French) Thank you very much, @MatthewVernon! Great, so I think the remaining tasks here are: * Consolidating into `component/php81` and reti...