[00:45:04] 06serviceops, 10SRE-swift-storage, 07Datacenter-Switchover, 13Patch-For-Review: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237#10940520 (10Scott_French) [07:29:25] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Fix alternatives entries in helm and kubernetes-client packages - https://phabricator.wikimedia.org/T387548#10940922 (10Jelto) The refactoring of bash-completion as a slave alternative caused issues during the K... [08:15:52] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#10941009 (10elukey) The plan looks good! [08:37:32] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Fix thumbor discovery records and make swift use them - https://phabricator.wikimedia.org/T397618#10941079 (10JMeybohm) p:05Triage→03High [08:47:33] 06serviceops, 10MW-on-K8s, 10Prod-Kubernetes, 07Kubernetes: Make mw-mcrouter Pods use a higher priorityClass - https://phabricator.wikimedia.org/T397683 (10JMeybohm) 03NEW [08:48:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Build something capable of deploying all services to a cluster - https://phabricator.wikimedia.org/T397684 (10JMeybohm) 03NEW [08:49:51] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Build something capable of deploying all services to a cluster - https://phabricator.wikimedia.org/T397684#10941113 (10JMeybohm) [08:53:43] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team, 10Scap: helmfile/scap does not reliably bootstrap mediawiki - https://phabricator.wikimedia.org/T397685 (10JMeybohm) 03NEW [08:57:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update wikikube codfw to kubernetes 1.31 - https://phabricator.wikimedia.org/T397148#10941153 (10JMeybohm) [09:16:29] 06serviceops, 06Commons, 06MW-Interfaces-Team, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10941204 (10akosiaris) >>! In T380544#10929363, @LucasWerkmeister wrote: > I guess the most likely explanation is that the processing of the updat... [10:14:57] 06serviceops, 10Content-Transform-Team (Work In Progress): Mobileapps cannot be deployed in staging k8s - https://phabricator.wikimedia.org/T397703 (10Jgiannelos) 03NEW [10:17:18] Is there anyone available to help me unblock staging deployments on PCS? ^ [10:20:26] nemo-yiannis: I'll take a look, give me a minute [10:20:36] thanks [10:21:08] 06serviceops, 10Content-Transform-Team (Work In Progress): Mobileapps cannot be deployed in staging k8s - https://phabricator.wikimedia.org/T397703#10941583 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High a:03Clement_Goubert [10:27:18] 06serviceops, 06Infrastructure-Foundations, 07ARM support: Incorporate new arm64 host in our tooling - https://phabricator.wikimedia.org/T397653#10941613 (10akosiaris) [10:58:39] 06serviceops, 10Scap: scap lock seems broken - https://phabricator.wikimedia.org/T397644#10941765 (10Clement_Goubert) Thank you! [11:58:01] 06serviceops, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Mobileapps cannot be deployed in staging k8s - https://phabricator.wikimedia.org/T397703#10941901 (10hnowlan) 05In progress→03Resolved a:05Clement_Goubert→03hnowlan mobileapps can be deployed on staging again [12:05:53] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10942023 (10Aklapper) a:05Clement_Goubert→03None @Clement_Goubert Removing task assignee as this open task has been assigned for more than t... [12:06:51] 06serviceops, 06SRE, 10MediaWiki-Platform-Team (Radar): k8s/mw: traffic to eventgate dropped by iptables - https://phabricator.wikimedia.org/T249700#10942047 (10Aklapper) a:05akosiaris→03None @akosiaris Removing task assignee as this open task has been assigned for more than two years - See the email sen... [12:25:20] 06serviceops, 07Documentation: Create template on Wikitech for documenting production services - https://phabricator.wikimedia.org/T336354#10942311 (10Aklapper) a:05jijiki→03None @jijiki: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22.... [12:25:34] 06serviceops, 06SRE, 10Wikimedia-Apache-configuration: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241#10942309 (10Aklapper) a:05jijiki→03None @jijiki: Removing task assignee as this open task has been assigned for more tha... [12:30:09] 06serviceops, 06SRE, 10Thumbor: Thumbor-k8s performance improvements - https://phabricator.wikimedia.org/T333445#10942470 (10Aklapper) a:05hnowlan→03None @hnowlan: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22. Please assign this tas... [12:30:13] 06serviceops, 10RESTBase: Add monitoring of API endpoints for API gateway - https://phabricator.wikimedia.org/T324231#10942476 (10Aklapper) a:05hnowlan→03None @hnowlan: Removing task assignee as this open task has been assigned for more than two years - See the email sent on 2025-05-22. Please assign this... [12:30:41] 06serviceops, 10Observability-Metrics, 06SRE, 10Maps (Kartotherian): Get Kartotherian SLO metrics into Prometheus - https://phabricator.wikimedia.org/T320748#10942480 (10Aklapper) a:05hnowlan→03None @hnowlan: Removing task assignee as this open task has been assigned for more than two years - See the e... [13:06:43] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10942721 (10Clement_Goubert) a:03Clement_Goubert [13:29:09] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Clean up remaining videoscaling and jobrunner components - https://phabricator.wikimedia.org/T383317#10942846 (10hnowlan) 05Open→03Resolved a:03hnowlan [13:32:18] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10942871 (10Clement_Goubert) 05In progress→03Resolved [13:38:37] 06serviceops, 06Infrastructure-Foundations, 07ARM support, 13Patch-For-Review: Incorporate new arm64 host in our tooling - https://phabricator.wikimedia.org/T397653#10942899 (10akosiaris) Response from the redfish API, using my gerrit change above to print out the response Unsuprisingly, given the error,... [13:47:45] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#10942948 (10MoritzMuehlenhoff) When doing the 8.3.x/8.4.x backports, we should also backport https://github.com/php/php-src/commit/c5e7490963a1c53ec12c09264d0dbe16... [14:27:54] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Fix alternatives entries in helm and kubernetes-client packages - https://phabricator.wikimedia.org/T387548#10943115 (10Jelto) >>! In T387548#10940922, @Jelto wrote: > ..- > I'll upload a patch to prevent this f... [14:55:52] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10943226 (10Jdforrester-WMF) [14:59:52] 06serviceops, 06Abstract Wikipedia team, 07Essential-Work, 07Wikimedia-production-error: Wikifunctions orchestrator service in staging k8s cannot make network calls, gets getaddrinfo EAI_AGAIN / no healthy upstream - https://phabricator.wikimedia.org/T397341#10943254 (10Jdforrester-WMF) Thank you! Conf... [15:28:52] 06serviceops, 10SRE-swift-storage, 07Datacenter-Switchover, 13Patch-For-Review: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237#10943403 (10Scott_French) [15:35:55] 06serviceops, 10SRE-swift-storage, 07Datacenter-Switchover, 13Patch-For-Review: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237#10943437 (10Scott_French) [16:00:32] 06serviceops, 10Page Content Service: mobileapps returns 501 on unsupported language - https://phabricator.wikimedia.org/T397735 (10hnowlan) 03NEW [16:00:38] 06serviceops, 10Page Content Service: mobileapps returns 501 on unsupported language - https://phabricator.wikimedia.org/T397735#10943543 (10hnowlan) p:05Triage→03Low [17:08:47] 06serviceops, 10Page Content Service: mobileapps is comparatively slower to handle changeprop events - https://phabricator.wikimedia.org/T397750 (10hnowlan) 03NEW [17:18:44] 06serviceops, 10ChangeProp, 10Page Content Service, 13Patch-For-Review: Investigate significant changeprop backlogs after PCS migration - https://phabricator.wikimedia.org/T397072#10944026 (10hnowlan) Do we know whether we are generating the correct number of events? It appears that a sustained period of ~... [18:05:38] 06serviceops, 10Page Content Service: mobileapps is comparatively slower to handle changeprop events - https://phabricator.wikimedia.org/T397750#10944231 (10hnowlan) 3GB limit [[ https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?orgId=1&from=2025-06-24T17:34:47.631Z&to=2025-06-24T18:01:07.097Z&timezone=utc&... [19:11:48] 06serviceops, 10Wikidata, 10Discovery-Search (2025.06.13 - 2025.07.04), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10944436 (10EBernhardson) Moving back to reported, as we are tracking resolving the memory issue in T395465 [23:40:38] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade httpd images to bullseye or bookworm - https://phabricator.wikimedia.org/T378128#10945409 (10Scott_French) As of ~ 17:30 UTC today, both mw-api-ext and mw-web are serving ~ 5% of traffic via the migration releases, which are in turn usin...