[06:40:39] 06Traffic, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12097952 (10Nikerabbit) 05In progress→03Resolved a:03Nikerabbit This seems resolved now. I've improved our error handling as well. There was a couple 50... [07:38:40] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Add X-Provenance to `wmf.webrequest` - https://phabricator.wikimedia.org/T431256#12098111 (10JAllemandou) [08:15:24] 10netops, 06Traffic, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12098290 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5e02ef3f-9732-4694-88a1-3e0f414cfe47) set by ayounsi@cumin1003 for 2... [08:48:56] 10netops, 06Traffic, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12098486 (10ayounsi) 05Open→03Resolved alright, everything is done! [09:15:52] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12098641 (10brouberol) [09:16:03] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12098648 (10klausman) The only remaining section is IPIP for the workers/ingress in codfw+eqiad. I will coordinate... [09:38:56] 06Traffic, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12098781 (10Tacsipacsi) >>! In T430613#12097952, @Nikerabbit wrote: > There was a couple 503 errors from gerrit last night, but that seems unrelated. I can s... [09:40:43] 06Traffic, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12098784 (10Clement_Goubert) We had a few Gerrit crashes yesterday that probably explain the 503s y'all got cc @hashar @ABran-WMF [09:45:44] 06Traffic, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12098807 (10ABran-WMF) I can confirm that yesterday was a bit chaotic for Gerrit so it might have been the root cause of these 503s. Downtimes are visible on... [10:16:37] \o I am here to bother you once more (hopefully the last time) with IPI-related pybal restarts. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308586 is the to-be-submitted change, The cookbooks (check, dry-run of the actual deploy) run fine on both clusters. So a +1 on that change and an ok for doing two pybal restarts would be delightful. [11:44:24] 06Traffic, 06MediaWiki-Media-Platform-Team: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150#12099456 (10Krinkle) [11:45:35] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Wikimedia-Performance-recommendation: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150#12099458 (10Krinkle) [11:59:45] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12099545 (10BWojtowicz-WMF) **Update** We've noticed when deploying qwen3-14b model on 2x 24GB VRAM partitions of MI300X - https://phab... [12:07:22] 06Traffic, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12099599 (10hashar) @Nikerabbit your task detail has two different errors. 429 rate limiting ============ `counterexample The requested URL returned error:... [12:07:36] 06Traffic, 06collaboration-services, 10Gerrit, 07affects-translatewiki.net: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12099607 (10hashar) [12:45:41] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12099781 (10Ahoelzl) [12:46:03] 06Traffic, 06Data-Platform-SRE, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12099791 (10Ahoelzl) [12:46:42] 06Traffic, 06Data-Platform-SRE, 10Wikidata, 10WMDE Analytics, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#12099801 (10Ahoelzl) [12:51:03] Friendly post-CEST-lunch-ping: \o I am here to bother you once more (hopefully the last time) with IPI-related pybal restarts. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308586 is the to-be-submitted change, The cookbooks (check, dry-run of the actual deploy) run fine on both clusters. So a +1 on that change and an ok for doing two pybal restarts would be delightful. [12:56:18] klausman: LGTM, please mark the CR as active so I can +1 it! [12:56:26] oh, oops, just a sec [12:56:34] done [12:56:53] {{done}} [12:57:03] for me it's a go for the pybal restarts [12:57:18] mille grazie. [12:57:30] prego :D [12:58:07] Willl start sre.loadbalancer.migrate-service-ipip for codfw now [13:09:06] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12099970 (10klausman) codfw workers/ingress is now also IPIP. I have run the httpbb tests: `# pwd /srv/deployment... [13:12:33] cookbook completed. testing via httpbb revealed some grubmlings, but that may be unrelated. [13:12:42] fwiw, pybal should be a-ok [13:13:06] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100009 (10elukey) @klausman `recommendation-api-ng.discovery.wmnet` is a CNAME to k8s-ingress-ml-serve.discovery... [13:13:07] I will hold off on eqiad until I've confirmed the httpbb failures are unrelated [13:29:48] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100059 (10klausman) >>! In T420438#12100009, @elukey wrote: > @klausman `recommendation-api-ng.discovery.wmnet`... [13:41:18] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100104 (10elukey) @klausman for RR wikidata sometimes it fails and sometimes it doesn't, but I noticed this in t... [13:41:36] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100118 (10klausman) Revertrisk-wikidata was intermittent, so that rules out IPIP being at fault, so only this re... [13:50:39] I will continue with eqiad, since the remaining failures are very unlikely IPIP matters [13:59:49] ok! [14:01:15] cookbook done, all looking good. [14:08:20] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100318 (10klausman) Notably, the check works in eqiad (though it does take several seconds): `# httpbb --host i... [14:09:34] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12100320 (10klausman) 05In progress→03Resolved The subject matter of this bug (IPIP migration) is done and... [14:12:16] Hi, I have a question about probes configured for balancer. If the LVS has http probe configured, does it mean that the backend nodes will only be pooled if they pass HTTP check? E.g. https://w.wiki/S9zs [14:14:10] Or does the probe applied to an LVS endpoint, and is not what is used to check if it is okay to pool a host? [14:15:11] atsukoito: probes is just monitoring on the endpoint, lvs.monitors is the check on whether an individual host is healthy and can be pooled [14:15:38] taavi: thanks, I was confused about the terminology here [14:18:41] atsukoito: what taavi said, and then note that the depool threshold is closely related to what nodes will be pooled or not, even if the LVS healthcheck fails [14:18:46] let us know if you have any more questions [14:19:54] I'm trying to figure out why does the node that was returning 503 was pooled, T431538 [14:19:55] T431538: cirrussearch node in diconnected state pooled in lvs - https://phabricator.wikimedia.org/T431538 [14:22:12] atsukoito: looking in a bit [14:30:44] sukhe: thanks, I found the issue: we do check on `/`, but it always returns the banner whenever the service is up. It doesn't return 503 even if the service is not healthy [14:32:07] atsukoito: yep that makes sense! your curl in the task is curl -v http://0:9200/_cat/nodes; echo [14:32:17] the check itself is on / like yo usaid [14:34:56] sukhe: i just didn't expect that opensearch would serve the banner without errors :) [14:35:40] Good catch both of y'all, we should def fix that ;) [14:37:09] The implication of changing the healthcheck is that if the cluster will go red, we won't be responding to the API at all. [15:09:54] 06Traffic, 13Patch-For-Review, 07Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106#12100744 (10BBlack) Additional metrics that would be informative: * Differential of network bandwidth on the host interface port (cp3081 vs peers in same... [15:26:24] 10netops, 06Traffic, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12100808 (10Scott_French) The codfw etcd cluster is serving traffic as usual again. No issues encountered. I'll be refreshing our docs to reflect t... [15:54:42] 10netops, 06Infrastructure-Foundations, 06SRE: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12100957 (10cmooney) Found a few things testing this in container-lab: # We need to set "policy-result accept" in each statement or it will continu... [16:51:49] 10netops, 06Infrastructure-Foundations, 06SRE: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12101183 (10cmooney) Also discussed this on the SR Linux discord and got some good info. My observation in containerlab was that when we set the lo... [17:19:45] 10netops, 06Traffic, 06Data-Persistence, 06Infrastructure-Foundations, 06ServiceOps new: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909#12101299 (10Scott_French) Docs refresh: https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Depool_etcd_client_traffic_from_a_cluster [17:47:58] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: Move TATA IP Transit circuit 1243318 cross-connect from rack B12 to rack B13 - https://phabricator.wikimedia.org/T431605 (10cmooney) 03NEW p:05Triage→03High [17:48:08] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: Move TATA IP Transit circuit 1243318 cross-connect from rack B12 to rack B13 - https://phabricator.wikimedia.org/T431605#12101446 (10cmooney) [17:56:34] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: Move TATA IP Transit circuit 1243318 cross-connect from rack B12 to rack B13 - https://phabricator.wikimedia.org/T431605#12101471 (10cmooney) [18:24:39] 10netops, 06Infrastructure-Foundations, 10ops-drmrs, 06SRE: Move TATA IP Transit circuit 1243318 cross-connect from rack B12 to rack B13 - https://phabricator.wikimedia.org/T431605#12101591 (10cmooney) [18:49:33] 06Traffic: Add ability to validate JWTs in haproxy - https://phabricator.wikimedia.org/T400238#12101708 (10BCornwall) JWT support has been enabled for upload as well - this will help craft request limits that can be more relaxed for authenticated users. [19:13:38] 06Traffic, 06Commons, 06DC-Ops, 10MediaWiki-Core-Revision-backend, and 3 others: ESAMS and others serving older revisions of overwritten files - https://phabricator.wikimedia.org/T425216#12101800 (10ssingh) Thanks for reporting @AlexisJazz. We discussed this in Traffic. @Bblack suggested that since it's di... [20:22:24] 06Traffic, 06SRE: Investigate / Fix upload.wikimedia.org lack of Cache-Control headers - https://phabricator.wikimedia.org/T431621#12102056 (10ssingh) [20:40:58] 06Traffic, 06SRE: Investigate / Fix upload.wikimedia.org lack of Cache-Control headers - https://phabricator.wikimedia.org/T431621#12102110 (10ssingh) Thanks for filing the task! I think we will need to investigate a bit more. For //a// data point to start with: Image from the front page: https://upload.wiki... [21:18:23] 06Traffic: Relax request limits for upload cluster requests with JWT - https://phabricator.wikimedia.org/T431627 (10BCornwall) 03NEW [21:31:47] 06Traffic: Relax request limits for upload cluster requests with JWT - https://phabricator.wikimedia.org/T431627#12102277 (10BCornwall)