[08:00:02] 06Traffic, 10requestctl: Re-think how external_cloud_vendors works - https://phabricator.wikimedia.org/T438658 (10Joe) 03NEW [08:00:18] 06Traffic, 10requestctl: Re-think how external_cloud_vendors works - https://phabricator.wikimedia.org/T438658#12343350 (10Joe) p:05Triage→03High [08:23:30] 10netops, 06Infrastructure-Foundations: Upgrade gNMIc to 0.49.0 - https://phabricator.wikimedia.org/T438291#12343438 (10ayounsi) 05Open→03Resolved a:03ayounsi All done. [09:17:13] hey folks! [09:17:30] If anybody has time I have https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338937 (and next patches) to add LVS to pki.discovery [09:18:17] Sukhbir already provided some suggestions etc.. lemme know if the patches make sense [10:05:51] elukey: I'm puzzled there... are we checking port 80 but prod traffic runs elsewhere? [10:06:17] that's gonna put port 80 on the LVS VIP [10:07:37] vgutierrez: okok then we can probably wait, I am following what we already have but it is a bit convoluted. Port 80 is there since we require mTLS for pki.d.wmnet on 443 [10:07:48] I didn't want to drag you in but https://phabricator.wikimedia.org/T436809#12328343 is the summary [10:08:01] my idea is to drop the mTLS requirement, that should simplify a lot things [10:08:13] so I'll hold for a moment [10:10:15] my concern is that https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338937 puts port 80 behind the LVS, not ports 443 and/or 8443.. so only the metrics endpoint would be reachable via the VIP [10:10:31] if that's the intended result ok, but that's unclear to me [10:11:13] nono ok that is wrong [10:11:56] if mTLS is enforced for every request on ports 443/8443 the next best thing is a L4 check [10:12:11] so at least pybal will know that the port is open [10:12:17] it is not enforced on 8443, only on 443 [10:12:38] so you could use a https probe for 8443 and tcp for 443 [10:13:14] for 8443 there is a catch - we allow only k8s pod ip ranges to use it [10:15:05] anyway, it seems better to wait for the final shape of the service to settle to avoid problems [10:15:20] I thought to proceed with port 443 anyway but it may complicates more things [10:15:50] sorry :( [10:16:01] nono please thanks for the review :) [10:18:22] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12344369 (10Jelto) [12:54:20] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12345067 (10Jelto) [13:16:02] elukey: I see you already have accessed our Premium customer support but let us know if we can help :) [13:16:48] sukhe: o/ already got scolded on his first day back, I think I hold a record :D [13:17:23] hahah [13:32:55] 06Traffic, 10Citoid, 06MediaWiki-API-Platform-Team, 06ServiceOps, and 2 others: Consider adding a php interface over the citoid api in the Citoid extension - https://phabricator.wikimedia.org/T435916#12345302 (10Mvolz) Thanks, I was waiting to hear the results of the discussion from api-platform before sta... [14:04:04] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#12345439 (10Vgutierrez) `purge_old_cert` was a guard introduced to test... [14:07:34] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12345453 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir3006.esams.wmnet with OS trixie [14:57:24] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12345760 (10dkertesz) I did some local testing to validate the approach of my [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343039 | CR ]] and it seems to b... [15:04:00] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12345795 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir3006.esams.wmnet with OS trixie completed: - ncredir3006 (**WARN**) - Downtimed on Icin... [15:09:26] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12345835 (10ayounsi) [15:09:43] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12345840 (10ayounsi) [15:17:59] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12345894 (10CDobbins) [15:39:44] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12346109 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir6002.drmrs.wmnet with OS trixie [16:36:42] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12346400 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir6002.drmrs.wmnet with OS trixie completed: - ncredir6002 (**WARN**) - Downtimed on Icin... [16:48:06] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12346448 (10CDobbins) [16:52:42] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12346466 (10CDobbins) [16:54:14] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12346469 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir5003.eqsin.wmnet with OS trixie [18:24:41] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12347091 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir5003.eqsin.wmnet with OS trixie completed: - ncredir5003 (**WARN**) - Downtimed on Icin... [18:57:08] 06Traffic, 07Wikimedia-production-error: Error: 503, Backend fetch failed at Wed, 16 Sep 2026 16:50:41 GMT - https://phabricator.wikimedia.org/T438210#12347184 (10Umherirrender) [19:01:21] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12347199 (10CDobbins) [20:11:53] 06Traffic, 10bot-traffic-requests, 06Future-Audiences, 06MediaWiki-Core-Platform-Team (Radar), 07SEO: Link preview from iMessage on macOS blocked - https://phabricator.wikimedia.org/T432373#12347492 (10Krinkle) 05Open→03Resolved Yes. [22:19:48] 06Traffic, 06Data-Platform-SRE, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Provide a scheduled data download service from Google Cloud Storage - https://phabricator.wikimedia.org/T427457#12347874 (10Ahoelzl) 05Open→03Resolved [22:20:30] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Add X-Provenance to `wmf.webrequest` - https://phabricator.wikimedia.org/T431256#12347885 (10Ahoelzl) 05Open→03Resolved [22:21:13] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12347907 (10Ahoelzl) 05Open→03Resolved