[07:07:03] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12394216 (10JAllemandou) I have reduced the replication factor for the netflow dataset to 1 instead of 2. This should help while we (hopefully) fix the problem with... [07:33:30] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12394267 (10ayounsi) @wiki_willy can you ask for credits on that one too ? HE ticket 7282921 for circuit dc4841.dal5.sin1 showing issues between 2026-10-02 06:20 and 2026-10-03 04:4... [07:37:47] just FYI i'm about to update dns entries for liftwing studio, aka hummingbird [08:56:04] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12394549 (10Jelto) [09:09:10] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12394575 (10Jelto) 05Open→03Resolved All wikikube services have been migrated. The last services were `mw-web*` and `mw-api-ext*`. I... [09:20:13] 06Traffic, 13Patch-For-Review: Ephemeral port range clashes with bound daemon ports on cache proxy hosts - https://phabricator.wikimedia.org/T433005#12394596 (10dkertesz) a:03dkertesz I have merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1345963 which move the lower bound of ephemeral ports from... [09:32:25] 06Traffic: Observability for ephemeral ports on cp servers - https://phabricator.wikimedia.org/T440164 (10dkertesz) 03NEW [09:32:37] 06Traffic: Observability for ephemeral ports on cp servers - https://phabricator.wikimedia.org/T440164#12394660 (10dkertesz) [09:32:39] 06Traffic: Ephemeral port range clashes with bound daemon ports on cache proxy hosts - https://phabricator.wikimedia.org/T433005#12394661 (10dkertesz) [10:32:03] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12394958 (10dkertesz) p:05Triage→03Low [11:04:27] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12395074 (10BTullis) >>! In T434046#12390832, @ssingh wrote: > @BTullis: The simplest thing to consider here would be setting `datasets.wikimedia.org` caching to `normal` an... [11:40:33] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12395146 (10dkertesz) [13:15:00] hello hello [13:15:19] pki backends should support ipip now, I tested it via sre.loadbalancer.check-ipip [13:15:24] and filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1351310 [13:15:29] ok for me to retry? [13:24:09] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12395576 (10dkertesz) [13:31:16] elukey: +1 [13:31:22] <3 [13:34:06] {◕ ◡ ◕} [13:44:10] 06Traffic: Redact logs for 5XX response + Set-Cookie in Varnish - https://phabricator.wikimedia.org/T440187 (10dkertesz) 03NEW [14:09:29] 06Traffic, 07Sustainability (Incident Followup): sre.dns.wipe-cache cookbook should look up DNS hosts via etcd, not puppet - https://phabricator.wikimedia.org/T440190 (10hnowlan) 03NEW [14:20:49] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12395920 (10ssingh) Looks good, thanks @BTullis for rolling the change. ` $ curl -Is https://analytics.wikimedia.org/published/datasets/ml/tts/experiment-v1/enwiki/662/1371... [14:23:09] 06Traffic, 07Sustainability (Incident Followup): sre.dns.wipe-cache cookbook should look up DNS hosts via etcd, not puppet - https://phabricator.wikimedia.org/T440190#12395933 (10ssingh) a:03ssingh [14:32:00] 10netops, 06Infrastructure-Foundations: Junos: ae speed not reported correctly through gNMI - https://phabricator.wikimedia.org/T439202#12396008 (10Aklapper) [14:37:05] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12396029 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:37:56] 06Traffic, 06Infrastructure-Foundations, 06SRE, 07LDAP, 13Patch-For-Review: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12396033 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:37:58] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12396035 (10LSobanski) p:05Triage→03Medium [14:39:20] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, 07Kubernetes: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12396041 (10ssingh) Thank you @Jelto for the excellent work on this! [14:43:31] 06Traffic: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667#12396060 (10dkertesz) [14:49:53] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12396091 (10BTullis) @ssingh you mentioned this: >The simplest thing to consider here would be setting datasets.wikimedia.org caching to normal and **then disabling caching... [15:11:07] 06Traffic, 06Infrastructure-Foundations, 06SRE: Support LVS backend servers using nftables - https://phabricator.wikimedia.org/T433601#12396223 (10Urbanecm_WMF) This appears to cause {T440201} [15:18:01] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12396261 (10ssingh) >>! In T434046#12396091, @BTullis wrote: > @ssingh you mentioned this: >>The simplest thing to consider here would be setting datasets.wikimedia.org cach... [15:36:03] 06Traffic, 06Collaboration-Services, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12396389 (10LSobanski) 05Open→03Resolved a:03LSobanski I believe we can consider this one resolved, with {T440068} being the follow up task. [16:12:33] 06Traffic: Observability for ephemeral ports on cp servers - https://phabricator.wikimedia.org/T440164#12396563 (10BCornwall) IMO this is beyond our scope - this is non-trivial work for an issue that's unlikely to rear its head again (we only rarely ever had port clashes after years of inappropriate configuratio... [16:23:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:28:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:09:05] FIRING: [2x] DnsboxServiceMismatch: Service ntp-c state mismatch on dns1006:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:13:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-c state mismatch on dns1006:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:19:14] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12396922 (10CDanis) I put together a cool interactive data visualization for the overload event, along with some explainers. If you're interes... [17:24:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns2004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:27:05] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns2004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:10:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns3004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:15:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns3003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:24:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns4003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=ulsfo&var-instance=dns4003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:29:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns3004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:34:07] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12397292 (10BTullis) >>! In T434046#12396261, @ssingh wrote: >> By the backends, do you mean the target webservers `clouddumps100[1-2]`? I made a mistake there. The backend... [18:40:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns4003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:45:50] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns4004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=ulsfo&var-instance=dns4004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:55:05] 06Traffic, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12397418 (10Jgreen) The load balancers and servers have been configured to accept the new domain, ready to go once the new cert and... [18:56:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns5003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:01:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns4004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:11:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns5004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:16:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns5003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:18:03] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12397527 (10wiki_willy) Thanks for the heads up @ayounsi, I went ahead and submitted an email to them requesting for the credits. [19:26:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=drmrs&var-instance=dns6001:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:31:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns5004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:33:49] 06Traffic, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12397620 (10XenoRyet) 05Open→03Resolved [19:40:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [19:42:10] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12397695 (10isarantopoulos) > So are 1 hour TTLs sufficiently long for this requirement @isarantopoulos? > If you're OK with the current settings, then we might be done here... [19:45:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [20:12:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns7001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [20:17:50] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns7002:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=magru&var-instance=dns7002:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [20:45:14] 06Traffic, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 4 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12397979 (10ssingh) Please let us know when you want us to retire donate.wikimedia.org. There are a few Puppet-related cleanups...