[05:01:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp3081:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=esams%20prometheus/ops&var-instance=cp3081&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [05:06:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp3081:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=esams%20prometheus/ops&var-instance=cp3081&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [08:17:55] 10netops, 06Traffic, 06Infrastructure-Foundations, 06cloud-services-team (FY2025/2026-Q3-Q4), 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12074704 (10ayounsi) This service is using LVS with IPIP so I don't unders... [08:22:36] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12074721 (10ayounsi) @fgiunchedi maybe silly question but to make sure we're preserving public IPs, why can't we use the same IPs for... [08:49:22] 06Traffic, 06Data-Platform-SRE, 10Liberica, 10Prod-Kubernetes, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12074801 (10klausman) [08:49:48] 06Traffic, 06Data-Platform-SRE, 10Liberica, 10Prod-Kubernetes, and 3 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12074818 (10klausman) Staging is done & tested. I will get work going on prod-codfw today [12:18:31] 06Traffic, 10bot-traffic-requests: Python urllib.robotparser cannot be used on Wikimedia sites - https://phabricator.wikimedia.org/T430738#12075643 (10Joe) I don't see why we should exclude robots.txt from our robot policy. By not allowing to fetch it and indicating the UA policy as soon as someone wants to br... [12:29:51] 06Traffic, 10bot-traffic-requests: Python urllib.robotparser cannot be used on Wikimedia sites - https://phabricator.wikimedia.org/T430738#12075676 (10Krinkle) I think the following is likely: * Set up script with proper UA. * Import robots.txt library, which finds no policy (HTTP 40x) so none is used. Lots of... [12:37:53] 10netops, 06Infrastructure-Foundations: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12075732 (10Aklapper) [12:57:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12075843 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0e1e62e2-283a-46e8-8264-f9dc495aa361) set by... [13:11:06] hello [13:11:48] how to use wikimedia dnses on ios 26 [13:11:59] i need configs [13:16:56] hello [13:19:53] MG2021: hi [13:20:19] we don't have official configs yet but in theory, all you need is the public IP addresses of Wikimedia DNS to generate the configs? [13:20:31] is there something that changed in iOS 26? [13:21:07] could something like https://github.com/paulmillr/encrypted-dns be adapted for Wikimedia DNS in theory? [13:21:58] for example I also see https://barleybobs.github.io/encrypted-dns-configurator/ [13:22:21] and here you can plug in the v4 and v6 address of Wikimedia DNS and that should all be required [13:22:39] sukhe could you please generate a config for me [13:22:53] perhaps we should look into putting these up as well -- I will try later but if you are interested in helping test it out, that would be great since I don't have an iOS device [13:23:14] oh thanks [13:23:29] i'll try and generate [13:24:12] yeah IP addresses are here https://meta.wikimedia.org/wiki/Wikimedia_DNS [13:24:16] IP address (IPv4) 185.71.138.138 [13:24:17] IP address (IPv6) 2001:67c:930::1 [13:24:33] 185.71.138.138 and 2001:67c:930::1. you should put both I think [13:26:18] sukhe Should I type the IP addresses one after another or press enter? [13:26:23] enter [13:26:30] oh thanks [13:27:22] I don't need to add this (v6/IPv4) either, right? [13:28:29] sukhe [13:28:51] no, just 185.71.138.138 and then 2001:67c:930::1 on the next line [13:28:54] and that config should work I think [13:29:36] (y) [13:35:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12075997 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c1065c24-e5fc-46f8-8e8c-c08a491f7877) set by... [13:37:03] sukhe now ho to install it on ios 26.5.2 [13:37:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12076012 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c9fa9a87-4c81-4974-9c0a-6a8c8205b869) set by... [13:38:10] sukhe installed it [13:38:14] how to use it [13:39:07] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12076020 (10fgiunchedi) >>! In T411248#12074721, @ayounsi wrote: > @fgiunchedi maybe silly question but to make sure we're preserving... [13:39:27] MG2021: once you have installed it, visit https://check.wikimedia-dns.org/ to check if you are connected over Wikimedia DNS [13:39:39] sukhe It works great, thanks a lot! [13:40:09] checked using https://browserleaks.com/dns [13:40:47] MG2021: nice, good to know. we should add the profiles to our documentation as well [13:41:10] 10netops, 06Infrastructure-Foundations, 06SRE: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12076032 (10cmooney) [13:42:26] sukhe Please get this done as soon as you can, thanks [14:03:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12076313 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=aab5730e-3995-411e-aa0c-e48fc42c7ad5) set by... [15:00:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12076680 (10Papaul) Information sent to DR [15:10:44] FIRING: [8x] VarnishHighThreadCount: cp3066:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:12:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: rack A8 maintenance 2026-07-01 10:00 am CT - https://phabricator.wikimedia.org/T429856#12076785 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c0d4701e-c375-434f-b42c-5115f14a1c21) set by pt1979@cumin1003 fo... [15:22:28] purged lag [15:22:42] can be just the recovery from the network event or something else [15:22:48] might be worth looking into it [15:22:53] most likely the former [15:23:05] ack [15:23:12] looking [15:25:40] RESOLVED: [8x] VarnishHighThreadCount: cp3066:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:25:49] yeah, I think it's settling [15:28:38] ok awesome [15:28:41] thanks for checking bret [15:28:42] t [16:06:55] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12077048 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e7fbbad5-cec3-439b-bf2e-a0a0bbae2b6b) set by pt1979@cumin100... [16:47:21] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12077348 (10FCeratto-WMF) This triggered the alert `ProbeDown: Service wikikube-ctrl2004...` - the probe indicated a downtime from 16:32:... [16:51:45] 10netops, 06Traffic, 06Data-Persistence, 06Data-Platform-SRE, and 5 others: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12077406 (10FCeratto-WMF) It impacted db2226 https://grafana.wikimedia.org/goto/bfqtb4uge6i9sf?orgId=default and db2203 https://grafana.w... [18:31:49] 06Traffic, 10bot-traffic-requests: Python urllib.robotparser cannot be used on Wikimedia sites - https://phabricator.wikimedia.org/T430738#12077926 (10LucasWerkmeister) >>! In T430738#12073399, @LucasWerkmeister wrote: > Also, there is apparently a **[workaround](https://stackoverflow.com/a/37935740/1420237)**... [18:58:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12078004 (10cmooney) Going to re-schedule this for next Wednesday July 8th at 14:00 UTC. Fairly confident things went ok... [19:28:02] 10netops, 10Infrastructure Security, 06Infrastructure-Foundations, 06SRE: Lumen transport eqiad codfw down July 2026 - https://phabricator.wikimedia.org/T430874 (10cmooney) 03NEW p:05Triage→03Medium [20:15:36] hey traffic, we'll be turning off the no-longer-in-use ipoid service tomorrow Thursday, in the puppet window at 16Z (and pushing into the infra window at 17Z if needed) [20:15:54] I can manage the LVS removal steps if everything goes normally -- if there are any surprises, will anybody from traffic be around? [20:17:04] (that's https://phabricator.wikimedia.org/T416623)