[00:55:05] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: new CNAME record for WikiLearn - https://phabricator.wikimedia.org/T429628#12043293 (10BCornwall) The DNS changes have been applied. [01:00:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: codfw: rack A8 maintenance 2026-07-01 10:00 am CT - https://phabricator.wikimedia.org/T429856 (10Papaul) 03NEW [02:05:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: rack A8 maintenance 2026-07-01 10:00 am CT - https://phabricator.wikimedia.org/T429856#12043444 (10Papaul) p:05Triage→03Medium [02:24:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12043459 (10Papaul) a:03Papaul [02:25:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12043469 (10Papaul) [05:26:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: rack B2 maintenance 2026-07-01 11:00 am CT - https://phabricator.wikimedia.org/T429861#12043625 (10ayounsi) [07:48:19] Hello team, regarding https://gerrit.wikimedia.org/r/c/operations/puppet/+/1283745/3 rollback which I mentioned yesterday I can of course roll this back myself unless you deem I shouldn't so let me know : ) [07:50:34] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12043882 (10cmooney) >>! In T429773#12043077, @BTullis wrote: > So how about... [07:54:04] hi dpogorzelski, I think it's safe to roll back, let us know if something doesn't work as expected! [08:36:19] 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12044038 (10cmooney) [08:41:29] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12044054 (10ayounsi) Yep! Sounds great! [08:49:40] 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12044120 (10cmooney) [08:54:23] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12044146 (10brouberol) This will only be used by NFS clients, right? Will it impact humans going to https://dumps.wikimedia.org/ in a... [08:56:57] 10netops, 06Data-Persistence, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: codfw: rack A6 maintenance - https://phabricator.wikimedia.org/T429812#12044155 (10cmooney) [09:04:18] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: new CNAME record for WikiLearn - https://phabricator.wikimedia.org/T429628#12044210 (10Mjalaluddin) Thankyou @BCornwall [11:46:05] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12044919 (10fgiunchedi) >>! In T411248#12044146, @brouberol wrote: > This will only be used by NFS clients, right? Will it impact hum... [12:03:56] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12045019 (10brouberol) No objection from my part! [12:15:40] 06Traffic, 10Liberica, 10Prod-Kubernetes, 07Kubernetes, and 2 others: Migrate ML k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420438#12045079 (10ops-monitoring-bot) Roll-reboot of nodes in ml-staging-codfw cluster started by klausman: * ml-staging-ctrl[2001-2002].codfw.wmnet [12:26:02] \o I am having trouble with switching the ML staging cluster to IPIP. The master change (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1294223) has been merged and Puppet ran, master nodes have been rebooted, but sre.loadbalancer.check-ipip still fails, and I aj seeing ulogd entries for the relevant traffic: [12:26:10] 2026-06-23T12:23:28.628297+00:00 ml-staging-ctrl2001 ulogd[551]: [fw-in-drop] IN=ens13 OUT= MAC=aa:00:00:bb:2b:bc:02:00:5e:44:44:44:08:00 SRC=172.16.1.1 DST=10.192.16.93 LEN=64 TOS=00 PREC=0x00 TTL=62 ID=1 PROTO=4 MARK=0x0 [12:26:24] Any suggestions about what might be amiss? [12:28:29] fwiw, iptables-save shows no rules containing the 172.16.0.0/20 net [12:30:30] 06Traffic, 10Lift-Wing, 06ServiceOps new, 10ServiceOps-SharedInfra, and 2 others: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12045125 (10isarantopoulos) Reporting back here after some slack conversations. The reported errors in the related phabricator pastes https... [12:48:21] Fixed it, I was missing https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305110 [12:57:01] fabfur: roger [13:05:26] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07Kubernetes: Calico IPv4/IPv6 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12045319 (10cmooney) I have assigned [[ https://netbox.wikimedia.org/ipam/pre... [13:13:43] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12045381 (10Gehel) Moving this to "watching" as I don't expect #data-platform-sre to be involved in the implementation. Ping us on Sl... [13:42:24] 06Traffic: Test HAProxy 3.2 with AWS-LC libraries - https://phabricator.wikimedia.org/T419825#12045562 (10Fabfur) [14:50:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12046011 (10Papaul) [14:57:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12046036 (10Papaul) [17:07:56] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047255 (10Ahoelzl) [17:08:50] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047298 (10Ahoelzl) At this point lets increase the alerting thresholds to reduce the noise on the data engineering alert mailing list, indiv... [17:09:04] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047299 (10Ahoelzl) a:05Ahoelzl→03JAllemandou [18:13:22] 06Traffic, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 13Patch-For-Review: Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#12047624 (10Ahoelzl) a:05JAllemandou→03Snwachukwu [18:33:18] 06Traffic, 06Data-Engineering: Tune refine webrequest data loss threshold to avoid noisy irrelevant alerts. - https://phabricator.wikimedia.org/T429809#12047684 (10Ahoelzl) a:03Snwachukwu [22:06:39] 06Traffic, 07Sustainability (Incident Followup): Ensure the pre-repooling checklist includes to restart liberica services whenever realserver IPs has changed - https://phabricator.wikimedia.org/T426299#12048435 (10BCornwall) 05Open→03In progress @hnowlan: Which checklist are we talking about? In https://wi...