[04:07:09] FIRING: LVSHighRX: Excessive RX traffic on lvs5005:9100 (ens1f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [04:12:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs5005:9100 (ens1f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [08:03:46] 06Traffic, 06MW-Interfaces-Team, 06ServiceOps new, 07Epic, and 3 others: Epic: Enforce API rate limits (WE5.1.3c) - https://phabricator.wikimedia.org/T412585#11762934 (10daniel) [08:18:06] 10netops, 06Infrastructure-Foundations: mr1-eqiad: move from OSPF to BGP - https://phabricator.wikimedia.org/T421238#11762957 (10ayounsi) Overall that LGTM, you need to add BGP to `security_zones -> production -> services: ['ssh', 'ping', 'traceroute', 'snmp', 'ospf', 'ospf3', 'bgp']` [08:31:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674 (10ayounsi) 03NEW p:05Triage→03Low [09:26:36] 06Traffic, 06ServiceOps new, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of the secure-api.js file - https://phabricator.wikimedia.org/T421464#11763201 (10JMeybohm) I feel li... [09:34:22] 06Traffic, 06ServiceOps new, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of the secure-api.js file - https://phabricator.wikimedia.org/T421464#11763244 (10kostajh) >>! In T42... [10:02:05] 06Traffic, 10ServiceOps-Services-Oids, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 06ServiceOps new (Next quarter), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of th... - https://phabricator.wikimedia.org/T421464#11763352 [10:07:51] 06Traffic, 10ServiceOps-Services-Oids, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 06ServiceOps new (Next quarter), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of th... - https://phabricator.wikimedia.org/T421464#11763400 [10:50:56] 06Traffic, 10ServiceOps-Services-Oids, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 06ServiceOps new (Next quarter), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of th... - https://phabricator.wikimedia.org/T421464#11763567 [12:33:17] 06Traffic, 10ServiceOps-Services-Oids, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 06ServiceOps new (Next quarter), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of th... - https://phabricator.wikimedia.org/T421464#11764104 [13:07:37] 06Traffic: Traffic: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421709 (10ayounsi) 03NEW [13:08:16] 06Traffic: Traffic: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421709#11764399 (10ayounsi) [13:45:15] 06Traffic: Traffic: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421709#11765313 (10ssingh) The `cp` hosts are not a problem of course, but the LVS move is a bit more involved. In theory though, the switch to Liberica and IPIP should make this a lot more... [13:51:03] 06Traffic: Traffic: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421709#11765342 (10ayounsi) No deadline, no rush, best effort :) [14:00:02] 06Traffic: Traffic: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421709#11765390 (10ssingh) OK thank you. I will follow up when we plan to work on it. [14:22:26] hi traffic friends, I'm going to start rolling out https://gerrit.wikimedia.org/r/c/operations/puppet/+/1264640 shortly :) [14:22:47] !!! [14:23:22] +1 [14:24:29] fabfur: shipping your change as well [14:24:46] which might actually be relevant here 🤔 [14:25:04] thanks [14:25:10] I was waiting for your merge :) [14:25:42] perched on puppetserver1001 like a vulture waiting for your puppet-merge to complete :D [14:26:02] ahaha [14:26:04] that should be a noop [14:26:05] btw [14:26:07] ack [14:50:39] ok I'm hands-off :) [14:52:21] 06Traffic, 10Liberica, 10Prod-Kubernetes, 07Kubernetes, 06ServiceOps new (Next quarter): Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#11765761 (10JMeybohm) After enabling IPIP on the workers there are two which do not accept IPIP packets: ` RuntimeError:... [14:55:45] 10netops, 06Infrastructure-Foundations: mr1-eqiad: move from OSPF to BGP - https://phabricator.wikimedia.org/T421238#11765782 (10cmooney) p:05Triage→03Medium [15:17:39] 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission codfw cp hosts cp2027-cp2040 - https://phabricator.wikimedia.org/T419753#11765947 (10Jhancock.wm) 05In progress→03Resolved a:03Jhancock.wm [16:02:02] 06Traffic, 13Patch-For-Review: Upgrade HAProxy to version 3.2 - https://phabricator.wikimedia.org/T421402#11766284 (10Fabfur) [16:09:13] 06Traffic, 13Patch-For-Review: Upgrade HAProxy to version 3.2 - https://phabricator.wikimedia.org/T421402#11766318 (10Fabfur) Had to partially revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/1261484 because configuration gets applied before upgrading HAProxy package (with the cookbook) so it would... [16:10:55] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11766321 (10Ladsgroup) Page previews is still requesting non-standard sizes still. For example, go to https://en.wikipedia.org/wiki/M... [16:11:26] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11766326 (10BCornwall) [17:14:46] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw-eqiad NAT for frmx1002.wikimedia.org - https://phabricator.wikimedia.org/T421750 (10Jgreen) 03NEW [17:40:03] hello traffic friends - I'd like to merge a patch [0] that closes up the remaining config validation coverage gaps in our ATS lua test suites. any concerns or conflicts if I move ahead with that soon? [17:40:03] [0] https://gerrit.wikimedia.org/r/1262152 [17:40:34] no concerns swfrench-wmf [17:40:45] awesome, thanks sukhe :) [17:57:11] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11766852 (10BCornwall) [18:02:10] Hi Traffic, I'd like to coordinate putting a new service behind LVS this week. Patch reviews still in progress, but for awareness here are a few in case you see anything obvious: [18:03:20] [0] - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260767 [18:03:20] [1] - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1248611 [18:03:20] [2] - https://gerrit.wikimedia.org/r/c/operations/dns/+/1248617 [18:03:20] [3] - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260765 [18:04:07] hi jasmine_. brett can help you with that [18:04:10] please sync up with him [18:04:13] thanks! [19:47:17] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11767394 (10BCornwall) [20:52:03] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11767740 (10BCornwall) [21:00:47] 06Traffic, 06SRE: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097#11767763 (10BCornwall) [21:01:05] 06Traffic, 06SRE: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097#11767765 (10BCornwall) 05Open→03In progress p:05Triage→03Medium [21:17:45] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11767824 (10BCornwall) [21:47:52] Hi Traffic, I’m adding a few new stacked k8s control planes (wikikube-ctrl2006 + ctrl200[4-5]) but I would like to confirm, do I need to add a dns record for these explicitly? For instance, [0] or at these records generated from puppet? We don’t seem to have ones for wikikube-ctrl200[4-6] but the CI on these [1] also haven’t failed/complained as we might expect them to [2]? [21:47:52] [0] - https://gerrit.wikimedia.org/g/operations/dns/+/9d3506c3cb6f8b49cb3a1d3921b522b6bd5abeda/templates/0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa#337 [21:47:52] [1] - https://gerrit.wikimedia.org/r/c/operations/dns/+/1249423 [21:47:52] [2] - https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_control-planes [21:50:31] s/at/are* [22:48:54] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11768128 (10CDobbins) [23:02:21] 06Traffic, 06SRE, 13Patch-For-Review: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup - https://phabricator.wikimedia.org/T411097#11768221 (10BCornwall) [23:04:38] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11768278 (10BCornwall) [23:13:47] jasmine_: https://phabricator.wikimedia.org/T376291 seems to be tracking some work for automating it, though it's not happened yet. From my vantage it seems like it'd be nice to have those added, though it's not yet expected/part of a flow quite yet [23:13:59] Though I could be wrong :)