[05:52:23] 06Traffic, 10Math, 06MediaWiki-Platform-Team, 10PageViewInfo, 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11431554 (10Physikerwelt) @Tgr Math appears in codesearch but not in the list. Could you either add it or remove the Math tag? [07:17:35] 06Traffic, 06Infrastructure-Foundations: x-provenance header: identify WMCS - https://phabricator.wikimedia.org/T411503#11431594 (10daniel) > Although the A category within x-trusted-request flag includes WMCS (as well as other wikimedia network traffic), it would indeed be valuable to better differentiate int... [09:30:19] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11431822 (10ayounsi) From that comment : T410989#11429115 cloudcephosd1052 still needs to be migrated. Both interfaces are still doing significant traffic : https://libre... [10:51:13] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11432052 (10fgiunchedi) I took a look at why cloudcephosd1052 still has second nic up, currently: ` 4: ens1f1np1: mtu 9000 qdisc mq stat... [11:34:23] volans: I got a question for you.. https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1213549/8/cookbooks/sre/loadbalancer/admin.py would work or it will fail cause we are using a TZ aware datetime now? [11:36:46] vgutierrez: reboot_since is fine after https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/spicerack/+/9b86a1fbf7a97e1b9c0dc3091a9ea058427895f2%5E%21/ [11:36:50] vgutierrez: I think it has been make future/backward compatible by this patch https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1196139/2/spicerack/remote.py [11:37:00] but you're asking the wrong person nowadays ;) [11:37:24] volans: you will never be the wrong person :P [11:37:30] thx folks <3 [11:37:36] <3 [11:37:48] jokes aside, who is the PoC for spicerack nowadays? [11:38:07] as it was before, I/F :D [11:38:47] I think it's always better to avoid having a single go-to person in mind [11:58:56] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11432253 (10cmooney) >>! In T399180#11432052, @fgiunchedi wrote: > I think the easiest would be to: > > * Remove the spurious `enp13s0f1np1` config, run puppet to verify... [12:28:20] 06Traffic, 10Math, 06MediaWiki-Platform-Team, 10PageViewInfo, 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11432370 (10Tgr) Math needs a config change. [13:12:16] 06Traffic, 10Math, 06MediaWiki-Platform-Team, 10PageViewInfo, 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11432460 (10Clement_Goubert) [13:12:18] 06Traffic, 10Math, 06MediaWiki-Platform-Team, 10PageViewInfo, 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11432462 (10Clement_Goubert) [13:12:32] 06Traffic, 10Math, 06MediaWiki-Platform-Team, 10PageViewInfo, 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11432464 (10Clement_Goubert) [13:13:11] 06Traffic, 10Math, 06MediaWiki-Platform-Team, 10PageViewInfo, 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11432470 (10Clement_Goubert) I've linked the tasks I'd created as children of {T410198}, feel free to dedupe as wanted. [13:55:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack link to asw2-c2-eqiad xe-2/0/13 - https://phabricator.wikimedia.org/T411781 (10cmooney) 03NEW p:05Triage→03Medium [14:01:29] 10netops, 06cloud-services-team, 10Horizon, 06Infrastructure-Foundations, 10Striker: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783 (10taavi) 03NEW [14:03:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack link to asw2-c2-eqiad xe-2/0/13 - https://phabricator.wikimedia.org/T411781#11432684 (10Vgutierrez) the assessment is OK and the link can be removed safely [14:06:08] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11432701 (10Jclark-ctr) [14:09:34] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: lvs1018: decom links to asw2-c2-eqiad and asw2-d7-eqiad - https://phabricator.wikimedia.org/T410661#11432726 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr duplicate to T411781 [14:09:36] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: lvs1018: decom links to asw2-c2-eqiad and asw2-d7-eqiad - https://phabricator.wikimedia.org/T410661#11432732 (10cmooney) 05Resolved→03Declined Duplicate task made in error, will use T411781 [14:10:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack link to asw2-c2-eqiad xe-2/0/13 - https://phabricator.wikimedia.org/T411781#11432739 (10Jclark-ctr) https://netbox.wikimedia.org/dcim/interfaces/29150/trace/ https://netbox.wikimedia.org/dcim/interfaces/29151... [14:29:13] 06Traffic, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work, 13Patch-For-Review: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222#11432854 (10Gehel) [14:30:45] Hello traffic! We might need a bit of help to activate the `druid-public-coordinator` LVS endpoint . It is currently in `service_setup` state, the service is now setup! - T406222 [14:30:45] T406222: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222 [14:39:30] hi gehel , it's a k8s service? [14:41:26] fabfur: no, that's real hardware [14:41:31] ok [14:42:02] running on druid1009-1013 if that helps [14:42:12] are you following the procedure on WT, right? [14:44:53] I think we are. I'm taking over that task and don't have all context :( [14:45:35] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11432921 (10Jclark-ctr) 05Open→03Resolved a:05cmooney→03Jclark-ctr [14:45:58] so I think you should be at this point now : https://wikitech.wikimedia.org/wiki/LVS#Add_the_IPs_on_the_backend_servers [14:46:01] can you verify? [14:48:05] looking [14:49:12] yep, I think we're good [14:50:14] ok so the VIP has been already brought to the realserver lo interface then? [14:53:44] yep, looks like it is there [14:54:00] https://www.irccloud.com/pastebin/3PpFGg4u/ [14:57:11] ack so now you need to do work on the lvs servers so disable puppet on eqiad and codfw [14:57:47] and submit the patch for changing the state to lvs_setup [14:57:52] I can review that too if you want [14:58:19] I'm somewhat scared of touching those servers :/ [14:58:30] eheh, no worry [14:58:47] instructions on WT are good [14:58:57] I need to jump in another meeting, I'll get back to that tomorrow, or more probably next week. Thanks for the help and for checking where we are! [14:59:32] no worry, if you want today I'm here for another couple of hours to follow up on this, or later other people from traffic team [14:59:50] It's not the instructions that I don't trust :) My knowledge of networking is seriously lacking. [15:02:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433019 (10cmooney) [15:02:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433025 (10cmooney) [15:02:43] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11433026 (10cmooney) [15:08:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433060 (10cmooney) [15:16:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11433083 (10cmooney) >>! In T408892#11330727, @cmooney wrote: > Additionally for the rebuild we should aim to: > > # Convert the existing ganeti hosts to rout... [15:17:38] vgutierrez: did you have a scenario in mind where we'd want to depool gerrit-https but not cache_text? in my mind gerrit-https is really just an alias for the CDN [16:12:39] cdanis: not really, it's ok if we use it [16:12:51] just wanted to be explicit/verbose about ti [16:12:52] *it [16:26:43] ack! [16:32:41] hmmm [16:42:23] 06Traffic, 10MediaWiki-Revision-deletion, 06MW-Interfaces-Team: 404 error when using action=raw on an admin-level hidden revision - https://phabricator.wikimedia.org/T351688#11433434 (10BPirkle) p:05Triage→03Medium [17:06:51] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433554 (10BCornwall) [17:14:48] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433580 (10BCornwall) [17:22:43] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433608 (10cmooney) [17:28:50] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433621 (10BCornwall) [17:29:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11433626 (10RobH) [17:30:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lvs1018: remove cross-rack links to rows A, C and D - https://phabricator.wikimedia.org/T411781#11433627 (10RobH) [17:30:47] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433629 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@... [17:32:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11433638 (10RobH) Day 13 Update: * all hosts in rows C and D migrated ** lvs1018 in row B has links into C and D need removal via T411781 before we can kill... [17:40:59] 06Traffic, 07Essential-Work, 05MW-1.46-notes (1.46.0-wmf.7; 2025-12-16), 13Patch-For-Review, 06Test Kitchen (Experiment Platform Sprint 16): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11433660 (10Sfaci) The experiment (ht... [17:41:21] 06Traffic, 07Essential-Work, 05MW-1.46-notes (1.46.0-wmf.7; 2025-12-16), 13Patch-For-Review, 06Test Kitchen (Experiment Platform Sprint 16): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11433663 (10Sfaci) [17:47:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11433701 (10RobH) [18:04:00] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433761 (10Jclark-ctr) [18:06:01] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433764 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumi... [18:16:10] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433805 (10cmooney) [18:16:55] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433810 (10cmooney) [18:17:25] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433811 (10Jclark-ctr) [18:17:36] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11433812 (10cmooney) [18:18:46] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433817 (10cmooney) 05Open→03Resolved [18:21:59] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11433829 (10BCornwall) [20:01:37] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11434460 (10BCornwall) [20:29:06] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11434543 (10BCornwall) [20:50:40] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11434578 (10BCornwall) [21:04:05] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11434605 (10BCornwall) [22:03:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [22:08:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [22:12:32] 06Traffic, 06collaboration-services, 06SRE, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11434875 (10Dzahn) We have to switch these hosts from nftables back to ferm as firewall provider. Reason: liberica does not support nftables yet. [22:59:42] what's going to happen to our host name scheme once we have 2 more POPs and get to 9xxx :p [23:05:59] 06Traffic, 06collaboration-services, 06SRE, 06Release-Engineering-Team (Radar): Deploy a TCP proxy across all DCs - https://phabricator.wikimedia.org/T408532#11435041 (10Dzahn) downtimed, ran puppet, rebooted the 14 VMs and verified ferm service is running via cumin/cookbook. they are all on ferm now. [23:17:07] 06Traffic, 10DNS, 13Patch-For-Review: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11435074 (10Dzahn) The current status is that T408592 is going to use this domain which means this will eventually point to miscweb on...