[00:36:47] 06Traffic: ncredir should not force HTTPS to a domain we control but don't have a certificate for - https://phabricator.wikimedia.org/T376852 (10jeremyb) 03NEW [00:55:23] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10216097 (10Papaul) We did phase 2 today, all the 1G nodes are now connected to the new fasw2-c8a/b. We will me moving the 10G nodes next we... [08:53:12] 06Traffic: Gather site pooled/depooled information for Grafana - https://phabricator.wikimedia.org/T376876 (10Fabfur) 03NEW [08:59:49] 06Traffic, 10conftool, 07Epic: Deprecate sync, add apply command to requestctl - https://phabricator.wikimedia.org/T376877 (10Joe) 03NEW [09:29:55] 06Traffic: Gather site pooled/depooled information for Grafana - https://phabricator.wikimedia.org/T376876#10216838 (10Vgutierrez) I like the black box approach, but it would require maintaining yet another map of DCs and ranges per service, but I guess you could infer it from `dig +short $cluster-lb.$dc.wikimed... [09:31:06] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879 (10aborrero) 03NEW [10:25:31] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 13Patch-For-Review: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10217030 (10aborrero) 05Open→03Resolved [10:39:58] topranks: given that we do primary/secondary differently in liberica than in pybal it would be feasible to have a primary LB running pybal and a secondary running liberica? [10:40:29] topranks: or the other way around, a primary running liberica with pybal as secondary [10:40:30] vgutierrez: yeah it shouldn't be a problem [10:40:48] ultimately on the network it's done the same way - we match the community and set the local-preference on each device [10:41:06] the difference is with PyBal we add the community - based on MED - in the policy talking to the LVS itself [10:41:18] and in Liberica you will set that at source and we won't use MED [10:41:33] but ultimately quite similar and they are compatible, i.e. the same policy is in effect for both [10:41:39] cause right now the easiest/least risky way of testing liberica would be configuring a primary load balancer in ulsfo to use liberica.. so we can see how it handles upload [10:41:51] (upload cluster) [10:42:25] stop testing, disable liberica, traffic goes back to pybal and we restore the primary LB to use pybal [10:45:56] ok yep it should be fine [10:46:29] we need to set up the policy on the CRs (or switches) for it... do we have a dedicated hostname for liberica hosts or are we planning one? [10:46:54] currently the BGP group a host gets mapped to (if 'bgp' flag in netbox is set to true) is based on the hostname [10:47:14] so if there were a separate hostname it would be cleanest and we could use a separate group/policy [10:47:46] but that said we can also have a "combo" policy which works with the host setting either MED or the community, and have them both in the same group [10:51:04] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10217086 (10aborrero) 05Resolved→03In progress p:05Triage→03Medium I have detected there is no V... [11:03:42] topranks: so right now the idea is to install liberica on an existing lvs host [11:03:52] we don't have hardware allocated for liberica at the moment [11:05:12] ok [11:06:25] we can go with option 'b' for now and leave it in the "pybal" group, but if we're leaving it long term might be best to reimage with a new name [11:39:54] vgutierrez: I've rolled out the updated policy to ulsfo so I think we are good [12:09:04] 10netops, 06Infrastructure-Foundations, 06SRE: Move public-vlan host BGP peerings from CRs to top-of-rack switches in codfw - https://phabricator.wikimedia.org/T360772#10217316 (10cmooney) 05Open→03Declined [12:29:03] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 13Patch-For-Review: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10217375 (10aborrero) still not working. I saw this weird tcpdump capture on cloud... [12:29:33] 10netops, 06Infrastructure-Foundations, 06SRE: Move codfw dns hosts to per-rack vlans and BGP peer with top-of-rack switch - https://phabricator.wikimedia.org/T376894 (10cmooney) 03NEW p:05Triage→03Low [12:39:18] 10netops, 06Infrastructure-Foundations, 06SRE: Consolidate Automation Templates for DC Switches - https://phabricator.wikimedia.org/T312635#10217448 (10Aklapper) [12:56:22] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10217541 (10cmooney) One thing I might be messing you up is the "authentication" section in /etc/keepali... [13:10:04] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10217599 (10aborrero) >>! In T376879#10217541, @cmooney wrote: > One thing I might be messing you up is... [13:10:26] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10217600 (10aborrero) there is also this warning in the logs: Oct 10 13:07:05 cloudgw2002-dev Keepalive... [13:39:43] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10217752 (10cmooney) [13:42:31] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10217761 (10cmooney) [13:46:52] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10217784 (10cmooney) 05Open→03Resolved This is now complete, the cloudsw is set up to route the networks are required and announcing them upst... [13:50:16] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10217807 (10cmooney) >>! In T375847#10195673, @aborrero wrote: > `lang=shell-session > root@ipv6-test-1:~# ip -br a > lo... [13:50:21] topranks: awesome, thjx [13:54:11] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10217836 (10cmooney) The edge (cloudsw/cr) networking is now complete, elements in the range are reachable externally. ` cathal@officepc:~$ mtr -z -b... [15:03:17] 06Traffic, 10Infrastructure Security, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog, 07Security: Integrate In-App Internet censorship circumvention by domain fronting - https://phabricator.wikimedia.org/T327286#10218096 (10Diskdance) @Naruse_shiroha Thank you for the patch. But to make thi... [15:27:00] 06Traffic, 06Movement-Insights: Investigating unique devices traffic data - https://phabricator.wikimedia.org/T375562#10218181 (10Hghani) For September 2024 we see about 1800% increase in Unique Devices from Singapore, YoY which puts total unique devices counted in Singapore at 125,911,935 for a population of... [15:44:08] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10218317 (10Papaul) [15:44:54] 06Traffic, 10Infrastructure Security, 06Privacy Engineering, 06Wikipedia-Android-App-Backlog, and 2 others: Integrate In-App Internet censorship circumvention by domain fronting - https://phabricator.wikimedia.org/T327286#10218321 (10sbassett) [15:46:08] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10218350 (10Papaul) [15:48:03] 06Traffic, 10Infrastructure Security, 06Privacy Engineering, 06Wikipedia-Android-App-Backlog, and 2 others: Integrate In-App Internet censorship circumvention by domain fronting - https://phabricator.wikimedia.org/T327286#10218365 (10Naruse_shiroha) > a simple change from WMF...will stop it from working Y... [15:59:26] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10218489 (10aborrero) >>! In T376879#10217600, @aborrero wrote: > there is also this warning in the logs... [16:17:52] 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-For-Review, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#10218586 (10Pppery) [17:16:54] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, and 2 others: openstack: work out IPv6 and designate integration - https://phabricator.wikimedia.org/T374715#10218766 (10cmooney) Reverse delegation is now working for the ranges we've assigned to OpenStack. I've not gotten an ans... [18:52:23] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10219169 (10Papaul) [20:15:43] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: keepalived: it doesn't support mixing IPv4 and IPv6 VIPs on the same VRRP instance - https://phabricator.wikimedia.org/T376879#10219564 (10Multichill) Ipv6 vrrp is all link-local if I recall correctly. Did you configure it like that?