[05:07:04] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138360 (10Marostegui) Changing the network cable didn't have any effect. Errors are still there [05:11:03] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138365 (10Marostegui) [05:22:08] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138369 (10Marostegui) RX buffers changed: ``` root@db1114:~# ethtool -g eno1 Ring parameters for eno1: Pre-set maximums: RX: 2047 RX Mini: 0 RX Jumbo: 0 TX: 511 Current... [05:25:01] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138370 (10Marostegui) [06:11:27] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138424 (10Marostegui) For the record, this is the amount of dropped packets per server, of all the servers that are on that switch: ``` ores1008 RX errors 0 dropp... [06:32:03] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138427 (10Marostegui) [07:40:54] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138493 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs3003.esams.wmnet ``` The log can be found in `/var/lo... [07:45:46] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138499 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs3003.esams.wmnet'] ``` Of which those **FAILED**: ``` ['lvs3003.esams.wmnet'] ``` [07:46:04] E_COFFEE :) [07:46:46] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138502 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs3003.esams.wmnet ``` The log can be found in `/var/lo... [07:46:49] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138503 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs3003.esams.wmnet'] ``` Of which those **FAILED**: ``` ['lvs3003.esams.wmnet'] ``` [07:47:23] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138504 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs3003.esams.wmnet ``` The log can be found in `/var/lo... [08:29:40] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138553 (10Marostegui) >>! In T191996#4129814, @ayounsi wrote: > > And switch is now seeing received MAC pause frames. Which confirms that the server is receiving busts of... [08:32:02] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138555 (10Marostegui) [08:35:06] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138557 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs3003.esams.wmnet'] ``` and were **ALL** successful. [09:05:06] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138614 (10Vgutierrez) [09:37:03] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138707 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs2005.codfw.wmnet ``` The log can be found in `/var/lo... [09:55:22] 10Traffic, 10Operations, 10Prod-Kubernetes, 10Pybal: Pybal support of configuration from the kubernetes API - https://phabricator.wikimedia.org/T192437#4138733 (10Joe) [09:55:49] <_joe_> vgutierrez, ema ^^ I'm sure you'll be interested :) [09:57:08] it looks cool [09:57:31] 10Traffic, 10Operations, 10Prod-Kubernetes, 10Pybal: Pybal support of configuration from the kubernetes API - https://phabricator.wikimedia.org/T192437#4138744 (10Joe) Just to clarify, this is a bare support for kubernetes. In theory, it would be nice to gather all information about services we have to con... [10:00:45] 10Traffic, 10Operations, 10Prod-Kubernetes, 10Pybal: Pybal support of configuration from the kubernetes API - https://phabricator.wikimedia.org/T192437#4138747 (10Vgutierrez) This twisted-based kubernetes client could become handy: https://github.com/LeastAuthority/txkube [10:06:12] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138752 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs2005.codfw.wmnet'] ``` and were **ALL** successful. [10:09:37] interesting [10:09:43] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4138757 (10Vgutierrez) [10:09:54] _joe_: how's TLS termination handled in our k8s world? [10:09:56] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138759 (10Vgutierrez) [10:09:59] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4127315 (10Vgutierrez) 05Resolved>03Open [10:13:28] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4138763 (10Vgutierrez) @ayounsi something got messed up when lvs2006 port descriptions were updated: ``` vgutierrez@lvs2006:~$ sudo lldpcli show neighbors |e... [10:13:57] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138764 (10Marostegui) I have captured iostat ouput during two bursts of errors. And there is some reads and cpu spike on both of them, but nothing too worrying or two mass... [10:14:54] <_joe_> ema: at the moment, it's unmanaged, but the answer is a bit complex :) [10:15:15] <_joe_> ema: kubernetes treats internal and external connections differently [10:15:35] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138768 (10Marostegui) [10:16:43] hmm I can feel ema's OCD tingling here.. LLDP ports description mixing "-" and ":" as a separator.. lvs2006:eno1 but lvs2006-eno2 [10:17:09] (mine is itching too) [10:17:30] :) [10:20:03] so... every secondary LVS is now running stretch (except for eqiad, those are not getting reimaged) [10:20:27] ema: how do you feel about reimaging the primaries on eqsin while it's depooled? [10:21:44] vgutierrez: I'd like that, but eqsin is not depooled :) [10:21:55] hmmm [10:22:11] /o\ [10:22:13] right :) [10:23:15] oh I see the problem.. varnish traffic dashboard got its time window set to a fixed time frame on April 10th [10:23:38] lol [10:23:57] let's fix that :) [10:25:04] _joe_: we are going to run TLS terminators inside k8s though, correct? Another approach would be terminating TLS before k8s and use an http load balancer instead [10:25:12] godog once mentioned https://github.com/torchbox/trafficserver-ingress-controller [10:25:13] <_joe_> yes [10:25:25] or rather https://github.com/torchbox/k8s-ts-ingress [10:26:13] <_joe_> so, that's ATS as ingress [10:26:37] <_joe_> we were thinking of using envoy, but it's one possibility [10:26:53] <_joe_> with the advantage of being a piece of software we want to use in other ways too [10:28:44] which one of ATS / envoy? :) [10:31:28] so with eqsin back in the game.. I think that the first DC getting stretch on the primary LVSs should be ulsfo (it's handling less traffic right now) [10:31:46] +1 [10:32:36] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138794 (10Vgutierrez) [10:33:15] of course after actually finishing with the secondaries.. I'm missing lvs2004 :) [10:39:09] bblack: I've restricted the 4xx ttl cap to 404s only after checking whether we are getting any cache hits for non-404 4xx at all (we aren't) [10:40:14] I've used varnishncsa for that as we don't have http status based X-Cache-Status info in prometheus (perhaps we should!) [10:42:01] so now the wm_common_backend_response_{cacheable,uncacheable} refactoring has no functional changes in the _cacheable branch https://gerrit.wikimedia.org/r/#/c/421542/ [10:47:37] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138823 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs2004.codfw.wmnet ``` The log can be found in `/var/lo... [10:54:58] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4138836 (10Marostegui) [10:55:45] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4126641 (10Marostegui) [11:13:36] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138858 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs2004.codfw.wmnet'] ``` and were **ALL** successful. [11:19:12] 10Traffic, 10Operations, 10Puppet: Puppet: tlsproxy localssl default_server make a Notify at each run - https://phabricator.wikimedia.org/T191393#4138867 (10Volans) @Joe no it would not be super easy to solve in a DRY way, I agree. But I've noticed that all the calls to `tlsproxy::localssl` in our puppet re... [11:29:30] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138939 (10Vgutierrez) [11:30:33] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4138940 (10Vgutierrez) [11:30:53] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4127315 (10Vgutierrez) [12:30:51] _joe_, moritzm: https://wikitech.wikimedia.org/wiki/DNS/Discovery#Remove_a_service_from_production [12:38:14] handy! [12:47:53] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 6 others: Proxies information gone from Zero portal. Opera mini pageviews geolocating to wrong country - https://phabricator.wikimedia.org/T187014#4139073 (10Mholloway) [12:49:23] godog: my hope/understanding is that this https://gerrit.wikimedia.org/r/#/c/427373/ should not mess our current dashboards up, and that it would provide valuable info without making the metric count explode [12:49:50] but I'm sure you're gonna shatter my dreams [12:55:29] haha I'll take a look ema [13:07:14] oh.. it's going to break the current caching dasboard, right? [13:09:19] is it? [13:10:56] hmm nope.... current dashboard is doing a sum() so we'll have everything aggregated.. so no difference [13:11:10] but you'll be able to tell between status from now on [13:12:32] like godog said on the CR "We might be already using x_cache metrics from aggregation rules in dashboards so effectively it shouldn't be a problem but should be checked. [13:12:35] " [13:13:15] even without sum() adding a label shouldn't change the results returned by, eg: varnish_x_cache{x_cache="miss"} [13:13:20] (I think!) [13:14:07] https://github.com/wikimedia/puppet/blob/production/modules/role/files/prometheus/rules_ops.conf#L142-L143 [13:14:50] mmh, so those rules should perhaps be changed to also exclude status [13:14:58] as in: exclude(instance, status) ? [13:15:04] s/exclude/without/ [13:19:42] or not, I guess it'd be useful to be able to query the aggregation rule for status info too [13:19:49] https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators [13:21:42] yup.. especially if you want to graph how 4xx behaves for instance [13:31:13] 10Traffic, 10Operations, 10Prod-Kubernetes, 10Pybal: Pybal support of configuration from the kubernetes API - https://phabricator.wikimedia.org/T192437#4139192 (10ema) p:05Triage>03Normal [13:32:56] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4139205 (10Marostegui) For the record, the irq for eno1 is balanced across CPUs, so I don't think it is the bottleneck here: ``` root@db1114:/srv/tmp# for i in `cat /proc/i... [13:33:32] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4139207 (10Marostegui) [13:50:17] 10Traffic, 10Operations: Unconditional return(deliver) in vcl_hit - https://phabricator.wikimedia.org/T192368#4139309 (10ema) [13:52:26] 10Traffic, 10ChangeProp, 10ORES, 10Operations, and 2 others: [Discuss] Split ORES scores in datacenters based on wiki - https://phabricator.wikimedia.org/T164376#4139329 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup I think it's clear that we should not do this thus closing it as resolved. [14:00:53] 10Traffic, 10ChangeProp, 10ORES, 10Operations, and 2 others: [Discuss] Split ORES scores in datacenters based on wiki - https://phabricator.wikimedia.org/T164376#4139359 (10akosiaris) 05Resolved>03declined Declined actually. [14:06:12] 10Traffic, 10ChangeProp, 10ORES, 10Operations, and 2 others: [Discuss] Split ORES scores in datacenters based on wiki - https://phabricator.wikimedia.org/T164376#4139385 (10akosiaris) 05declined>03Resolved Since this was a `[Discuss]` task, `resolved` was conceptually correct. [14:24:49] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4139429 (10Marostegui) During the errors spike I have captured the CPU stats and it is interesting to see that some sys or usr CPU get totally overloaded some seconds befor... [14:41:30] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4139494 (10Marostegui) At the time of the errors (14:30:10), this is what I saw running for a couple of seconds before the errors: 14:30:06 ``` 9476 root 0 -20 29... [14:53:49] bblack: https://odns.cs.princeton.edu/ [14:58:12] am I wrong or the ODNS auth server would still see everything? [15:21:36] bblack: so.. what do you think about reimaging lvs400[56] to stretch tomorrow EU morning (smallest amount of traffic @ ulsfo during those hours)? [15:22:38] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4139624 (10Cmjohnson) All the cross connects have been made and connected to the switch ports. I updated the descriptions in both switches on each row. The server still need... [15:23:08] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4139626 (10Cmjohnson) [15:23:38] vgutierrez: +1 [15:25:21] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4139627 (10Vgutierrez) >>! In T184293#4137521, @ayounsi wrote: > @Cmjohnson: would that works for you? > > |lvs1016|eth0/eno1|asw2-d:xe-7/0/15|cable #4061| > |lvs1016|eth1/e... [15:25:33] oh.. lvs1016 got cabled... we are almost there <3 [15:26:21] XioNoX: yes, their scheme seems pretty dubious. the claim that the ODNS roots don't see privacy-sensitive data seems to rest on the claim that the recursive cache in the way effectively anonymizes users behind it, which is (a) somewhat-false and (b) identical argument for traditional DNS and (c) requires avoiding edns-client-subnet and de-optimizing the internet :P [15:26:43] XioNoX: it probably won't gain any significant traction [15:27:28] thx for the tldr :) [15:28:00] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4139635 (10Cmjohnson) @Vgutierrez I set the descriptions on the switches with both jic. xe-4/0/5 description "lvs1016 eth3/ens1f1 #3918" [15:29:04] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4139637 (10Vgutierrez) >>! In T184293#4139635, @Cmjohnson wrote: > @Vgutierrez I set the descriptions on the switches with both jic. > > xe-4/0/5 description "lvs1016 eth3... [15:31:07] vgutierrez: yeah if hardware is out of the way, we should get lvs1016 up and going. There's some ??? about how we initially configure it vs the long haul, in puppet. [15:32:04] vgutierrez: probably our best bet at present is to add it to the configuration places as a 3rd node for the low-traffic class for now. (as in, make the puppetization look like an extra lvs1006). [15:32:37] vgutierrez: and once it's up and running sanely, we'll switch it to be the primary with lvs1003/6 as backups, for that class (and deal with the multi-failover like lvs400[567] do later, once we have more new lvses up) [15:32:53] ack [15:33:09] I'll do that as soon as dcops handle us the server :) [15:34:56] eventually where we'll end up at is: lvs1013:primary-high-traffic1 lvs1014:primary-high-traffic2 lvs1015:primary-low-traffic lvs1016:fallback-for-all, which is closer to how the lvs[45]xxx configs look, just with an extra class in the core DCs' case. [15:36:34] I hate that "low-traffic" label every time I have to type it. It used to be the lower-traffic of the 3, because high-traffic[12] handle our high-bandwidth public traffic into the caches, and low-traffic handles the cache misses into the internal applayer. [15:37:14] but now that we have $services/$microservices all over talking to each other low-traffic's use for internal service<->service traffic makes it the largest traffic volume of the 3 :P [15:38:32] so we should switch names.. from high/low traffic.. to public/internal faced.. [15:39:52] BTW, I've tried to clean-up a little bit the ntp module after the stretch update: https://gerrit.wikimedia.org/r/#/c/427101/ [15:40:19] yeah, but switching names is probably non-trivial, there's all kinds of config fallout that could happen during the switch that might be disruptive, so it needs a lot of careful validation. [15:40:26] (and then update docs on wikitech too I guess) [15:41:24] vgutierrez: on the ntp cleanup: [15:42:30] 1) modules/profile/manifests/ntp.pp - don't get rid of the stretches' empty $wmf_server_upstreams pass-through, as we may end up using it in a later refactor, and it makes it confusing if we have to go digging for how to set it (or someone might see it missing and try to remove it in ntp.conf later) [15:43:13] 2) we can't clean up modules/ntp/templates/ntp-conf.erb at all yet, because we still have trusty hosts using that template for old ntpd versions as client-only. [15:43:30] (I assume we still have trusty hosts anyways, it'd be nice if they all finally went away!) [15:44:10] oh.. I wasn't aware of that [15:44:17] thx, I'll fix those points [15:45:30] (later refactor mentioned in (1) above: we might re-structure things where non-core DCs use core DCs ntp servers as actual servers rather than peers) [15:46:05] I think it might be the right thing to do at this phase, but I haven't had time to dwell on it deeply [15:48:15] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4139734 (10ayounsi) That's some great investigation! >>! In T191996#4138553, @Marostegui wrote: > @ayounsi does that mean that the switch is the one not being able to cope... [15:50:56] 10netops, 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1114 connection issues - https://phabricator.wikimedia.org/T191996#4139750 (10Marostegui) >>! In T191996#4139734, @ayounsi wrote: > That's some great investigation! > >>>! In T191996#4138553, @Marostegui wrote: >> @ayounsi does that mean... [15:54:12] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 6 others: Proxies information gone from Zero portal. Opera mini pageviews geolocating to wrong country - https://phabricator.wikimedia.org/T187014#4139751 (10dr0ptp4kt) The lack of tagging did not appear to be related to any config b... [15:59:16] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 6 others: Proxies information gone from Zero portal. Opera mini pageviews geolocating to wrong country - https://phabricator.wikimedia.org/T187014#4139773 (10Nuria) @dr0ptp4kt trying to understand: is this the bug that makes the requ... [16:10:31] 10Traffic, 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, and 6 others: Proxies information gone from Zero portal. Opera mini pageviews geolocating to wrong country - https://phabricator.wikimedia.org/T187014#4139798 (10dr0ptp4kt) >>! In T187014#4139773, @Nuria wrote: > @dr0ptp4kt trying to und... [17:22:51] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4140111 (10ayounsi) [17:23:30] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4140116 (10ayounsi) [17:23:33] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4127315 (10ayounsi) 05Open>03Resolved Indeed, thanks for double checking. All 3 hosts done and verified. [18:14:50] vgutierrez: about lvs1016, will "lvs1016 eth0/eno1 asw2-d:xe-7/0/15 cable #4061" be used as main interface for that server? [18:22:15] bblack: ^ [18:23:33] XioNoX: yes [18:23:39] thx! [18:24:01] (in general, for the core DCs' LVSes the first/primary interface should be the same-row interface) [18:26:52] ok [18:27:52] I wanted to doublecheck with the interfaces renaming going on [18:45:52] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4140393 (10ayounsi) Ports vlan configuration done.