[08:52:56] godog: can we give some love to https://phabricator.wikimedia.org/T193488#4172235 ? :) [09:02:42] vgutierrez: for sure, a couple of things need to happen, I won't have time today though if you'd like to try I can assist [09:02:58] sure :) [09:03:13] what needs to be done? [09:06:40] heh build and upload teh updated version, I'll expand on that shortly [09:09:02] oh, nothing special then [09:15:55] lvs2003 behaviour is funny.. pretty small number of connections, but it handles way more traffic per second than lvs2001 or lvs2002 [09:16:58] I guess it's handling some ingest service [09:23:28] mmh, on https://grafana-admin.wikimedia.org/dashboard/db/load-balancers we're plotting the number of "active" incoming connections from the standpoint of ipvs [09:24:15] there's also the "inactive" connections count which we should plot too [09:24:35] actually we plot the rate of active connections [09:24:47] so we're plotting new connections per second [09:24:52] most conns on lvs2003 seem to be considered "inactive", see ipvsadm -L [09:25:14] (well now on lvs2006 of course) [09:25:29] yup [09:25:56] also how ipvs tracks active/inactive connections still bugs me [09:28:10] till a few seconds ago lvs2003 was still showing active connections [09:28:23] but bps and pps was 0 for a few minutes already [09:29:06] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4173920 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs2003.codfw.wmnet ``` The log can be found in `/var/lo... [09:38:22] 10Traffic, 10Operations, 10Performance-Team: Update Media dashboard in Grafana to use Prometheus metrics - https://phabricator.wikimedia.org/T193445#4173951 (10ema) p:05Triage>03Normal [09:53:45] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4173979 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs2003.codfw.wmnet'] ``` and were **ALL** successful. [10:18:36] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174051 (10Vgutierrez) [10:19:08] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4138499 (10Vgutierrez) [10:19:11] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4174052 (10Vgutierrez) 05Resolved>03Open [10:37:21] vgutierrez: I've added the steps I used for python-logstash in https://phabricator.wikimedia.org/T193488#4174105 it is done though let me know if something doesn't make sense [10:42:02] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4174159 (10Vgutierrez) >>! In T184293#4162375, @Cmjohnson wrote: > @ayounsi Can you create a subnet for LVS for row D please. According to https://wikitech.wikimedia.org/wi... [10:42:33] godog: thx :D [10:45:58] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174181 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs2002.codfw.wmnet ``` The log can be found in `/var/lo... [11:29:22] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174284 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs2002.codfw.wmnet'] ``` and were **ALL** successful. [11:44:52] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174322 (10Vgutierrez) [11:46:21] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4174328 (10Vgutierrez) [12:50:33] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174470 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on neodymium.eqiad.wmnet for hosts: ``` lvs2001.codfw.wmnet ``` The log can be found in `/var/lo... [12:52:01] where is class varnish in puppet used? I don't see any prod host having it according to puppetdb [12:56:22] nevermind, pebcak [13:03:52] class varnish is a mess :) [13:04:18] if there's going to be any corner case some script or grep or policy or standard is going to miss, it will be the case for class varnish [13:18:42] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174538 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['lvs2001.codfw.wmnet'] ``` and were **ALL** successful. [13:40:40] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Remove wildcard vhost for *.wikimedia.org - https://phabricator.wikimedia.org/T192206#4174594 (10BBlack) FWIW on my end, the following hostnames are definitely non-functional: ``` text-lb text-lb.codfw text-lb.eqiad text-lb.eqsi... [13:46:39] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4174610 (10Vgutierrez) [14:03:51] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Remove wildcard vhost for *.wikimedia.org - https://phabricator.wikimedia.org/T192206#4174688 (10Jgreen) >>! In T192206#4174594, @BBlack wrote: > FWIW on my end, the following hostnames are definitely non-functional: > ``` > tex... [14:10:29] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Upgrade LVS servers to stretch - https://phabricator.wikimedia.org/T177961#4174710 (10Vgutierrez) [14:10:33] 10Traffic, 10Operations, 10Pybal: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4174706 (10Vgutierrez) 05Open>03Resolved a:03Vgutierrez [14:10:40] \o/ [14:14:48] bblack, XioNoX re https://phabricator.wikimedia.org/T184293#4174159 I don't know if I misunderstood cmjohnson1 comment and thus I replied wrongly [14:16:12] vgutierrez: I'm not sure I fully understand it either. We don't have LVS-specific subnets. He may have meant to create the per-row interface groupings for the vlan stuff rather than actually creating a new vlan. [14:16:24] XioNoX: ^ can probably take a peek and/or ask and figure out what's needed. [14:16:28] vgutierrez: I followed up with him on IRC, he meant an interface group on the juniper switch for that specific port/vlan [14:16:38] oh ok [14:16:41] ack [14:16:45] Tldr it's done [14:17:08] so... can we assign IPs to lvs1016? [14:17:49] yeah, following the model of the others [14:18:10] there's some oddities to copy about how we name them in DNS and how we provision them and IPv6 and blah blah blah [14:18:22] I think so, ports are enabled and have the proper vlan config [14:18:48] Check with Chris for the actual host status [14:18:58] there are some improvements we should be making in how we handle that (e.g. we don't put IPv6 for these in DNS at all, even though they critically-rely on the add_ip6_mapped stuff) [14:19:11] but for now I'd copy existing practice just to get the new host up with minimal fuss [14:21:32] (like lvs1003's dns entries, but with row D as the home row for the first interface, followed by A, B, C on the others with the per-interface+vlan DNS naming) [14:22:46] hmmm like 1003 or like 1012? [14:23:02] take into account that 1003 it's lvs1003.wikimedia.org and 1012 lvs1012.eqiad.wmnet [14:23:22] yeah on that kind of thing, lvs1012 is a better example maybe, lemme look [14:24:13] yeah lvs1012 is a better example, although it's in row C [14:25:08] so basically the first/primary interface/IP will be private1-c-eqiad [14:25:41] oops [14:25:44] right, and for lvs1016 we need to be in 1-d-eqiad [14:25:44] so basically the first/primary interface/IP will be private1-d-eqiad [14:25:47] :) [14:25:51] 10.64.48.0/22 [14:26:01] everything else gets the vlNNNN-iface.lvs1016.eqiad.wmnet naming, with the next 3 ports being in rows a b c for that naming. [14:26:17] not that public1-d-eqiad also uses the lvNNN-iface thing, on the same port name as the primary interface [14:26:20] *note [14:27:28] only the mgmt entry is created in DNS so far [14:29:03] you should end up adding 8x total new lines to templates/wmnet and matching that on the reverse side: 4x new total lines in templates/10.in-addr.arpa, and 4x total lines in templates/15[45].80.208.in-addr.arpa (155 file has public1-d) [14:36:14] re DNS, let me know if this makes sense bblack https://gerrit.wikimedia.org/r/#/c/428888/ [14:40:05] so I was playing a bit with the numa stats for cache hosts: https://grafana-admin.wikimedia.org/dashboard/db/caches-numa-stats?orgId=1 [14:40:35] looking at upload@eqiad, the two hosts with numa_networking on are 1074 and 1048 [14:41:08] if you look at the numa hitrate on node 0 for the past 6 hours, it's been perfect on both (1.0) [14:42:25] if you look at node 1, they've been the worse (0.0something) [14:44:35] other random observations: text is much better than upload in terms of numa hitrate, busy DCs are worse than quiet ones [14:50:54] it's hard to say what that means of course, it's too low-level a detail for us to understand in this scenario. [14:51:41] better would be some visible effects elsewhere: e.g. differential in long-term loadavg, cpu%s, latency, memory-pattern-strangeness (e.g. the earlier observation that it cleared away some of the spikiness there) [14:52:27] but if there's no notable differential on those fronts, we still know it's better in general not to be sending the actual network card IRQs cross-numa-domain to reach the initial receiving daemon. [14:53:39] (better in terms of cpu cache locality and minor latency bumps, general efficiency, etc) [14:55:37] probably as good as we can get until we come up with some workable way to actually pin hwirq->nginx_proccess to specific cores without jumping around. [14:55:59] (the whole SO_ATTACH_REUSEPORT_EBPF conversation) [15:00:02] the biggest source-level problem for implementors with that, is handling the socket ordering. quoting the manpage: [15:00:07] Sockets are numbered in the order in which they are added to the group (that is, the order of bind(2) calls for UDP sockets or the order of listen(2) calls for TCP sockets). New sockets added to a reuseport group will inherit the BPF program. When a socket is removed from a reuseport group (via close(2)), the last socket in the group will be moved into the closed socket's position. [15:00:47] so if you had a daemon that did its smooth restarts by opening a new set of REUSEPORT listening sockets in parallel with the old ones, and then closed the old ones as they drain and procs/threads die.... [15:01:16] you end up with a kind of randomized total ordering in the new set for thise purpose (whereas on clean startup, you can create them in cpu-core-number ordering initially) [15:01:42] you'd have to get the shutting-down copy of the daemon to close its sockets in exact reverse order of creation to preserving ordering of the new ones. [15:02:09] (or do some kind of horrible tracking and remapping that can be used be the EBPF program) [15:03:14] or alternatively, on smooth restarts do SCM_RIGHTS-based handoff instead of opening a new set of reuseport sockets. you might still need to close or add some, but you have better control over the ordering that way to make it work out right. [15:04:27] (I think nginx now does that? but I'm not 100% sure) [15:07:55] it would be slightly-nicer if the behavior had instead been to shift the numbers to close the gap. e.g. if you have a set 1,2,3,4, when you close(2), 3 becomes 2 and 4 becomes 3. The way it is now, 3 stays 3 and 4 becomes 2 :/ [15:51:14] bblack, XioNoX hopefully this makes sense https://gerrit.wikimedia.org/r/#/c/430402/ it should solve lvs[1013-1016] DNS entries [15:53:55] vgutierrez: ack, will review in a little bit (these are rough to validate for tiny mistakes!) [15:55:32] thx :) [16:02:56] there [16:03:11] proxyfetch and idleconnection now have 100% unit test coverage [16:03:17] proxyfetch had 0 [16:03:21] i guess it's well tested in production ;) [16:07:17] vgutierrez: I haven't validated it all yet, but will against the existing naming, but should we set this up with proper stretch iface names in DNS now? [16:07:58] we never fixed the dns naming for stretch ifaces on the other ones. I don't *think* it breaks anything or requires any coordination, but should double-check that assumption... [16:10:47] [a quick check through the puppet repo seems to indicate nothing keys off of the DNS names for the vlNNNN.ethX naming for anything functional) [16:10:59] arg! s/)/]/ :) [16:16:15] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: Rack/cable/configure ulsfo MX204 - https://phabricator.wikimedia.org/T189552#4175273 (10ayounsi) [16:19:33] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Remove wildcard vhost for *.wikimedia.org - https://phabricator.wikimedia.org/T192206#4175277 (10BBlack) @Jgreen yeah if you don't have any special purpose for them, then they're basically the same as the text-lb ones (we like h... [16:29:37] vgutierrez: +1 on patch, I think the mappings all look correct. re: stretch iface names in DNS: perhaps leave your patch as-is and get lvs1016 initially puppeted into (non-primary) service first. then we can use it as an example case and switch the DNS interface naming and confirm nothing goes crazy before switching the others. [20:54:27] 10Traffic, 10netops, 10Operations, 10ops-codfw: Interface errors on asw-d-codfw:xe-2/0/47 - https://phabricator.wikimedia.org/T193677#4176406 (10ayounsi) [22:07:41] 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Patch-For-Review: Remove wildcard vhost for *.wikimedia.org - https://phabricator.wikimedia.org/T192206#4176764 (10EddieGP) Right, I already wondered whether we need them or they can be removed. I pushed that idea back because I don't want to mix... [22:56:33] 10Traffic, 10Operations, 10Pybal: Reimage LVS servers as stretch - https://phabricator.wikimedia.org/T191897#4176886 (10ayounsi) [22:57:39] 10Traffic, 10netops, 10Operations, 10Pybal: Rename lvs* LLDP port descriptions after upgrading to stretch - https://phabricator.wikimedia.org/T192087#4176884 (10ayounsi) 05Open>03Resolved [23:23:34] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: Rack/cable/configure ulsfo MX204 - https://phabricator.wikimedia.org/T189552#4176934 (10RobH) one of the two routers is now temp racked (not enough rack studs to actually mount, its resting on top of the other servers) with temp power/mgmt leads run. stole t...