[12:33:51] 10Traffic, 10Discovery, 10Operations, 10WMDE-Analytics-Engineering, and 3 others: Allow access to wdqs.svc.eqiad.wmnet on port 8888 - https://phabricator.wikimedia.org/T176875#4001024 (10Addshore) Bump as this is probably trivial but needs the right pair of hands to get it done. [15:05:34] ema: re:numa: https://gerrit.wikimedia.org/r/#/c/414676/ . Haven't tested yet (not even that it doesn't accidentally mess up existing hosts), but in theory this allows for $numa_networking=on to be sanely set with just a depool -> agent run, avoiding the rebooting mess [15:06:55] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Pybal stuck at BGP state OPENSENT while the other peer reached ESTABLISHED - https://phabricator.wikimedia.org/T188085#4001464 (10Vgutierrez) On pybal-test2001 the following behaviour can be observed: ``` pybal -d | grep -i bgp pybal -d 2>&1 | grep -i b... [15:07:46] it looks like current pybal bgp logging is misleading us [15:11:36] I don't have all the deep context, but on a quick scan of that comment, keep in mind that TCP's "ESTABLISHED" in netstat is not the same as BGP being "ESTABLISHED" [15:12:15] sure [15:12:35] but the quagga show bgp neighbours |grep state it's pretty valid :) [15:13:01] I was comparing the "Connect attempts" of the pybal output against the netstat output [15:13:14] yeah that log does seem odd [15:13:24] and the pybal state is now IDLE against the quagga show bgp... [15:13:51] as I try to explain in the issue, pybal plays as TCP client and TCP server at the same time with the configured BGP peers [15:13:52] is it possible pybal's messages are related to more than one TCP-level BGP connection? [15:14:10] I wonder if there's some way to include pybal's TCP port for the BGP connection being logged about to be sure [15:14:13] so several BGP/FSM instances are logging in the same time [15:14:22] *at the same time [15:14:24] right [15:14:32] and we cannot tell one from the others in the log right now [15:14:45] I'm working now in a patch that fixes the logging issue [15:14:51] ok [15:15:40] pybal side TCP port could be a nice option :) [15:16:06] I was thinking in __hash__() attribute of the instance, but your approach has more meaning and it's useful when examining network traffic [15:17:02] one thought that crosses my mind here: it's quite possible that both sides of this BGP session are attempting active open with each other, and not dealing well with solving the race? [15:17:33] (as in, they each both listen from a TCP-level connection from the other, and try to open TCP to the other via SYN, and behavior depends on which side establishes as the client or server first) [15:17:43] indeed [15:17:52] I was checking that possibility as well [15:18:22] since in this particular car we control both sides, we could choose what to do here (e.g. config router side to always be passive (server side of TCP) and pybal to always active-open (be the client) [15:18:27] s/car/case/ [15:19:25] I think (IIRC, I may not!) in the common case between two real routers, both sides attempt active opens when they need to, and routers use something in the state machines to resolve the race and end up with just one established BGP connection? [15:19:44] yes they do [15:19:46] yep, we have that implemented in our BGP FSM implementation [15:19:50] it's in the bgp spec [15:19:57] and pybal does follow it very closely [15:20:06] (could still have a bug there of course, in fact, looks like it does ;) [15:20:15] there https://github.com/wikimedia/PyBal/blob/master/pybal/bgp/bgp.py#L2138-L2175 [15:20:38] at this pace I'm going to know bgp.py by heart... [15:21:16] that's not a bad thing! :) [15:21:33] actually I'm enjoying debugging this :D [15:21:43] * mark enjoyed writing it [15:21:54] was a very welcome distraction during my thesis :D [15:35:16] btw, pybal now supports multiple bgp connections, so just having the pybal-side tcp port may not be enough anymore either [15:35:23] perhaps log the local and destination side? [15:36:42] that's tricky, sometimes local and destination side are not there yet [15:37:10] i.e: the FSM is set in ST_IDLE state before initiating the TCP connection [15:37:23] and you want to tell one FSM from another going to ST_IDLE state :) [15:42:17] yes, also other debug output does use instance addresses [15:42:20] so that may also be useful [15:48:42] i guess start with instance address, add ports where available [15:51:16] I just patched on a quick & dirty way bgp.py to tell between instances on the state log line: https://phabricator.wikimedia.org/P6746 [15:51:40] and looks like it's working as expected [15:51:41] [bgp/8766985089205] INFO: State is now: ESTABLISHED [15:51:46] [bgp/8766990527373] INFO: State is now: IDLE [15:51:47] [-] Stopping factory [15:51:52] try to do something similar I'd say [15:51:58] hex address [15:51:58] yup [15:52:05] bgp@0x7f... [16:04:31] bblack: nice, so the idea is that we should depool hosts with numa on before merging that? [16:05:08] as in, depool, set numa_networking=on, merge the patch? [16:06:35] ema: well the idea is first we merge this new patch (working on validating it's sanity now). Then after it's merged, we do hieradata patches that set numa_networking=on, but we need to depool the affected hosts before merging and applying their hieradata patches (and repool afterwards of course). [16:06:43] s/it's/its/ [16:07:38] or I guess we could take the approach of setting numa_networking=on for a large set of hosts, but before merging the hieradata patch, puppet-disable them all, and then step through them doing "depool; agent-enable+run; pool" [16:08:43] I think you have some recent script related to such sequences :) [16:08:49] (but maybe that was varnish-upgrade-specific) [16:09:48] yes the script is specific to varnish upgrades but could serve as a starting point for one that isn't :) [16:12:43] the net effect of the agent run that applies numa_networking=on will be to do a service reload on nginx (seamless, but cuts process count in half in the new config) and also to blip the ethernet interface, which will cause a couple seconds of dropped packets. [16:14:36] given the "unless" -based execution, probably the sanest way to be absolutely sure that the pending patch doesn't screw with existing otherwise-untouched cache hosts will be to merge it with puppet disabled across them all and trial-run a couple to be sure. [16:15:06] doing that in a few, after I do a few more manual validation steps to try to ensure I'm not deploying something with stupid errors [16:15:12] ok [16:22:32] ah I found an issue, this needs limiting to bnx2x in the general case, so it doesn't try+fail to set such parameters for e.g. lvs100x's bnx2 interfaces [16:23:59] (or cp300{7,8,10}) [16:24:25] oh no, those older cp3's do have bnx2 [16:24:27] err [16:24:29] bnx2x :) [16:29:53] yeah the non-bnx2x cases are going to be cp1008 + lvs1001-6 [16:32:45] hmmmm [16:33:02] handling the cp1008 case proves tricky to do in a puppet-clean fashion with the existing module structure [16:33:20] maybe should have a fact for ethernet interface driver names, that would make this + existing LVS tweaks simpler. [16:50:53] 10netops, 10Operations: cr1-eqsin faulty interfaces - https://phabricator.wikimedia.org/T187807#4001998 (10ayounsi) Most recent update was: > We are pushing the delivery by next week if there everything is smooth and no customs clearance issue. Send on the 24th. Still asking for a more accurate ETA. [16:51:52] 10netops, 10Operations, 10ops-codfw: codfw: mgmt switch replacement in D4 - https://phabricator.wikimedia.org/T187816#4002005 (10Papaul) 05Open>03Resolved Switch replacement complete. - Racktables update - Test serial console of all 3 servers connected to msw-d4-codfw Resolving this task. [17:21:14] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Pybal stuck at BGP state OPENSENT while the other peer reached ESTABLISHED - https://phabricator.wikimedia.org/T188085#4002221 (10Vgutierrez) As suggested by @mark, https://gerrit.wikimedia.org/r/414716 provides unique logging for bgp.py classes based o... [19:50:06] added a new director on cache::misc, design.wm.org -> bromine. running puppet on misc cp, no issues, just fyi [20:12:22] 10netops, 10Operations, 10ops-codfw: switch port configuration for wdq200[4-6] - https://phabricator.wikimedia.org/T188303#4003165 (10Papaul) p:05Triage>03Normal [23:27:38] 10netops, 10Operations, 10ops-codfw: switch port configuration for wdq200[4-6] - https://phabricator.wikimedia.org/T188303#4004161 (10ayounsi) 05Open>03Resolved Descriptions added, enabled (for the 1 that was not already), moved to the private vlan.