[07:26:20] 10Traffic, 10Operations, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Joe) [07:35:25] 10netops, 10Operations, 10User-Elukey: migrate netinsights from rhenium to sulfur - https://phabricator.wikimedia.org/T212011 (10faidon) a:05faidon→03MoritzMuehlenhoff Moritz was working on that. [09:59:32] 10Traffic, 10Operations, 10Phabricator: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10ema) 05Open→03Resolved a:03ema Yes @mmodell, now I can. Thank you! [10:45:04] 10Traffic, 10Operations, 10Phabricator: Allow traffic team to manage the traffic blog on phame - https://phabricator.wikimedia.org/T204355 (10Peachey88) a:05ema→03mmodell [12:43:26] mmmh [12:44:16] vgutierrez: so, I've just restarted pybal on lvs2003 and the BGP FSM graph looks interesting https://grafana.wikimedia.org/d/000000488/pybal-bgp?orgId=1&var-datasource=codfw%20prometheus%2Fops&var-server=lvs2003&var-asn=64600&from=1545049887490&to=1545050581101 [12:45:58] somehow pybal reports 1 established-active, 1 established-passive and 1 idle-active bgp connections [12:46:21] that did not happen when bouncing pybal on lvs2006 (we always have just 1 established-active there) [12:47:49] CC: mark [12:49:25] according to the BGP logs it does seem like everything went fine though (and indeed traffic has been re-routed to lvs2003 properly) [12:50:05] that's normal between pybal restarts [12:50:25] mmh [12:50:36] how comes that did not happen on lvs2006 tho? [12:50:55] chance [12:51:34] there the old pybal instance acted as the active peer [12:51:40] and the second one as passive [12:55:16] mh [13:07:53] also in the new one [13:08:38] the active FSM is on idle state cause the router managed to connect faster and got the active role [13:11:07] 10Traffic, 10Operations, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:11:13] 10Traffic, 10Operations, 10Maps (Kartotherian), 10Patch-For-Review: kartotherian TLS support - https://phabricator.wikimedia.org/T211970 (10ema) 05Open→03Resolved [13:15:27] 10Traffic, 10Operations, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) I do agree with @Joe, without a proper PKI this is going to be painful. For now however I've added TLS support to kartotherian (T211970) as that's part of cache_upload, which is the cluste... [13:15:41] 10Traffic, 10Operations, 10serviceops: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [17:33:40] 10netops, 10Operations, 10ops-codfw: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10Papaul) fpc2-fpc8 connection xe-2/0/41 and xe-2/0/42 fpc7-fpc8 connection xe-7/0/43 and xe-7/0/44 [17:34:00] 10netops, 10Operations, 10ops-codfw: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10Papaul) [17:38:55] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row B recable and add QFX - https://phabricator.wikimedia.org/T210456 (10Papaul) [17:39:58] 10netops, 10Operations, 10ops-codfw: upgrade all codfw switch stacks to include additional 10G switch per row - https://phabricator.wikimedia.org/T196489 (10Papaul) [17:40:01] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row B recable and add QFX - https://phabricator.wikimedia.org/T210456 (10Papaul) 05Open→03Resolved Complete [18:28:25] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10Papaul) connected to scs-c1-codfw on port 48 [18:28:38] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10Papaul) [18:59:17] 10netops, 10Operations, 10ops-codfw: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10ayounsi) [19:00:21] 10netops, 10Operations, 10ops-codfw: codfw row A recable and add QFX - https://phabricator.wikimedia.org/T210447 (10ayounsi) Some more shuffling around to facilitate DCops work (as row A has more machines), scheduling row A Tomorrow and row D Wednesday. [19:00:56] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10ayounsi) [19:01:02] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10ayounsi) Some more shuffling around to facilitate DCops work (as row A has more machines), scheduling row A Tomorrow and row D Wednesday. [19:23:12] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10Papaul) [19:38:18] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review, 10User-jijiki: codfw row D recable and add QFX - https://phabricator.wikimedia.org/T210467 (10ayounsi)