[02:06:28] 10Traffic, 10Wikimedia-Apache-configuration, 10DNS, 10Operations: m.{project}.org portal/redirect consistency - https://phabricator.wikimedia.org/T78421#4320893 (10MZMcBride) [02:06:30] 10Wikimedia-Apache-configuration, 10Discovery, 10Zero, 10Mobile: m.wikipedia.org and zero.wikipedia.org should redirect differently - https://phabricator.wikimedia.org/T69015#4320890 (10MZMcBride) 05Open>03Resolved a:03MZMcBride Yessir. Thank you for noting the behavior as of 2018-06-27 in this task! [07:55:48] elukey: how do you feel today about doing cache::text nodes? [07:56:04] gooood! [07:56:11] remember that we have 3 vk instances in there [07:56:37] yup [07:56:47] do I need to wait between vk restarts in the same instance? [07:58:00] vgutierrez: there shouldn't be any issue but let's add a couple of seconds between each just to be sure [08:01:23] cool [08:15:48] elukey: it's on its way.. how do you want to proceed with the config change? [08:16:13] aka https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440544/ [08:18:28] vgutierrez: if possible I'd move ssl_cipher_suites, ssl_curves_list and ssl_sigalgs_list as profile parameters, with defaults as ECDHE-ECDSA-AES256-GCM-SHA384, undef, undef (ssl_cipher_suites is not in the scope but since we are refactoring it seems good for consistency) [08:18:37] and then apply the new values only to the role cache misc [08:18:48] let it boil one day just to observe TLS latencies etc.. [08:19:25] then upload and finally just changing the defaults to P-256 and ECDSA+SHA256 [08:19:40] if you want I can take care of the config + restarts this time [08:20:18] I know that this change should be harmless but again, better safe than sorry [08:21:22] sure, that's ok [08:21:27] super [08:21:38] let me refactor that CR then [08:22:06] <3 [08:32:40] elukey: IMHO we should split it in two commits [08:32:54] 1. Enable the parameters configuration [08:32:58] 2. Configure it for cache::misc [08:33:32] first one should be a noop in puppet terms of course [08:37:28] +1 [08:37:32] I like it [08:42:35] elukey: cache::text done BTW, looking good here [08:44:33] super [08:50:44] elukey: so [08:50:45] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440544/ [08:50:51] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/442794/ [08:52:30] +1 and +1 [08:52:34] thanks! [08:52:54] terrific! [09:02:45] elukey: I'm going to let puppet run on one cache::misc node and restart manually varnishkafka there, if that goes well I'll let puppet run in the rest of cache::misc nodes [09:03:23] cp3007.esams.wmnet [09:03:26] vgutierrez: I don't recall if we have auto-restart on config change, but in any case +1 [09:04:20] cp3007 looks good [09:04:56] Info: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/File[/etc/varnishkafka/webrequest.conf]: Scheduling refresh of Service[varnishkafka-webrequest] [09:05:00] Notice: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/Base::Service_unit[varnishkafka-webrequest]/Service[varnishkafka-webrequest]: Triggered 'refresh' from 1 events [09:06:03] looks like puppet triggered the refresh of the service [09:07:03] elukey: I'm going to restart vk there [09:07:22] 10netops, 10Operations: Allow labnet/labnodepool/labvirt to connect to debmonitor hosts/443 - https://phabricator.wikimedia.org/T198375#4321131 (10MoritzMuehlenhoff) [09:07:48] done :) [09:10:27] elukey: can I hit the whole cache::misc cluster? [09:10:45] sure [09:27:45] all done :D [09:28:06] elukey: let's schedule upload for Monday and text for Tuesday? [09:30:27] I think that upload is fine tomorrow [09:30:40] ack [09:31:42] thanks for the work and the patience! [09:31:47] will keep metrics monitored [09:32:11] no problem :) [12:07:59] 10Traffic, 10Operations, 10Goal: Establish timeline and methodology for upcoming deprecation of non-forward-secret ciphers and TLSv1.0 - https://phabricator.wikimedia.org/T192559 (10Vgutierrez) Our [[ https://grafana.wikimedia.org/dashboard/db/tls-ciphersuite-explorer?panelId=2&fullscreen&orgId=1&from=now-30... [12:45:06] vgutierrez: cache misc looks good, if you have time we can proceed with upload and then tomorrow text [12:56:42] elukey: awesome [13:04:44] 10Traffic, 10Operations, 10Goal: Establish timeline and methodology for upcoming deprecation of non-forward-secret ciphers and TLSv1.0 - https://phabricator.wikimedia.org/T192559 (10BBlack) Going a bit beyond the explicit scope of this ticket, there are really a few different legacy-support risks we'd like t... [13:06:50] elukey: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/442840 [13:07:20] +1 [13:17:43] applying it already, it's going to take a while :) [13:18:07] we've 45 upload nodes, our biggest cache cluster [13:18:41] ack! [13:24:11] 10Traffic, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993 (10Ottomata) Woo hoo! Annnnd soon we disable IPSec?! :D [14:06:29] 10Traffic, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993 (10Vgutierrez) >>! In T182993#4321845, @Ottomata wrote: > Woo hoo! > > Annnnd soon we disable IPSec?! :D As soon as we rollout this on cache::... [14:07:58] vgutierrez: about --^ - how should we rollout the ipsec deprecation? Depooling each cp host one by one, removing ipsec, etc.. ? [14:10:52] maybe bblack have some insights ^^ [14:11:21] s/have/has/ [14:28:01] elukey: cache::upload done as well, all good :) [14:31:24] \o/ \o/ [14:52:43] 10netops, 10Operations, 10ops-eqdfw: eqdfw: Patch GTT cross-connect - https://phabricator.wikimedia.org/T194515 (10ayounsi) 05Open>03Resolved The LOA was incorrect, Equinix moved it to the proper one and link is up. [15:00:42] I'll be late by a bit, in another meeting [15:01:14] ack [15:12:47] 10Traffic, 10DNS, 10Operations, 10ops-eqiad: rack/setup/install authdns1001.wikimedia.org - https://phabricator.wikimedia.org/T196693 (10Cmjohnson) [19:33:00] 10netops, 10Operations: Allow labnet/labnodepool/labvirt to connect to debmonitor hosts/443 - https://phabricator.wikimedia.org/T198375 (10ayounsi) 05Open>03Resolved a:03ayounsi Policy added: ```lang=diff [edit firewall family inet filter labs-in4] + term debmonitor { + from { +... [22:56:55] 10netops, 10Operations, 10ops-codfw: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10Papaul) 05Open>03Resolved a:03Papaul switch configuration done Interface Admin Link Description ge-5/0/17 up down graphite2003 [23:08:44] 10netops, 10Operations, 10ops-codfw: Swith port information for authdns2001 - https://phabricator.wikimedia.org/T198126 (10Papaul) 05Open>03Resolved a:03Papaul switch port configuration done Interface Admin Link Description ge-5/0/5 up down authdns2001 [edit interfaces interface-rang... [23:16:55] 10netops, 10Operations, 10SRE-Access-Requests: Get Papaul access to network equipment - https://phabricator.wikimedia.org/T198344 (10ayounsi) 05Open>03Resolved Talked to Papaul on IRC, key push to asw-a/b/c/d-codfw and will be pushed progressively to more devices. I gave him a Juniper configuration and... [23:18:49] 10netops, 10Operations, 10ops-codfw: Swith port information for authdns2001 - https://phabricator.wikimedia.org/T198126 (10Papaul) [edit interfaces interface-range vlan-public1-a-codfw] member ge-5/0/23 { ... } + member ge-5/0/5; [edit interfaces interface-range disabled] - member ge-5/...