[08:26:35] 10Traffic, 06Discovery, 06Operations, 03Discovery-Search-Sprint: Setup load balancing for elasticsearch service on relforge servers - https://phabricator.wikimedia.org/T142098#2532030 (10Gehel) [08:44:52] 10Traffic, 06Discovery, 06Operations, 03Discovery-Search-Sprint: Setup load balancing for elasticsearch service on relforge servers - https://phabricator.wikimedia.org/T142098#2532106 (10Gehel) The relforge cluster is not H/A, with 2 nodes we can only have a single master if we want to protect again a spli... [08:50:39] 10Traffic, 06Discovery, 06Operations, 03Discovery-Search-Sprint: Setup load balancing for elasticsearch service on relforge servers - https://phabricator.wikimedia.org/T142098#2532123 (10dcausse) @Gehel it's perfectly OK to run without H/A, this cluster is not meant to serve production traffic but only opt... [12:03:04] 10netops, 06Discovery, 06Discovery-Search-Backlog, 10Elasticsearch, and 2 others: Enable access to relforge clusters from virtual machines running on labs - https://phabricator.wikimedia.org/T142211#2532869 (10Gehel) Could someone in #netops open traffic from labs-instance to the 2 relforge servers on port... [12:29:01] 10netops, 06Discovery, 06Discovery-Search-Backlog, 10Elasticsearch, and 2 others: Enable access to relforge clusters from virtual machines running on labs - https://phabricator.wikimedia.org/T142211#2532907 (10Gehel) I realize that some context is probably missing here. The detailed discussion about the re... [12:52:14] 10Traffic, 10Varnish, 06Operations: Varnish 4 stalls with two consecutive Range requests using HTTP persistent connections - https://phabricator.wikimedia.org/T142233#2532925 (10ema) Issue [[https://github.com/varnishcache/varnish-cache/issues/2035 | submitted upstream]]. [13:36:15] 10netops, 06Discovery, 06Discovery-Search-Backlog, 10Elasticsearch, and 2 others: Enable access to relforge clusters from virtual machines running on labs - https://phabricator.wikimedia.org/T142211#2527394 (10BBlack) We don't in general open firewall holes between prod and labs like that. This may need r... [15:53:45] so, I'm tired of waiting for our stupid new vendor to finish our second {es,kn}ams-eqiad link [15:53:57] this is currently blocking the upgrade of cr1-eqiad, which I'd like to get it over with [15:54:39] so you're thinking about depooling esams and doing the upgrade first? [15:55:23] no [15:55:27] my current thinking is [15:55:32] drain esams [15:55:43] then move GTT to cr2 (and Level3 to cr1) [15:55:48] undrain esams [15:55:51] then upgrade cr1 [15:57:02] you'd need someone on-site for moving it right? [15:57:04] yeah [15:58:03] thoughts/objections? [16:01:53] seems sane, if you can have someone onsite to do it easier than continuing to wait [16:46:58] 10netops, 06Discovery, 10Elasticsearch, 06Operations, 03Discovery-Search-Sprint: Enable access to relforge clusters from virtual machines running on labs - https://phabricator.wikimedia.org/T142211#2533836 (10ksmith) [17:08:37] 10Traffic, 10Analytics, 06Operations: Correct cache_status field on webrequest dataset - https://phabricator.wikimedia.org/T142410#2533998 (10Nuria) [19:50:52] 10Traffic, 10Analytics, 10Analytics-Cluster, 06Operations: Enable Kafka native TLS in 0.9 and secure the kafka traffic with it - https://phabricator.wikimedia.org/T121561#2534758 (10Ottomata) a:05Ottomata>03None [20:33:00] so I was rummaging through raw clienthello data, and I've found some clients aren't covered by cloudflare's chapoly preference hacks [20:33:28] at least, I think. I'm running a newer/better capture now. [20:33:46] but it's possible there are some clients that fall into one of two buckets: [20:34:09] 1) They support chapoly higher on their list than aes-gcm, but some other stupider cipher ahead of both (so cf patch turns off chapoly for them) [20:34:25] 2) They support chapoly as their *only* AEAD cipher (no aes-gcm), but again not first on the list [20:38:38] I'm not really sure though, I may have had a code bug with sorting them, so, capturing some samples again [20:39:33] full equal-preference-group support wouldn't be perturbed by such things, but the minimal cloudflare "turn on chapoly if the client has it first on their list" approach would be [20:40:33] fixing it isn't too hard though, on the openssl side [21:02:16] 10Traffic, 06Operations, 13Patch-For-Review, 05WMF-deploy-2016-08-09_(1.28.0-wmf.14): Decom bits.wikimedia.org hostname - https://phabricator.wikimedia.org/T107430#2534974 (10Krinkle) >>! In T107430#2520799, @Krinkle wrote: > The Commons app for Android (previously by Wikimedia, now community-maintained) a... [21:50:09] yup, even in fairly limited short samples, I'm finding such clients [21:51:11] there's one unique string that's nearly a full percent of traffic in my sample, where it has 6 crappy ciphers at the start of the list, then chapoly, then the good aes-gcm stuf. [21:52:03] (and a handful of other such lists with less percentage) [21:52:43] and I've found two unique ciphersuite lists that have chapoly and no form of strong aes-gcm (so chapoly is their only "strong" option), but they do have chapoly at the front of their respective lists [21:54:09] one thing I haven't found yet, though, is any client that implements a DHE+chapoly cipher and doesn't haven an ECDHE equivalent we'd like better. [22:48:24] 07HTTPS, 10Traffic, 06Operations, 06Release-Engineering-Team: Retire gerrit.wikimedia.org SSL cert - https://phabricator.wikimedia.org/T142131#2523742 (10Dzahn) deleted cert from public repo. deleted key from private repo. this should be all then [22:52:15] 07HTTPS, 10Traffic, 06Operations, 06Release-Engineering-Team: Retire gerrit.wikimedia.org SSL cert - https://phabricator.wikimedia.org/T142131#2535257 (10Dzahn) 05Open>03Resolved a:03Dzahn /etc/ssl/localcerts and /etc/ssl/private on lead were already cleaned