[07:49:23] 10Traffic, 10Analytics, 10Analytics-EventLogging, 10Operations, 10Performance-Team: Increase EventLogging limit from 2K to 5K - https://phabricator.wikimedia.org/T208282 (10ema) p:05Triage>03Normal [09:02:50] 10Certcentral: retrying policy currently ignores self_signed status - https://phabricator.wikimedia.org/T208378 (10Vgutierrez) [09:03:31] 10Certcentral: retrying policy currently ignores self_signed status - https://phabricator.wikimedia.org/T208378 (10Vgutierrez) p:05Triage>03Normal [10:36:39] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row C recable and add QFX - https://phabricator.wikimedia.org/T208272 (10elukey) There is a problem in the schedule I am afraid.. Nov 1st is holiday for most of the Europeans, plus I am a bit concerned about DBA presence since @Banyek and and M... [10:37:27] about --^ there seems to be a network maintenance in codfw happening tomorrow that is holiday for most of the EU [10:41:51] and it would be probably better to reschedule when the data persistence team is available [10:41:57] Cc: XioNoX --^ [10:43:24] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row C recable and add QFX - https://phabricator.wikimedia.org/T208272 (10Joe) >>! In T208272#4706141, @ayounsi wrote: > Here is the full list of hosts in that row. No outages expected, but brief (5s) connectivity interruption for some racks is... [10:44:55] heh "losing a full row can be an issue for cassandra" [10:45:04] what's the purpose of having machines spread across rows if not that? :P [10:45:06] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row C recable and add QFX - https://phabricator.wikimedia.org/T208272 (10Joe) To be clear: I think we should do the maintenance **without depooling anything** and check what would happen when we lose a row, even if in an inactive datacenter. Bu... [11:21:30] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row C recable and add QFX - https://phabricator.wikimedia.org/T208272 (10fgiunchedi) >>! In T208272#4708611, @Joe wrote: > we will most likely need to run switftrepl after the outage to catch up on missing originals. Should we failover traffic... [12:08:55] 10Certcentral, 10Traffic, 10DNS, 10Operations: Allow Let's Encrypt issue wildcard certificates - https://phabricator.wikimedia.org/T208390 (10Vgutierrez) [12:09:39] 10Certcentral, 10Traffic, 10DNS, 10Operations: Allow Let's Encrypt issue wildcard certificates - https://phabricator.wikimedia.org/T208390 (10Vgutierrez) p:05Triage>03Normal [12:22:16] vgutierrez: https://gerrit.wikimedia.org/r/c/operations/dns/+/470816/1/templates/wikimedia.org [12:22:28] should be fine to push that when you're ready, and then I think revert it later when testing done [12:44:43] elukey: yeah definitely [12:46:19] super thanks! [13:10:57] 10netops, 10Operations, 10ops-codfw, 10Patch-For-Review: codfw row C recable and add QFX - https://phabricator.wikimedia.org/T208272 (10Eevans) >>! In T208272#4708611, @Joe wrote: >>>! In T208272#4706141, @ayounsi wrote: >> >> [ ... ] >> >> restbase2003 >> restbase2004 >> restbase2008 >> restbase2011 > >... [13:53:20] bblack: ack, thx! [14:20:52] mark, bblack, ema, vgutierrez etc.: https://code.fb.com/open-source/open-sourcing-katran-a-scalable-network-load-balancer/ [14:21:14] yes [14:21:20] we discussed it here a while back [14:21:31] oh you did? [14:21:34] sorry :) [14:21:58] I now see it's older, I just saw it linked from today's https://code.fb.com/open-source/linux/ [14:22:11] yes [14:24:22] 10Wikimedia-Apache-configuration, 10Patch-For-Review: Clean up redirects.conf/redirects.dat (remove en2.wikipedia.org, etc.) - https://phabricator.wikimedia.org/T105981 (10Krenair) a:05Krenair>03None [14:24:33] 10Wikimedia-Apache-configuration: Clean up redirects.conf/redirects.dat (remove en2.wikipedia.org, etc.) - https://phabricator.wikimedia.org/T105981 (10Krenair) [14:34:15] 10Certcentral: Test wildcard certificate issuance with certcentral - https://phabricator.wikimedia.org/T208424 (10Vgutierrez) [14:34:34] 10Certcentral, 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: Allow Let's Encrypt issue wildcard certificates - https://phabricator.wikimedia.org/T208390 (10Vgutierrez) [14:34:36] 10Certcentral: Test wildcard certificate issuance with certcentral - https://phabricator.wikimedia.org/T208424 (10Vgutierrez) [14:42:46] I'll test the wildcard cert issuance after my dentist appointment (in 18 minutes), I don't want to leave leaving things on an intermediate state [14:43:16] ack [15:36:02] CAA record updated [15:36:52] with our initial cercentral tests we attempted to get a wildcard certificate and we failed cause LE honoured our CAA record [15:37:07] but.. we didn't get any mail alerting us? [15:38:30] vgutierrez, about the failure? [15:38:39] sigh.. I'm not in dns-admin@wm.o mailing alias [15:38:57] oh right the CAA failure mail thing [15:50:11] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10Ottomata) Just had a great meeting with @chasemp, @faidon, @JAllemandou and @nuria. The main action item (after Nuria h... [15:50:59] I am and I didn't see anything about that [15:53:25] 10netops, 10Analytics, 10Analytics-Kanban, 10Operations: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster - https://phabricator.wikimedia.org/T207321 (10chasemp) My notes from the 2018-10-31 meeting: ```https://phabricator.wikimedia.org/T207321#4691776 * hosts that push... [16:05:50] 10Certcentral, 10Patch-For-Review: Test wildcard certificate issuance with certcentral - https://phabricator.wikimedia.org/T208424 (10Vgutierrez) certcentral has been able to get the certificates in both nodes. No manual operation has been required, the change https://gerrit.wikimedia.org/r/470846 has been mer...