[00:10:30] <wikibugs_>	 10Domains, 10Traffic, 06Operations, 06WMF-Legal, 13Patch-For-Review: Using wikimedia.ee mail address as Google account - https://phabricator.wikimedia.org/T158638#3120414 (10Dzahn) @Beetlebeard I have made the change just now (OKed by legal). It will change in about an hour due to caching.
[00:12:38] <wikibugs_>	 10Domains, 10Traffic, 06Operations, 06WMF-Legal, 13Patch-For-Review: Using wikimedia.ee mail address as Google account - https://phabricator.wikimedia.org/T158638#3120422 (10Dzahn) ``` ;; QUESTION SECTION: ;wikimedia.ee.   IN MX  ;; ANSWER SECTION: wikimedia.ee.  3600 IN MX 5 alt1.aspmx.l.google.com. wik...
[07:57:57] <wikibugs_>	 10Traffic, 06Operations, 06Performance-Team, 06Reading-Web-Backlog, and 3 others: Performance review #2 of Hovercards (Popups extension) - https://phabricator.wikimedia.org/T70861#3120888 (10Peter) Where can I find how we reason about the extra delay of 500 ms and 200 ms fade to view as described in https:...
[08:07:20] <wikibugs_>	 10Traffic, 06Operations, 06Performance-Team, 06Reading-Web-Backlog, and 3 others: Performance review #2 of Hovercards (Popups extension) - https://phabricator.wikimedia.org/T70861#3120893 (10phuedx) ^ /cc @Nirzar
[09:26:46] <_joe_>	 ema: ping
[10:21:59] <ema>	 _joe_: yo
[10:22:30] <ema>	 bblack: that seems like a interesting idea!
[10:23:32] <ema>	 bblack: I imagine we should first progressively lower TTL and bump keep anyhow, right? https://gerrit.wikimedia.org/r/#/c/343844/ https://gerrit.wikimedia.org/r/#/c/343845/
[10:28:15] <_joe_>	 ema: I have a few discovery-related dns/puppet changes; I think I know how to merge and test them but I wanted your green light
[10:28:32] <_joe_>	 or if you think I should wait for brandon to be around, that's ok as well
[10:30:13] <ema>	 _joe_: well I haven't dug much into discovery stuff, so if you think that the changes are potentially dangerous I'd suggest to wait for brandon
[10:30:50] <_joe_>	 can you think of a dns change that is not potentially dangerous? :P
[10:30:57] <_joe_>	 so yeah, I'll wait for him
[10:31:03] <_joe_>	 it's ok :)
[11:38:59] <wikibugs_>	 10netops, 06Operations: Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) - https://phabricator.wikimedia.org/T133387#3121483 (10akosiaris) 05stalled>03Open The upgrade to 14.1X53-D42.3 seems to have resolved the issue, at least partially. That is, during tcpdumps before e...
[11:41:36] <wikibugs_>	 10Traffic, 10Monitoring, 06Operations: Performance impact evaluation of enabling nginx-lua and nginx-lua-prometheus on tlsproxy - https://phabricator.wikimedia.org/T161101#3121488 (10ema)
[11:42:02] <wikibugs_>	 10Traffic, 10Monitoring, 06Operations: Performance impact evaluation of enabling nginx-lua and nginx-lua-prometheus on tlsproxy - https://phabricator.wikimedia.org/T161101#3121502 (10ema) p:05Triage>03Normal
[11:43:32] <wikibugs_>	 10netops, 06Operations: Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) - https://phabricator.wikimedia.org/T133387#3121505 (10akosiaris) 05Open>03stalled Resetting to stalled status for 24 hours after some discussions on IRC
[11:59:06] <wikibugs_>	 10netops, 06Operations: Enabling IGMP snooping on QFX switches breaks IPv6 (HTCP purges flood across codfw) - https://phabricator.wikimedia.org/T133387#3121541 (10faidon) So the situation is very unclear and quite messy, which is why I wanted to try this (thanks @akosiaris for tackling this): - We previously h...
[12:18:37] <wikibugs_>	 10Domains, 10Traffic, 06Operations, 06WMF-Legal, 13Patch-For-Review: Using wikimedia.ee mail address as Google account - https://phabricator.wikimedia.org/T158638#3121623 (10Beetlebeard) Hello again. The thing is, since the last verification process completion took too long, G Suite blocked and later del...
[12:52:36] <bblack>	 _joe_: pong
[12:52:52] <bblack>	 ema: yeah, just thinking out loud for later :)
[13:06:31] <bblack>	 _joe_: merged
[13:07:25] <_joe_>	 bblack: uh you merged the ttl patch already? great :)
[13:07:33] <bblack>	 (and seems to work, I bumped appservers-rw to TTL=275 and then queried via DNS)
[13:07:40] <_joe_>	 :)))
[13:07:43] <_joe_>	 great
[13:08:34] <bblack>	 I set the (configured, needs real patches to change) hard outer limits at max=300 and min=10 for them all
[13:08:42] <bblack>	 oh, and I should go set failod to fixed value of 10, I think
[13:09:41] <_joe_>	 seems sensible, yes
[13:10:27] <_joe_>	 https://gerrit.wikimedia.org/r/#/c/344088/ and https://gerrit.wikimedia.org/r/#/c/344093/ added a few discovery entries, I'd like to merge them later
[13:11:56] <bblack>	 later as in, don't shove them through now? :)
[13:14:23] <volans>	 bblack: if you want I've already failoid IPs, pending a puppet CR review for the whole thing to be definitive
[13:15:11] <volans>	 although they technically already works ;)
[13:15:34] <bblack>	 ok, the same host IPs you put in DNS earlier for the t-whatever and r-whatever hosts?
[13:15:48] <volans>	 yes, roentgenium.eqiad.wmnet and tureis.codfw.wmnet
[13:16:20] <volans>	 they're up and running, no iptables so they already do connection refused
[13:16:27] <volans>	 with the puppet patch the will have a proper role
[13:16:43] <volans>	 https://gerrit.wikimedia.org/r/#/c/343917/
[13:25:08] <bblack>	 ok all failoid things fixed up
[13:25:27] <bblack>	 and _joe_ I updated your new discovery dns patch for the 300/10 thing
[13:27:23] <volans>	 great
[13:29:06] * volans lunch
[13:39:39] <wikibugs_>	 10Traffic, 10Monitoring, 06Operations: Performance impact evaluation of enabling nginx-lua and nginx-lua-prometheus on tlsproxy - https://phabricator.wikimedia.org/T161101#3121792 (10ema) === Methodology === We have benchmarked nginx performance on pinkunicorn (cp1008). An additional nginx location block has...
[15:26:25] <ema>	 moritzm: FYI, I've removed linux-image-3.16.0-4-amd64, linux-image-amd64 and linux-image-4.4.0-1-amd64 from cache hosts as you suggested a while ago
[15:26:35] <moritzm>	 thanks
[15:26:53] <moritzm>	 once we've upgraded all jessie systems to 4.9, I'll do that cluster-wide
[15:27:11] <ema>	 nice
[15:50:02] <wikibugs_>	 10Domains, 10Traffic, 06Operations, 06WMF-Legal, 13Patch-For-Review: Using wikimedia.ee mail address as Google account - https://phabricator.wikimedia.org/T158638#3122174 (10Dzahn) @Beetlebeard Ok, no worries. This time it can be faster since we have a general ok and are just fixing it.  Here you go:  ht...
[18:17:00] <wikibugs_>	 10Traffic, 06Analytics-Kanban, 06Operations, 06Wikipedia-iOS-App-Backlog, and 2 others: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2915651 (10JMinor) The version returning to 10% sampling rate is up on the app store.
[19:27:27] <wikibugs_>	 10Traffic, 10DNS, 06Operations: AuthDNS CM/CI refactor - https://phabricator.wikimedia.org/T161148#3122942 (10BBlack)
[19:48:28] <wikibugs_>	 10Traffic, 06Analytics-Kanban, 06Operations, 06Performance-Team, 06Reading-Admin: Preliminary Design document for A/B testing - https://phabricator.wikimedia.org/T143694#3123030 (10Nuria) 05Open>03Resolved
[19:48:32] <wikibugs_>	 10Traffic, 10Analytics, 06Operations: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#3123031 (10Nuria)
[20:19:36] <wikibugs_>	 10Traffic, 06Discovery, 06Operations, 10Wikidata, and 3 others: LDF endpoint ordering is not stable between servers when paging - https://phabricator.wikimedia.org/T159574#3123124 (10Smalyshev) I don't think we need new external endpoint - it looks like our VCL routing allows switching by path (unless I mi...
[21:57:08] <wikibugs_>	 10Traffic, 06Operations, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#1824561 (10StevenJ81) I don't know if it's related, but over the last two weeks my own .js scripts (global, local and gadget) only run sporadically, and I...