[00:10:37] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2086965 (10Jdlrobson) >>! In T124356#2086258, @Jd... [01:19:28] 10Traffic, 6Operations, 10RESTBase, 6Services, and 2 others: Split slash decoding from general percent normalization in Varnish VCL - https://phabricator.wikimedia.org/T127387#2087179 (10GWicke) p:5High>3Normal [11:52:55] 10Traffic, 6Analytics-Kanban, 6Operations, 13Patch-For-Review: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2088214 (10elukey) @Ottomata and @faidon: thanks a lot for all the info! >For this, my own recommendation would be to either add JSON & librdkaf... [11:59:30] 10Traffic, 6Analytics-Kanban, 6Operations, 13Patch-For-Review: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2088220 (10BBlack) >>! In T124278#2088214, @elukey wrote: > but we had to import a lot of .c files into varnishkafka's source. An alternative p... [12:03:16] bblack: very good point, from what I've seen it seems feasible to import vut without breaking any licensing but I need to double check. We can definitely simply use directly the API if anything looks weird but as it might take a bit more time I'd prefer to do it after finishing the port of the parsing part. Does it sound reasonable? [14:05:23] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Switch ulsfo to backend to codfw rather than eqiad - https://phabricator.wikimedia.org/T127492#2088421 (10BBlack) [14:05:25] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Enable VCL source-DC switching via confd - https://phabricator.wikimedia.org/T127482#2088422 (10BBlack) [14:05:28] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2088423 (10BBlack) [14:07:10] 10Traffic, 6Operations: Port varnishlog.py to new VSL API - https://phabricator.wikimedia.org/T128788#2088424 (10ema) a:3ema [14:23:23] bblack: re: nginx/systemtap, yesterday evening I went digging a bit deeper, something like this seems to be a good way to hook into alpn: [14:23:26] probe process("/usr/sbin/nginx").function("ngx_http_ssl_alpn_select").return { printf("%s\n", out) } [14:23:46] 10Traffic, 6Operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2088448 (10BBlack) Status Update: The first chunk of work is done: we can supposedly do all of the switching in steps 1-4 in the descrip... [14:23:52] I couldn't *really* get the actual value because out is a char ** [14:23:57] but yeah, close ;) [14:24:07] :) [14:24:26] nginx is printing that thing in debugging mode [14:24:49] SSL ALPN selected: h2 [14:25:04] well yeah but we need the list the client sent, not the item from the list nginx selected [14:25:22] that's also there, eg: [14:25:22] 2016/03/04 00:32:43 [debug] 12241#12241: *1 SSL ALPN supported by client: h2 [14:25:24] (ditto for NPN) [14:25:26] 2016/03/04 00:32:43 [debug] 12241#12241: *1 SSL ALPN supported by client: h2-16 [14:25:29] 2016/03/04 00:32:43 [debug] 12241#12241: *1 SSL ALPN supported by client: h2-14 [14:25:36] ah yes, that [14:27:13] it would be nice to confirm whether the NPN assumptions we generally make hold true, either via those probes or via a more-complicated sniff that can decrypt and such [14:28:03] the NPN assumptions being (1) If client sends NPN, client must support SPDY and (2) If client sends NPN but no ALPN, client supports SPDY but not H2 [14:28:35] although point (2) may be irrelevant in practice if nginx only supports H2 w/ ALPN and dumped NPN. I think I looked at that before but don't remember now. [14:31:18] actually I think you can't send an h2 protocol over NPN anyways, so point (2) is irrelevant and (1) is the only true assumption. [14:32:03] but it's standards-legal that some client could choose to send NPN, but the NPN only indicates http/1.1 support and not SPDY [14:32:19] (which would move some of our supposed SPDY-only client stats into the None category) [14:32:58] we just expect that to be rare because there's was never a point to implementing NPN in the first except to support SPDY heh [14:33:38] right [14:33:57] (sending NPN with http/1.1 alone should be illegal BTW) [14:34:35] well sending NPN should be illegal, it never made it out of draft status and got replaced by something better :) [14:34:45] :) [14:35:53] oh jeez, I just re-read the NPN draft [14:36:01] actually, the client never sends a list, only the server does [14:36:26] clienthello is just "Yes I support NPN", then the server sends an NPN proto list, then the client picks one. [14:36:26] uh [14:37:06] so, no real point trying to decode that I guess [14:37:23] we could still look at existing connections to our SPDY servers today, though, and see if some NPN clients end up selecting http/1.1 [14:44:36] bblack: what is the goal of making the NPN assumptions again? [14:46:27] 10Traffic, 6Analytics-Kanban, 6Operations, 13Patch-For-Review: varnishkafka integration with Varnish 4 for analytics - https://phabricator.wikimedia.org/T124278#2088466 (10Ottomata) I think we have control over the varnishkafka license, so if we go that route and need to change it, we can. [15:04:31] ema: well the point is we want to split up all our clients into bins (h2 or spdy or both or none), to evaluate the tradeoff of disabling SPDY to gain H2. For clients that send ALPN, or don't send any NPN/ALPN, it's easy to bin them. [15:04:59] got it [15:05:06] we assume a client that sends only NPN means a client that falls into the spdy-only bin, but I don't know if that assumption's valid (maybe some of them go in the None bin, because they only do http/1.1+NPN) [15:07:06] and as crazy as it sounds to do NPN without SPDY, there is a possible scenario: a browser may implement NPN+SPDY, but for some pragmatic reason or other, when built for slightly-older OS versions the SPDY gets disabled in the build for some lack of supporting stuff, but the NPN part remains turned on, with only http/1.1 [15:09:16] but if that's happening for any significant volume of clients, even a hacky test for it should catch it within a short window checking on a few caches. we don't necessarily have to test that full-blast, just be able to do some sampling to confirm the assumption. [15:13:53] 10Traffic, 6Operations, 13Patch-For-Review: Port varnishlog.py to new VSL API - https://phabricator.wikimedia.org/T128788#2088523 (10ema) p:5Triage>3Normal [15:30:29] 10Traffic, 6Operations, 13Patch-For-Review: Upgrade to Varnish 4: things to remember - https://phabricator.wikimedia.org/T126206#2088570 (10Southparkfan) I know that you guys have way more knowledge of (upgrading) Varnish than I do (and Wikimedia's needs are totally different compared to mine), but I alread... [16:34:10] 10Traffic, 6Operations, 13Patch-For-Review: Port varnishlog.py to new VSL API - https://phabricator.wikimedia.org/T128788#2088709 (10ema) Note that lots of tags have changed. Taking a quick look at varnishreqstats, for example, we need to fix: TxRequest, RxStatus, RxRequest, TxStatus, ReqEnd. I couldn't fi... [17:14:07] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#1997604 (10bd808) https://www.mediawiki.org/wiki/... [18:13:59] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2089122 (10Samat) And [[ https://hu.wikipedia.org... [19:22:21] 7HTTPS, 10Traffic, 6Operations: SSL cert needed for benefactorevents.wikimedia.org - https://phabricator.wikimedia.org/T115028#2089525 (10CCogdill_WMF) a:3CCogdill_WMF Claiming this task since I think the ball is in my court. Eric at Trilogy confirmed he received the cert and will have SNI set up for us by... [22:17:43] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2090118 (10Legoktm) Hrm...so once we remove the F... [22:24:23] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2090145 (10Legoktm) I think fully reverting {30f3... [22:37:53] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2090177 (10Jdlrobson) @legoktm a full revert isn'... [22:43:22] 10Traffic, 10MobileFrontend, 6Operations, 5MW-1.27-release, and 6 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2090218 (10Legoktm) Uh. We need LESS hacks, not m...