[10:32:46] 10Traffic, 6Operations: don't serve upload.wikimedia.org 'root' from wmfrewrite/swift - https://phabricator.wikimedia.org/T130709#2143868 (10fgiunchedi) [10:32:57] 10Traffic, 6Operations, 10media-storage: don't serve upload.wikimedia.org 'root' from wmfrewrite/swift - https://phabricator.wikimedia.org/T130709#2143882 (10fgiunchedi) [13:02:41] bblack, ema - whenever you have time it would be great to discuss the next steps for varnishkafka [13:05:03] elukey: alright! My understanding is that it's pretty much good to go for initial testing except for the sleep time that needs to be lowered [13:07:06] ema: yes I'll need to do some testing to make sure that it is good perfomance wise [13:07:23] (better safe than sorry) [13:07:40] it would be also great to decide how the testing scenario would look like [13:09:02] "testing scenario" sounds so optimistic about how well we're able to test things :) [13:09:13] :) [13:09:16] 'morning [13:11:57] I imagine the testing scenario is deploying it on a prod machine and observing what goes wrong [13:13:43] bblack: testing prod environment sounds better? :P [13:14:09] we'll call it Early Beta Access for Awesome Deserving Users [13:14:15] lol [13:15:20] but more-seriously, if we think it works correctly based on limited manual testing, and the changes don't look like we've done stupid or scary things... [13:15:38] yeah the next best thing is watch maps stats/logs when we push it to a real maps host [13:16:50] elukey: on the sleep thing: 10ms is what VUT uses, and makes more sense. ema tested one instance like that and it didn't use much CPU. [13:16:56] that value should be good to go [13:17:24] 10ms is a long time for a CPU, but much better than 100ms in terms of window of requests to a single machine [13:17:25] all right, I am going to test it today in my VM and will update the code review [13:17:31] ok [13:18:55] in the meantime I'm looking at the compiler output for the vcl4 porting patch https://puppet-compiler.wmflabs.org/2131/cp1043.eqiad.wmnet/ [13:19:30] there are a few things we missed when rebasing and lots of indentation differences which are pretty annoying [13:19:31] bblack: the other thing that we should triple check with you (already looked with ema) are the new timings that we grab (TTFB, etc..) [13:20:00] yeah [13:26:54] code review updated: https://gerrit.wikimedia.org/r/#/c/276439/14 (nanosleep set with tv_nsec = 10000000L). Tested with and without load, perfomances looks good [13:27:05] nice [13:53:28] 10Traffic, 6Operations, 7discovery-system: Properly package confd and its dependencies - https://phabricator.wikimedia.org/T97971#2144177 (10Joe) 5Open>3declined [13:54:29] 10Traffic, 6Operations, 7discovery-system: Properly package confd and its dependencies - https://phabricator.wikimedia.org/T97971#1256145 (10Joe) I plan on distributing confd via a different method as building it now requires go 1.6, so there is no way I can do that with go 1.3 (which we have on jessie) [14:32:33] elukey: https://gerrit.wikimedia.org/r/#/c/276439/ <- bunch of comments, most of it's just code style stuff [14:33:08] the upside is when people can't find things to bitch about other than style, you're on the right track :) [14:38:14] bblack: thanks for the review! As you may have noticed I hate ifs without {} :P [14:38:52] yeah that was over half the comments lol [14:39:05] but either change the style of the whole file before this patch, or stick with its style [14:54:55] yep completely right [15:05:28] almost reaching 30 patchsets in the vcl porting CR [15:05:56] * ema wonders what the current record is [15:24:34] :O [15:28:32] alright, with the last patch we should have a decent puppet compiler diff [16:02:29] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Varnish support for shutting users out of a DC - https://phabricator.wikimedia.org/T129424#2144618 (10BBlack) 5Open>3Resolved [16:02:32] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#2144619 (10BBlack) [16:02:47] 10Traffic, 6Operations, 13Patch-For-Review, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Traffic Infrastructure support for Mar 2016 codfw rollout - https://phabricator.wikimedia.org/T125510#1989606 (10BBlack) 5Open>3Resolved a:3BBlack [16:15:48] bblack: compiler output with v4 patch, one host per cluster https://puppet-compiler.wmflabs.org/2141/ [16:22:57] ema: awesome, looks pretty sane [16:23:20] ema: can you re-run again with multiple DCs as well though? so we make sure we've covered the strange effects of tiering and such [16:23:43] I guess a run with 4x hosts per cluster (one each eqiad, codfw, esams, ulsfo) [16:23:51] except for maps, which doesn't have the other DCs heh [16:24:11] https://bagder.gitbooks.io/http2-explained/content/en/part6.html --> Multiplexed streams --> train pictures [16:25:10] I like the lego header thing too :) [16:25:21] * legoktm clicks [16:26:03] heh :) [16:30:26] 7HTTPS, 10Traffic, 10Monitoring, 6Operations, 7Icinga: ssl expiry tracking in icinga - we don't monitor that many domains - https://phabricator.wikimedia.org/T114059#2144698 (10Dzahn) [16:30:52] elukey: you should send that http2 guide to wikitech-l :-) [16:40:26] bblack: done, poor pcc [16:40:28] https://puppet-compiler.wmflabs.org/2142/ [16:43:43] ema: I've looked through a few so far, still looks sane :) [16:43:58] I have to run out for lunch and can't be late, but will read through all of them carefully after [16:44:15] IMHO assuming no issues, we should go ahead and merge this today and watch for puppetfails, etc [16:44:21] bblack: wonderful! enjoy your lunch [16:44:38] I'll keep on going through the diffs in the meanwhile [16:44:47] ok thanks [17:31:40] mmh [17:31:41] - if (req.http.X-Pass-Stream) { [17:31:41] + if (<%= @bereq_req %>.http.X-Pass-Stream) { [17:31:57] that's probably needed for v4 [17:32:05] except for that, it looks good! [17:35:29] with the above patch, all vtc tests are green on my maps instance (v3 and v4) [18:41:16] bblack: I have to go now, feel free to merge though if you wish! See you tomorrow :) [18:51:16] ok bye! [18:51:36] elukey: I'm splitting up your patch into a pre-patch for the source formatting cleanup on the old code and such, and then the functional one [19:27:39] elukey: ok, the patch is split up with the cleanup prepatch at https://gerrit.wikimedia.org/r/#/c/279173 and the real work still at https://gerrit.wikimedia.org/r/#/c/276439/ afterwards [19:27:55] I added another comment there after upload, one part still looks puzzling and potentially memory-leaky [19:50:36] ema: amended your patch to add the bereq_req bit for X-Pass-Stream