[00:09:10] 10Traffic, 10Operations, 10Patch-For-Review: Configuration for Asia Cache DC hosts - https://phabricator.wikimedia.org/T156027#3954594 (10BBlack) Copying from earlier commitlog commentary, known list of TODOs here (minus what's already been done since above): ``` * hieradata/common/cache/*.yaml: eqsin node l... [00:52:32] 10Traffic, 10Operations, 10Page-Previews, 10RESTBase, and 2 others: Cached page previews not shown when refreshed - https://phabricator.wikimedia.org/T184534#3954646 (10Pchelolo) I've done a little bit more research here and Varnish docs actually confirm that `age` header can effectively disallow the clien... [01:16:33] 10Traffic, 10Operations, 10Page-Previews, 10RESTBase, and 2 others: Cached page previews not shown when refreshed - https://phabricator.wikimedia.org/T184534#3954704 (10BBlack) I don't know if that sounds like quite the right answer, I think this needs more thinking/info about what behaviors we're trying t... [09:57:54] 10Traffic, 10Maps-Sprint, 10Operations: Decide on Cache-Control headers for map tiles - https://phabricator.wikimedia.org/T186732#3955088 (10Gehel) I'm not sure that our varnish configuration honors `stale-while-revalidate` headers. A quick look through the code shows a [[ https://github.com/wikimedia/puppet... [11:07:39] bblack: nice, merged [11:08:09] the vtc test does not compile on v4 as access to obj.ttl in vcl_deliver has been added in v5 [11:11:48] but yeah whatever :) [11:18:10] looks like it's doing its thing already [11:18:17] - TTL RFC 300 10 -1 1518088342 1518088342 1518088341 1518088642 300 [11:18:20] [...] [11:18:28] - TTL VCL 300 10 300 1518088342 [11:18:28] - TTL VCL 300 3600 300 1518088342 [11:18:28] - TTL VCL 295 3600 300 1518088342 [12:32:08] 10Traffic, 10Maps-Sprint, 10Operations: Decide on Cache-Control headers for map tiles - https://phabricator.wikimedia.org/T186732#3955341 (10BBlack) Yeah it doesn't honor `stale-while-invalidate` directly at this time. It does implement `stale-while-revalidate` -like behavior for all cache objects, but it's... [12:55:11] 10Traffic, 10Operations: Migrate to nginx-light - https://phabricator.wikimedia.org/T164456#3955364 (10MoritzMuehlenhoff) > First is upgrade tlsproxy hosts to `1.13.6-2+wmf1` (but still on existing `nginx-full` packages) I've upgraded all of mw* to 1.13.6-2+wmf1~jessie1 , this leaves only conf* to be upgraded... [14:51:06] hello everybody, after ipsec between cp and jumbo we'd like to proceed with adding the vk testing tls instance to all misc (https://gerrit.wikimedia.org/r/#/c/409027/) [14:59:46] 10netops, 10Analytics-Kanban, 10Operations, 10monitoring, and 2 others: Pull netflow data in realtime from Kafka via Tranquillity/Spark - https://phabricator.wikimedia.org/T181036#3955768 (10elukey) Finally we have something working! Example from stat1004 ``` elukey@stat1004:~$ hive [.. som output ..] h... [15:05:48] upload-codfw upgrade started, I'm using wmf-upgrade-varnish on neodymium (it works!) [15:06:16] CR here https://gerrit.wikimedia.org/r/c/409047/ [15:06:46] ema: do you need a review before actually using it? [15:06:50] cool! [15:07:45] volans: no, I'm just going commando running it from my $HOME [15:07:53] I know you're upset now [15:08:27] but of course a review would be very much appreciated! [15:09:45] https://i.imgflip.com/1jhlof.jpg [15:10:02] I've tested both a downgrade and an upgrade on pinkunicorn before using the script on codfw [15:10:16] volans: haha :) [15:12:39] ema: I trust you ;) I'll review as soon as you fix flake8 failures :-P [15:12:51] volans: they're fixed already! [15:13:25] ah sorry, was in a meeting, missed PS2 [15:13:49] crucially, there are now two newlines between functions [15:14:12] you never know what could have happened otherwise [15:15:33] lol [16:12:10] wow, A:eqsin in cumin already return bast5001, nice! [16:13:25] :) [16:36:55] upload @ codfw 70% done [16:37:26] ema: code review @ 70% done too :-P got slow down by the meeting ;) [16:38:00] give me a +0.7 then [16:59:21] bblack: o/ - whenever you have - do you think that we can proceed with https://gerrit.wikimedia.org/r/#/c/409027/ ? [17:02:10] elukey: can you paste a compiler link on a misc host? [17:02:26] err wait, I see it on the change [17:03:22] lgtm [17:03:48] super thanks! [17:04:12] the plan is to leave it running a couple of days, observe metrics and catch issues (if any) [17:04:28] and then think about the overall migration of webrequest/eventlogging/statsv to jumbo [17:51:59] the new vk instance seems to be working fine! [17:52:53] nice! [18:10:16] I also verified that without any Kafka ACL vk wasn't able to push to the topic [18:10:37] then I added [18:10:38] kafka acls --add --allow-principal User:CN=varnishkafka --producer --topic webrequest_misc_test [18:10:47] kafka acls --add --deny-principal User:ANONYMOUS --operation Write --topic webrequest_misc_test [18:11:31] (atm ANONYMOUS has perms to read from every topic) [18:12:13] so it seems that Kafka is also doing TLS auth, grabbing the principal from the client cert and then match it with its acls [18:12:30] everything expected but nice to see working finally [18:16:56] upload @ codfw upgraded \o/ [18:18:26] yay [19:48:54] 10Traffic, 10Operations, 10TemplateStyles, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3956465 (10Tgr) [20:02:51] 10Traffic, 10Operations, 10ops-ulsfo: replace ulsfo aging servers - https://phabricator.wikimedia.org/T164327#3956541 (10BBlack) [20:02:51] 10Traffic, 10Operations, 10ops-ulsfo: setup/deploy dns400[12]/wmf721[56] - https://phabricator.wikimedia.org/T179204#3956539 (10BBlack) 05Open>03Resolved These are live in service for local NTP+DNS now. [22:20:44] 10Traffic, 10netops, 10Operations, 10ops-ulsfo: replace ulsfo access switches - https://phabricator.wikimedia.org/T185228#3956940 (10RobH) 05Open>03Resolved