[03:19:00] 07HTTPS, 10Traffic, 06Operations: Secure connection failed when attempting to send POST request - https://phabricator.wikimedia.org/T134869#2311448 (10Thibaut120094) >>! In T134869#2281369, @BBlack wrote: > As a random experiment, perhaps some of those reporting could try this in FF 46.0.1? > > 1. Type 'abo... [06:35:12] 10Traffic, 10Beta-Cluster-Infrastructure, 06Labs, 06Operations: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2311660 (10Joe) @hashar that was exactly my plan [08:22:27] bblack, ema: it seems the varnish restarts for expat didn't happen yet? if needed I can deal with those [08:59:59] moritzm: nginx, I guess? [09:00:29] ema: oops, of course I mean nginx... [09:00:42] right, please go ahead! [09:06:48] ok, I'll start in approx. an hour [09:07:06] thanks [12:22:29] 07HTTPS, 10Traffic, 06Operations: Secure connection failed when attempting to send POST request - https://phabricator.wikimedia.org/T134869#2312346 (10Thibaut120094) OTRS members have the same issue with Firefox 46.0.1 on https://ticket.wikimedia.org/ https://lists.wikimedia.org/mailman/private/otrs-fr/2016... [12:37:04] 07HTTPS, 10Traffic, 06Operations, 07Browser-Support-Firefox: Secure connection failed when attempting to send POST request - https://phabricator.wikimedia.org/T134869#2312371 (10Danny_B) Adding #browser-support-firefox as it seems to be its issue only ATM (no other browser verified yet - in such case, remo... [12:51:08] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312406 (10phuedx) @BBlack: If Varnish is the part of the stack that this is to be done, have you taken a look at [libvmod-abtest](https://github.com/Destination/libvmod-abtest)... [13:56:44] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312587 (10BBlack) >>! In T135762#2312406, @phuedx wrote: > @BBlack: If Varnish is the part of the stack that this is to be done, have you taken a look at [libvmod-abtest](https... [15:19:36] elukey: any idea about https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=8&hoststatustypes=3&serviceprops=2097162&nostatusheader ? [15:19:45] (lots of vk drerrs/minute) [15:22:19] I guess it was temporary, they all vanished again [15:23:00] bblack: yep I was about to say that I couldn't see anything [15:23:29] there was a page full of cpNNNN hosts in various DCs all showing the drerrs/minue "unknown" status in icinga [15:23:57] I've never really understood that alert anyways [15:24:56] it is not that clear to me too :) [15:26:05] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312919 (10Jdlrobson) This all sounds great and I love that its generic and can be reused again! A few clarifications - if I'm understanding correctly experiments would be conf... [15:33:04] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312939 (10dr0ptp4kt) Quick question: how does this guarantee bucketing across browser restart? [15:33:24] bblack: depending on the message from the alert, it could also be errors talking to graphite, I've seen it happening when I'm bouncing graphite-web for example [15:34:21] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312967 (10Nuria) Non session cookies are kept after browser restarts, with an expiration set of 30 days (like last access cookie) the cookie is available. [15:35:09] iirc drerr would be delivery errors when vk can't ship logs to kafka, not sure what vk does when that's too many, if it spools some locally and retry + drop later if overflowing [15:35:41] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312971 (10dr0ptp4kt) I should note in the concrete cases: persistence across browser restart is probably not as important for lazy loaded images, whereas persistence across bro... [15:36:42] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312972 (10dr0ptp4kt) @nuria, should we add that to the Description as acceptance criteria? [15:39:11] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312984 (10BBlack) >>! In T135762#2312919, @Jdlrobson wrote: > A few clarifications - if I'm understanding correctly experiments would be configured in puppet? That would be th... [15:44:20] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2312986 (10BBlack) >>! In T135762#2312939, @dr0ptp4kt wrote: > Quick question: how does this guarantee bucketing across browser restart? >>! In T135762#2312967, @Nuria wrote: >... [15:51:51] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2313004 (10Nuria) >Since the binning is done independently of actual experiments (the binning is live all the time for all cookie-enabled agents), this actually is a problem, I... [16:12:59] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2313016 (10BBlack) Right, but if the user deletes cookies or goes incognito, that's probably a rare event for most, and possibly associated with not re-using browser cache acros... [16:15:42] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2313020 (10Krinkle) I think 1-100 might be a bit small. Especially considering our scale and considering most of our experiments will not have been load tested very much. For i... [16:27:27] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2313080 (10BBlack) >>! In T135762#2313020, @Krinkle wrote: > [...] > An experiment could start at 1 bucket (0.01%) and work its way up to 10 (0.1%). And if the experiment no lon... [16:31:16] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2313083 (10Nuria) >For comparison, our entire Navigation Timing data used to be based on 0.01% sampling. It is now tuned up to 0.1% (1:1000 sample; >$wgNavigationTimingSamplingF... [16:38:30] 10Traffic, 10Analytics, 06Operations, 06Performance-Team: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2313093 (10BBlack) >>! In T135762#2313083, @Nuria wrote:. > I know you know this but just clarifying that we do not have these restrictions here though, the restrictions come f... [22:23:49] 10Traffic, 10domains, 06Project-Admins: create a project for tasks related to WMF domain names - https://phabricator.wikimedia.org/T87465#2314326 (10Danny_B) [22:25:23] 07HTTPS, 10Traffic, 06Operations, 05MW-1.27-release-notes, 13Patch-For-Review: Insecure POST traffic - https://phabricator.wikimedia.org/T105794#2314347 (10bd808) @Steinsplitter reported to me on irc that > for protocol relative urls in mwclient, scheme='https' must be set in the config to enable https o...