[12:04:05] hi [12:19:09] why mutante , long time no see [12:19:14] mutante: how are you ? :) [12:20:02] heh, hello [12:20:26] https://stats.wikimedia.org/ (note the protocol) [12:20:46] i'm working on it [12:20:55] very interesting [12:20:57] to enable SSL on stats and metrics [12:21:05] not the content, that is you guys :) [12:21:13] ok [12:21:33] if you want to chip into our daily meeting (see the batcave link in /topic) you are welcome to, if you want to share your progress with us [12:22:25] mutante: this is part of the other things that went https lately right ? [12:22:37] oh, thanks, i still need to merge things , i eh.. live fixed [12:22:39] because I know we had wikimetrics go https, and wikipedia in general went https right ? [12:23:03] well, not really related since it doesn't run on the cluster [12:23:15] but kind of , because it needs puppet changes for the Apache setup [12:23:22] ok [12:23:31] ottomata: hi [12:23:56] mutante: ottomata is part of the ops team, and so he is a good person to talk about this if you want to :) [12:24:14] heh, i'm in ops as well:) [12:24:31] thanks! [12:24:38] ottomata: review my stuff?:) [12:25:00] i didn't want to run into your meeting though, sry [12:25:37] i'll comment on gerrit and RT/BZ when it's done, bbl [12:45:48] hey milimetric [12:46:05] morning average [12:46:18] hiiiiiiii everyone! [12:47:33] hi YuviPanda|train [12:47:39] hi average [12:47:40] hi halfak [12:51:11] it's still early for halfak [12:51:27] he's on Mountain Time 'cause he's from da mountains :) [12:51:40] Hey. No, I'm from central time. :P [12:51:47] oh rly? [12:51:52] duh, of course [12:52:08] We don't really have mountains in MN. More like rolling hills. We specialize in interesting weather. [12:52:15] and lakes! [12:52:17] :) [12:52:19] 11,000 lakes [12:52:22] Oh yeah. Those too. [12:52:34] Michigan has 10,000 and we've always been jealous [12:52:38] sucks coming in second [12:53:04] (PS1) QChris: Remove script that tried to compute still needed files [analytics/global-dev/dashboard-data] - https://gerrit.wikimedia.org/r/85975 [12:53:08] hey halfak, at some point I'm going to be talking to Oliver about querying data through Hive [12:53:16] if you're interested I can loop you in [12:53:35] That sounds like a good idea. Were you guys planning to do that today? [12:53:45] no plans yet [12:56:36] OK. Anything I should look over in the meantime? [12:57:07] (PS1) QChris: Remove web hook [analytics/global-dev/dashboard-data] - https://gerrit.wikimedia.org/r/85977 [12:57:18] morning qchris [12:57:27] Morning drdee [12:57:49] milimetric: did anything come out of the datawarehousing discussion we had? [12:58:02] definitely YuviPanda|train [12:58:13] ooo, details? [12:58:23] hey average, around as well? [12:58:27] we've basically decided to use data warehousing as we build up our Hadoop cluster [12:58:47] so right now we're getting our first requirements from I believe Oliver [12:58:59] I see [12:59:12] so we've been working in "vertical slices" and we like it [12:59:26] that means Oliver asks a question and we build only what we need to answer it [12:59:33] i still prefer the term 'data tent' more than datawarehouse [12:59:39] hahaha [12:59:50] surely you mean 'datalandfill' [12:59:51] :P [12:59:57] terms are loaded, indeed [13:00:29] I don't really care much for technical terms. It's accessible dimensional data, that's all I care about [13:00:49] nth-dimensional data [13:00:52] the new normal [13:02:14] batcave anyone? [13:03:19] * YuviPanda|train offers 'Kraken' [13:04:13] ? [13:04:45] milimetric: ^^ [13:05:16] batcave, sure, but where in the world is the batcave, I've got 3 different urls for it now [13:05:56] drdee: no, just saying we should call it Kraken [13:05:58] nice animal [13:06:24] milimetric: use the google short url from the irc topic [13:06:38] YuviPanda|train: it has a ring to it, we should consider! [13:06:56] hi drdee :) [13:07:01] morning average [13:07:46] milimetric: I am formatting an analysis to Dario's e-mail (in short, our query is good, so in my oppinion our metric is good, but I want to analyse it and prove it very thoroughly) [13:07:59] yeah, that's good average. [13:08:20] Dario knows what he's doing so it's important to find out where the difference is [13:08:32] the key is in the fact that rev_timestamp is not a date [13:08:37] so you can't do date_add on it [13:08:39] that's basically it [13:08:52] and neither is user_registration [13:09:01] so you can't do date_add on that either [13:09:17] unless you convert them to dates. but if you don't, it's not sound to do date_add on them if they're not dates [13:09:57] I am formatting an extended analysis with references to mysql documentation and thorough explanations [13:10:01] I'll be done in about 10-15m [13:11:49] drdee , milimetric I can run my analysis by you guys first, before I send it [13:13:29] sure, jump in the hangout [13:13:51] yup [13:14:34] but average, I was doing date_add with those columns yesterday, I'd be curious to see what's going on - it seems to be somewhat working [13:15:43] average: ^^ [13:16:31] yes, date_add works, but >= and < don't work because rev_timestamp is a string, while the return value of date_add is a date [13:17:08] I am finishing up formatting the e-mail and I'll jump in the hangout [13:19:25] I gotcha [13:19:32] well, average, all you really need to prove that is here: [13:19:33] select user_registration, date_add(user_registration, interval 1 hour) from user where user_id = @user_id; [13:19:37] oops [13:19:52] select user_registration, date_add(user_registration, interval 1 hour), user_registration < date_add(user_registration, interval 1 hour) from user where user_id = @user_id; [13:20:25] average ^ that returns 0 as the last column, which is obviously false [13:25:34] milimetric: e-mail sent [13:25:37] drdee: e-mail sent [13:25:41] we can discuss it in the batcave [13:41:28] hey aaverage, what change did mutante want reviewed? [13:41:30] averaget^ [13:41:35] average^ [13:42:08] ottomata: I poked him, hopefuly he'll get back with the gerrit link [13:43:54] i found them average [13:43:59] i think [13:44:00] https://gerrit.wikimedia.org/r/#/c/85971/ [13:44:05] https://gerrit.wikimedia.org/r/#/c/85971/ [13:45:29] ottomata: correct:) [13:45:40] and .. https://gerrit.wikimedia.org/r/#/c/85973/ [13:45:47] oh bad paste [13:45:48] ottomata: cough.. i cheated [13:45:51] that was supposed to be the second [13:46:13] hey so, q [13:46:19] since templates/apache/sites/stats.wikimedia.org [13:46:20] [13:46:21] is a template [13:46:28] there is a ton of repeat stuff in there [13:46:36] could we store in a ruby var and just print it out twice [13:46:39] that way it only appears once? [13:46:56] you can, but you're making it different from the other sites again right away:) [13:46:59] ? [13:47:21] check ./templates/apache/sites/ in puppet [13:47:34] all i wanted to do first was to have the Apache config in a single place [13:47:41] so we can enable SSL [13:47:42] oh yeah, no i mean still there [13:47:47] i'm saying in the template file [13:47:48] not in puppet [13:48:50] i guess, but isn't it an advantage that you can see config as it actually runs on the server without having to build it [13:49:31] of course we can make it nicer, i just took exactly what is on the server for the first change [13:49:36] like this [13:49:37] https://gist.github.com/ottomata/6699905 [13:50:23] hey mutante, welcome to the analytics channel [13:50:29] i see, but you missed another change in that [13:50:31] mutante: i think that is not an advantage really, one of the whole points of puppet and templating is DRY abstractions [13:50:35] like where it redirects to https instead of http [13:50:43] aahh ok, yeah so i'd only want to abstract the common stuff out [13:50:47] that was just a quick example [13:50:47] k [13:51:30] anyway, it doesn't matter to me tooooo much, we can merge this now as is [13:51:38] but i might make that change if someone asks me to make a modification later :p [13:51:48] the copy/paste offends my sensibilities :) [13:52:04] hey, enhance the template as you like, i just want to fix that it uses a template in the first place instead of the inline stuff and make SSL work [13:52:16] hehe, ok, i'll submit a patchset [13:52:58] and, it needs the other thing so that both sites work on SSL [13:54:24] apache2/ports.conf .. had to do that in several classes. it's this http://stackoverflow.com/questions/10658017/apache-error-default-virtualhost-overlap-on-port-443 [13:59:33] ok, i'm merging and watching it doesnt break things on puppet run [13:59:51] enhanced template needs to be on top of this in any case, right [14:00:18] ah..i see :) [14:00:44] ja how's that? [14:01:42] looks cool, i haven't used heredoc syntax in erb but i suppose it works) [14:02:07] ja its just ruby, so it should [14:02:15] oo, we should be able to use hashar's erb outputter to test! [14:02:35] ./expanderb.rb -f templates/apache/sites/stats.wikimedia.org.erb [14:02:53] hmm, tabbing is dumb [14:02:54] will fix [14:04:03] oh, good idea [14:06:28] ok there we go [14:06:31] mutante: i think that is good [14:09:32] thanks, cool, gimme 3 min, brb [14:16:21] ottoman, can you have a look at #291 [14:16:34] ottomata i mean and give us your best estimate to make that work? [14:16:54] where is your mingle bot! [14:16:55] ? [14:16:58] #291 [14:16:59] why why why? [14:17:23] i'm deploying that right now and watching puppet [14:17:31] k danke mutante [14:17:54] of course we still have a dependency error:) [14:17:59] but i'm looking [14:18:52] Could not find dependency Webserver::Apache::Module[ssl] for File[/etc/apache2/sites-available/metrics.wikimedia.org] [14:18:59] ehm.. but that is metrics. not stats ..wth [14:20:49] ha, hm [14:21:08] hey drdee,milimetric. does anyone use metrics.wikimedia.org anymore? [14:21:16] for old umapi, dario uses stat1, right? [14:21:21] and for new wikimetrics, its wmflabs? [14:22:05] i believe so, i would just sent an email to analytics and product mailinglists [14:22:12] and announce we wanna pull it down [14:22:14] 460 # ssl module already loaded by webserver.pp [14:22:18] and let's see how people respond [14:22:26] <-so i think we can just remove the dependency there [14:22:28] well, i mean, we'll need that url and site with ssl eventually anyway [14:22:52] mutante: probably. it is probably just a different way of loading the ssl resource [14:22:56] you could cahnge the dependancy to just [14:23:00] Class['webserver'] or whatever [14:23:01] if you want [14:23:23] that way the proper packages and config files are sure to be put in place before the metrics site [14:45:53] notice: /Stage[main]/Misc::Statistics::Sites::Stats/File[/etc/apache2/ports.conf]/content: content changed [14:45:57] notice: /Stage[main]/Misc::Statistics::Sites::Reportcard/Misc::Limn::Instance[reportcard]/Limn::Instance[reportcard]/Service[limn-reportcard]/ensure: ensure changed 'stopped' to 'running' [14:46:23] /Stage[main]/Misc::Statistics::Sites::Stats/File[/etc/apache2/sites-available/stats.wikimedia.org]/content .. etc. all good so far.. just slow run [14:47:14] fix was rather obvious on second look :p https://gerrit.wikimedia.org/r/#/c/85981/1/manifests/misc/statistics.pp [14:48:23] notice: Finished catalog run in 597.95 seconds [14:49:42] uh, oh.. and the next issue [14:49:54] Illegal override option Nonesta [14:54:58] Nonesta? [14:55:19] yea, how did that get there, do i have to understand that? [14:55:32] when i change to "None" in 2 places it's fine [14:55:46] AllowOverride Nonesta [14:56:20] that seems so unrelated but it prevented Apache restarts [14:56:37] and where did "sta" come from [14:57:27] or does this make other sense because Syntax:AllowOverride All|None|directive-type [directive-type] ... [15:10:13] drdee: summary: https://stats.wikimedia.org/ (no cert error), https://metrics.wikimedia.org/ (no cert error), -not using wildcard certs, -puppet run unbroken, -apache site is template [15:10:17] heh [15:11:18] and it's SNI / not wasting IP [15:11:43] besides, you'll know it, very old IE or something [15:22:25] ty mutant3e [15:28:38] yw:) i called the tickets resolved. tty [15:28:40] l [16:25:57] manually added a favicon.ico on stats.wm because hashar pointed out there was none [16:27:05] cute :) [16:34:40] neon still has not run puppet, its just sitting there [16:34:41] CMOOON [16:36:04] AGHH [16:36:05] err: Could not retrieve catalog from remote server: execution expired [16:36:10] oops [16:36:11] wrong chat [16:39:02] you just need it to remove your hosts from config right [16:39:07] you could ..l cheat(tm) [16:39:15] and remove them manually? [16:39:26] at least i'd do that for Nagios, ganglia less sure [16:47:10] yeah that's what I'm doing mutante [16:47:23] ganglia puppet runs fine on nickel [16:54:15] yep yep [17:01:02] ottomata: average: scrum [17:44:48] milimetric: Could I ask you for a demo on card 1091? [17:44:56] milimetric : batcave/ [17:44:59] ? [17:45:01] sure [17:45:16] sure to qchris AND drdee, to the batcave! [17:56:11] lunchtime, back in a bit [18:00:10] DarTar: if you have some time today can you let us know whether you are happy with the survival implementation? [18:01:24] yes, I replied to Stefan this morning, I'll confirm this afternoon if there's any other pending issue but I think we're good [18:16:45] qchris, around? [18:16:55] Yes. He is around. [18:17:37] he is an awesome dude [18:17:46] :-P [19:04:32] aaaaa installing ciscos takes all dayyyyyy [19:27:59] qchris: what's the difference between bugzilla 54612 and 54611 [19:28:37] geowiki-data vs. global-dev/dashboard-data. [19:28:40] They are different repos. [19:30:15] DarTar: I was reading the e-mail about survival sign off [19:34:02] but isn't the fix the same? [19:35:39] DarTar: https://meta.wikimedia.org/wiki/Research:Metrics/survival(t) [19:36:53] censored is not defined in the Metric definition [19:37:08] I am trying to read up old e-mails to find out what it actually is [19:37:44] milimetric: do you know what censored observations are ? I don't know [19:37:50] * average is an ignoramus [19:38:24] DarTar left just before I was going to ask him [19:38:37] hi average [19:38:43] yes, I didn't understand this either [19:38:52] you don't have to worry about censored stuff yet [19:39:08] basically it's the data that's scrubbed as we export from production -> labsdb [19:39:12] drdee: We only know that once we see the fix in place. [19:39:40] ok [19:39:48] what data do we scrub off ? [19:40:04] well, so one example is deleted pages I think [19:40:07] no [19:40:19] censored data in this case has nothing todo with private data [19:40:32] oh I don't have context then, sorry [19:40:42] * drdee read http://en.wikipedia.org/wiki/Censoring_%28statistics%29 [19:41:21] average: ^^ [19:41:32] "1) censorship: when you want to know whether a user completed 10 edits in 24 hours since registration but 24 hours haven't elapsed yet, you have a problem. This is what technically is called a "censored observation". We basically don't know whether the user will survive or not if we take a measurement after less than 24 hours and we should consider the survival value for that user as undefined. [19:41:43] " [19:41:52] -- Dario e-mail from 5th of September [19:42:06] drdee: reading your link now [19:49:11] milmietric: in am in the hangout [19:52:56] average: there needs to a column called censored in the output [19:53:03] 1 if the observation is censored [19:53:07] 0 if the observation is not censored [19:55:32] average: ^^ [20:01:27] average: sorry, I was afk [20:01:53] hey DarTar [20:02:26] what's up: survival? [20:02:39] yep [20:03:20] what should the censored field contain ? [20:03:50] the definition for it is not present in the metric definition or the mingle card 701 [20:03:56] https://meta.wikimedia.org/wiki/Research:Metrics/survival(t) <== metric definition [20:04:05] https://mingle.corp.wikimedia.org/projects/analytics/cards/701 <== card 701 [20:04:21] you're right, let me see if we captured it somewhere else [20:04:33] ok [20:04:59] there's this one (a legacy ticket from UserMetrics) https://mingle.corp.wikimedia.org/projects/analytics/cards/1045 [20:06:04] so the bottom line is that every boolean metric needs to allow for undefined values [20:06:22] if we don't know yet what the value for that user is at the time the report is generated [20:06:31] does that make sense? [20:07:20] "did user X reach 5 edits within 24hrs of registration?" yes / no / we don't know yet [20:08:09] if a boolean metric needs undefined values then it is not boolean any more [20:08:45] we enter the land of modal logic where we have 3-valued logic "yes/no/maybe" [20:09:26] but wait a moment, still reading those links [20:09:39] I'll get back in 5 minutes with some more questions about the censored field [20:10:07] no, it's not intuitionist logic ;) [20:10:34] we just have a set of users for which you can compute a boolean metric [20:10:52] and a number of users who should be excluded from the metric because you can't tell anything about them (yet) [20:11:13] and importantly, once the value is defined, it doesn't change [20:12:08] at least for metrics of the form "has user done X since registration + t" [20:16:00] for some context on censored observations -> https://en.wikipedia.org/wiki/Censoring_(statistics) [20:20:24] I'll take a guess. My understanding so far is that : if the timespan [T+t,today] < 24h , or the timespan [T+t,T+t+s] < 24h , then it is a censored observation [20:20:30] is this correct ? [20:21:08] in other words now - T+t < 24h or s < 24h (depending on the case we're in) [20:22:38] (if we're using sunset or not) [20:22:44] DarTar: is this correct ? [20:23:29] DarTar: let's go in the batcave and discuss a bit about it http://goo.gl/1pm5JI [20:23:36] I'm there [20:23:36] sure [21:39:43] (PS1) Milimetric: fixing google redirect uri [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86032 [21:39:52] (CR) Milimetric: [C: 2 V: 2] fixing google redirect uri [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86032 (owner: Milimetric) [21:55:23] (PS1) Milimetric: trying to deal with weird proxy issue [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86038 [21:55:33] (CR) Milimetric: [C: 2 V: 2] trying to deal with weird proxy issue [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86038 (owner: Milimetric) [22:05:40] hey milimetric - I talked to average a moment ago, now updating the survival card with info on censored data [22:48:28] well well well [22:48:52] 0 missing logs for my broker failure/leader change Snaps_, drdee, when producing from esams [22:49:12] going to turn on production from esams and from eqiad with 6 partitions and 2 replicas and just let it hum for a while :) [23:13:14] * less-than-averag knows he doesn't know