[13:33:25] morning, brb restarting [13:40:35] mooooorning [13:42:18] hey, back [13:44:56] hello world [13:46:05] hey average_drifter [13:49:16] hi drdee [13:49:19] hi milimetric [13:49:34] good morning sir! [13:50:56] grr, my chrome is now crashing a lot [13:50:57] yay [13:51:18] average_drifter: were you able to get before and after numbers for that run? [13:51:27] for the 1000 lines I mean? [13:52:35] ottomata: https://rt.wikimedia.org/Ticket/Display.html?id=4890 :D [13:52:48] if we are going to do it, then apparently we have to do it the right way (TM) [13:53:18] milimetric: no [13:53:43] I updated the codebase to be able to restrict to N lines (configurable) [13:53:55] I also added parameters in configuration to restrict to certain days in each month [13:53:58] also configurable [13:54:15] hm... that seems too nice :) [13:54:39] is the logic to compare url and referer titles done? [13:55:02] https://github.com/wikimedia/analytics-wikistats/blob/localdev/pageviews_reports/conf/local-rawdata.json#L9 [13:55:03] milimetric: yes [13:55:18] ok, cool [13:55:22] but the extrapolation after sampling is not done yet [13:55:29] and I actually have some unclear things about that [13:55:34] wow [13:55:37] well, let's forget about extrapolating [13:55:45] "but the extrapolation after sampling is not done yet" [13:55:52] we can do that manually [13:55:55] yep [13:55:56] x * 30 [13:56:06] so - what we wanna know is maybe this: [13:56:17] Count before change. Count after change [13:56:23] no, because if we sample the first 100_000 lines of a file there are many things that are inaccurate [13:56:38] I mean, there's a lot of factors that can make the extrapolation spiral out of control [13:56:40] forget that :) [13:57:11] we have a count before change in r46 [13:57:13] the point was to create a technique by which we can quickly get a sanity check [13:57:21] we can get a count after the change in the future r47 [13:57:23] in ETA 7h [13:57:26] how does that sound ? [13:57:29] bad [13:57:33] we need a count in 30 seconds [13:57:43] so that's why we're sampling [13:57:43] a count in 30s would be extremely inaccurate [13:57:54] yes, it would be very inaccurate [13:58:05] i know, and it's my idea and I'll take full responsibility if it makes us do something stupid [13:58:32] the point is, both methods would go against the same exact set of lines [13:58:40] no they won't [13:58:44] r46 is on full data [13:59:14] well what i mean is, we have to checkout r46, apply the new logic that limits it to 2, 3 and 1000 [13:59:21] then run the count [13:59:25] bam 30 seconds we have an answer [13:59:26] yes, that would be possible [13:59:45] here's the problem - this is now critical [14:00:06] but I can already tell you that I ran locally, and the results were close to meaningless due to re-sampling [14:00:16] re-sampling? [14:00:24] well, are both counts going to run on the same set of lines? [14:00:31] yes, you suggested that when we talked yesterday [14:01:13] uhm, ok we'll make two new runs then, I have to backport some of my stuff into a previous version of the code [14:01:20] ?????? [14:01:24] yep. [14:01:34] here's the plan: [14:01:50] 1. backport new stuff except new logic [14:01:57] 2. run [14:02:01] 3. run new logic [14:02:07] drdee: r46 was the last full data run made. it contained the logic before you added new details(url ~~ referer logic) [14:02:19] drdee: milimetric suggested that we do re-sampling so we can run faster [14:02:25] let us know if you deviate from those steps at all, because we want to control detours together [14:02:31] drdee: but he also suggested that we compare the previous run with this one [14:02:47] drdee: so we can't compare r46 because it contains full data [14:02:56] drdee: it would be like comparing apples to pineapples [14:02:57] by re-sampling you mean because the logs are already sampled right? [14:03:21] milimetric: yes, that's what I mean [14:03:21] average_drifter: milimetric's plan sounds right, please sitck to that [14:03:30] ok, cool. plan above look right? [14:04:04] in theory yes [14:04:24] well, forgetting about what the results are and just in terms of steps that we have to do [14:04:39] does 30 minutes sound like a fair estimate? [14:04:43] no [14:04:48] how come [14:05:39] I don't know how to answer that question [14:05:47] well let's take it step by step [14:05:53] 1. backport new stuff except new logic [14:06:00] how long? 20 minutes? [14:06:02] I'll get coding and I'll inform about my progress in the mingle card and in IRC [14:06:15] well forget about mingle, we want to have low overhead [14:06:25] then IRC [14:06:43] the point of the plan is to take a *lot* less time than 6 hours [14:06:52] because otherwise we could just rerun the data [14:07:00] so if it fails to do that, it's a bad plan [14:07:07] so, how long would step 1. take? [14:07:21] then we can both rerun full-data on stat1 and follow the plan you described in parallel [14:07:35] so if I fail doing re-sampled in less than 6h, I still get the full run [14:07:50] how long does it take to run the full data on stat1? [14:07:56] i mean to kick off t he run [14:07:59] ~6-7h [14:08:05] no just to kick it off [14:08:15] ~5m [14:08:26] or less [14:08:50] k, that's fine [14:08:56] parallelism is good :) [14:09:39] at 17:13 your time let's work on an estimate for how long step 1. above would take [14:10:09] feeling micromanaged but ok [14:10:16] we will talk about an estimate [14:12:07] hm, i'm very sorry, the intention is not to micromanage you [14:12:16] average_drifter: this is not about micromanagement, it is about getting a clear and predictable path to finish card 60 [14:12:17] it's to micromanage the process by which we all get this done [14:12:31] the key thing: you did not fail [14:12:33] there is no you [14:12:38] *we* failed [14:12:51] milimetric is asking questions to clarify the thinking and the path and to make a plan to move ahead [14:13:10] ok [14:13:45] so yeah, this is not personal in any way. We all have different ways of working but we have to figure out how to iterate quickly because this is a bitch of a problem [14:14:22] btw, IMHO #60 is not a task, it's a project [14:14:34] please see my updates on the card in Deliverables section [14:14:42] i can tell that, I think we have to be better at communicating what #60 was [14:14:54] IMHO thos things in Deliverables section are very important [14:15:06] Like, looking over your shoulder for the last few days i realized this was not even close to "go in, change some logic, rerun" [14:15:19] you like dramatically cleaned up the code, structure, documentation, etc. [14:15:22] it looks amazing [14:15:35] I just think there's been a bit of a disconnect between what people expected and what you did [14:15:56] requirements changed for it like many many times [14:16:07] imagine this: we're all sitting around in a circle and the problem has just lifted off your shoulders and into the middle. Everyone is getting out maginfying lenses and knives and we're about to operate :) [14:16:45] every step of the way was documented so far in mingle, asana, the user page I have on mediawiki, and all the reports in on stat1.wikimedia.org/spetrea [14:16:45] I feel your pain, that's why i'm not joking when i say it's not your problem. It's been a fuck-up for all of us in terms of management, giving you support, etc. [14:16:57] so we're trying to correct our mistakes [14:19:59] k, i know we kept you busy with talking but when you're done kicking off the job let's estimate how long it'll take to backport the logic [14:20:15] and i'll clone this so I can help if possible [14:20:57] alright [14:24:13] drdee: http://stat1.wikimedia.org:4000/ is still up [14:24:14] bad [14:24:35] also, I am trying to file a ticket to make the prod instance https [14:24:39] what's the url for that again? [14:55:47] milimetric , drdee http://bit.ly/ZDOdUc [14:56:05] the data.json (one of the formats for outputting pageviews) [14:56:15] now has a key which describes what commit was used to produce that report [14:56:22] so we can diff and see what changed easily [14:56:37] suggestive directory names can only go so far as we've seen in the past [14:57:00] so I hope this will solve the problem (at least for reports that are not far apart, in terms of time) [15:02:07] ========================= [15:02:11] started stat1 run [15:02:32] given that there are no problems(which can be seen in stat1:/tmp/pageviews/map/*.err files) [15:02:41] cool [15:02:42] I'll be able to start the plan outlined by milimetric [15:02:51] well no [15:03:01] the plan is crap unless it doesn't take long [15:03:08] so let's estimate [15:03:18] 1.) in my view this is just git commands [15:03:25] am I missing anything [15:03:49] hopefuly it should be just git commands and moving code around [15:03:53] k [15:03:58] rough estimate? [15:04:17] 1h [15:04:30] rough [15:04:36] hm, do you think it'd be faster if I helped? [15:05:06] well, let me ask you this [15:05:19] what files is the code that lets you re-sample? [15:05:29] what files is it in I mean [15:06:05] copy those somewhere, git checkout [SHA1 for r46], copy those files back [15:07:45] let's hangout [15:07:48] anyone is welcome [15:07:50] drdee: wanna come ? [15:07:54] coming [15:54:21] ottomata: is https://rt.wikimedia.org/Ticket/Display.html?id=607 still relevant? [16:29:05] drdee, dschoon: the hangout is in use. I'm setting up another one. [16:30:17] added hangout to meeting invitation as a backup to regular scrum hangout...https://plus.google.com/hangouts/_/712d28c4de5982b42048e79cf03f41cca807d7eb [16:30:55] ah, dschoon isn't here... [16:31:45] kraigparkinson: on my way [16:32:18] dschoon: go to this hangout instead...https://plus.google.com/hangouts/_/712d28c4de5982b42048e79cf03f41cca807d7eb [16:32:19] kraigparkinson: drdee: what scrum are we in? [16:32:22] other one is in use. [16:58:17] am gonna be 2-3m late to standup [17:00:42] erosen: scrum? [17:01:34] something must be wrong with my link [17:01:42] I'm at https://plus.google.com/hangouts/_/053b64aba37418dd6ee597f1630d15223077ff7a [17:01:55] are you guys really using the one from the calendar event? [17:02:05] drdee [17:02:16] wrong link [17:02:20] https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [17:02:30] your link is wrong ;) [17:02:34] but actually [17:29:01] you know [17:29:07] we didn't wait for opposed voices [17:29:29] so our voice vote probably isn't a valid vote by acclaimation [17:29:34] BRB BAGEL [17:30:59] brb lunch [17:51:08] ====> http://i.imgur.com/JOUHvWV.jpg [17:52:27] nice [17:55:36] i return, triumphant, replete with bagel [17:55:48] apparently spite is a coin they accept [18:00:56] milimetric , drdee I'm in the regular scrum link [18:01:33] milimetric: would you like to join to discuss scp -r and the process of getting files from stat1 faster ? [18:01:45] this is also connected with not exposing sensible data at the same time [18:02:16] so maybe we find something in between [18:08:35] reminder: rsync is monotonically better than scp [18:12:46] started stat1 run on full data for localdev branch with new logic from the diagram [18:12:56] I'm unblocked and I have free cycles in the meantime [18:14:05] I'll detail 353 and 295 on mingle [18:22:37] ok [19:47:49] What do you despise? By this are you truly known. -- Dune [19:51:25] i shutter to think that might actually be pretty profound [19:51:51] (says the man who paid for a bagel with spite today) [19:56:59] frozen fries are not the best choice :| [19:57:16] some aren't at least.. [19:57:27] * average_drifter is cooking a small meal now [19:58:21] * average_drifter tries to negotiate some fries for bagels with dschoon [19:58:33] i'd totally do it [19:58:49] but i don't think you'd appreciate fedex'd bagels [21:01:34] erosen: can you add a hangout linky to the umapi meeting? [21:01:48] on it [21:01:56] sorry about that [21:04:17] milimetric: turns out ryan needs to reschedule, and I would actually prefer that tooo [21:04:26] ok [21:04:31] so unless you object, we were hoping to move it back to 330 or so, PDT [21:04:31] for later today? [21:04:35] doest that work for you? [21:04:46] sorry if that keeps you working late on FRiday [21:04:48] mmm, add a hangout and I'll do my best to be there [21:04:52] :) [21:04:53] cool [21:05:12] I don't think any particular hangout is strictly necessary, it is probably just total knowledge transfer time that counts [23:42:07] milimetric: around?