[00:36:01] Hurmmmm. [00:36:12] I Have a serious dilemma. [00:37:15] Either I find somebody to deliver oven-roasted brussel sprouts to the office, or I leave now and do it myself at home. [00:37:42] * K4-713 squints [04:40:13] * jeremyb got email notif for 2014100510006271 but now it's in the wrong queue so I can't see it :( [04:40:18] * jeremyb is just curious :) [15:40:51] (CR) Awight: [C: 2] "As discussed in IRC, we're hand-waiving a thorough review of the contrib modules. This will have to be revisited if we determine CRM code" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/165017 (owner: Ejegg) [15:45:54] (PS6) Awight: Start a new branch for Drupal and embedding [extensions/DonationInterface] (drupal) - https://gerrit.wikimedia.org/r/165126 [15:54:54] hey awight :) [15:55:02] atgomez_: morneve! [15:55:04] awight I'm hacking on fundraising_code_deploy and friends today, did you happen to figure out where the reversed diff was? [15:55:13] Jeff_Green: oops, no lemme take a look [15:55:38] seems like it should be easy but i seem to have a mental block re git diffs [15:55:53] That probably prevents disease [15:56:11] ha, maybe cvs damaged those synapses or something [15:56:36] Jeff_Green: haha. I used rcs for a few seconds, at SETI@home. They seemed happy with it. [15:56:44] nice [15:57:02] it's basically the same as "chmod 700" on a file when you want to edit it. [15:57:15] Jeff_Green: remind me where the magic .pm lives? [15:57:35] /usr/lib/perl5/FrDeploy.pm [15:57:40] kthx [15:58:12] oic. yeah, it's safest to give diff two arguments to avoid confuzion [15:58:42] ya, we seemed to have done that in one of 4 places [15:58:55] should we change them all? [16:00:48] yeah but then I would have to understand what is happening on a deeper level :p [16:00:59] that was sort of my reaction [16:02:50] Jeff_Green: weird, the revision lock mode doesn't have a random-character prompt [16:03:17] hrm and the diff is different in that mode [16:04:06] where it goes into detached head? [16:04:24] hakking... [16:06:51] Jeff_Green: I don't understand this line, } elsif (defined $p->{'laststate'}->{'revision-lock'}) { [16:08:24] this is just a fugly hash reference, for state info that's stored between runs in the state file: $p->{'laststate'}->{'revision-lock'} [16:09:04] iirc the value is the commit id [16:09:30] or if you set to tracking HEAD that value is unset [16:09:40] does that make sense? [16:09:42] Jeff_Green: this is FrDeploy.pm line 139 [16:09:52] I'm trying to understand why it's in that conditional [16:10:00] p->revision-lock I get [16:10:06] ha i have to find an unmunged copy. sec [16:10:31] but why would this condition preclude the mode==update section? [16:10:47] so... we *used* to be on a detached head. we still want to update. [16:10:52] I mean, it works. I just don't get it. [16:11:29] oh. I think what happens is if you specify "i want a rev lock to this commit ID {BLARG}" you get to this test and run the chunk of git code between 132 and 138 [16:11:58] one sec, I haven't looked at this in a long time [16:11:59] yes that one looks right [16:12:07] but I don't see what laststat->revlock is abt [16:12:10] ok [16:12:17] i led you astray [16:12:29] laststate is what was stored in the state file after the last run [16:12:37] Please take a look at ~/FrDeploy.pm, I probably butchered it, but the diffs are more uniform now. [16:13:09] Jeff_Green: that makes sense. But why do we care if the last run was a revlock? and why would we not run the "update" code if so? [16:13:25] Seems like we can remove that conditional [16:13:40] looking [16:14:20] atgomez_: donno if you got my letter about the analytics fun [16:14:37] I'd like to at least plan rewriting the impressions snarfer [16:14:41] awight: i guess it just tries to clobber whatever is on disk with what's in the repo [16:14:45] & am not busy with recurring anything [16:15:15] I'm not sure it has any value, other than that it would complain if something has changed on disk [16:15:16] Jeff_Green: hrmph. So how is it ok to skip mode=update [16:15:45] update deploys HEAD [16:16:00] oh oh derp. thanks I see now [16:16:18] the first two blocks could probably be collapsed into one [16:16:33] if it was revlocked, yeah don't do anything else. [16:16:52] although...I guess the second block doesn't bother talking to the master [16:17:18] it just re-unpacks the cached copy [16:17:20] yeah that works [16:17:41] it skips the submodules though, which seems like fail [16:18:03] Well, it's probably just not doing anything at all :) [16:18:03] or does it? I don't know what git does there [16:18:17] we can test and see I guess [16:19:58] hah good luck grokking the manpage, I learned nothing [16:20:04] yeah empiricism FTW [16:21:20] yeah, git documentation is...uh....painful at best [16:31:18] atgomez_: hallooo [16:31:24] hey... on a call [16:31:27] did see... few minutes [16:31:30] oh k [17:08:28] awight: are you getting any 'undefined variable apiRequest' while running crm unit tests? [17:08:43] grr [17:08:54] that's a bug in a Civi catch statement [17:09:00] there's an underlying error that's being hidden [17:09:08] so yeah I've seen that [17:09:10] also, awight, atgo, anything you want me to bring up at the scrum of scrums? [17:09:40] ejegg: the idea is just for things that effect/work with other teams? [17:09:49] yep! [17:09:50] because... the MW +2 issue is the only thing i can think of [17:10:16] did you see that there were way fewer than we thought? [17:16:26] hey awight! [17:16:43] pizzzacat: mornight! [17:16:53] do you think you could spin off that script you wrote somewhere so I can get your chart into the FR report? [17:16:59] that is, [17:17:03] Is this about rebase :'-/ [17:17:04] hehe [17:17:09] there is none available for this year [17:17:13] ah sorry I did see that email, one moment, please [17:17:19] cool thanks :) [17:17:29] BTW I was tuning thresholds manually to get a nice graph [17:17:50] oh? [17:18:19] pizzzacat: eh, it's actually just the SQL here: http://www.mediawiki.org/wiki/File:Contribution_size_2012_-_pie.png [17:18:29] change @START and @END and it should be good to go. [17:21:45] hmm, doesn't fit so neatly into the /data/widget SQLgen. Might have to make an option for named placeholders outside of the generated WHERE clauses [17:22:30] oh no wait, can totally do that with multiple [[WHERE]] tokens [17:23:18] ejegg: I don't know about the sqlgen yet, but the placeholders are a stock ANSI SQL thing. [17:23:34] they should act just like string constants, here. [17:23:38] you mean the @SQL variables? [17:24:22] yah [17:24:49] oh, none of the stuff below matters? [17:25:05] oh because that's generating the pie [17:25:10] yah, those are mucho convenient for running by hand. But to feed input from the $filter qs into the query via the data route I'd want to make it act like the query template from the fraud widget [17:25:39] pizzzacat: yeah exactly [17:26:46] ejegg and awight, are we talking about the backend right now of Dash? [17:26:47] Though I will have to do something about the 'AND' bit of the WHERE clauses in case of no filter input [17:26:54] because I'm working on that [17:27:00] and changing strings and such [17:27:03] yeah, dash backend for this proposed pie widget [17:27:05] so lemme know if I need to not do that [17:27:13] pie widget? [17:27:33] you are trying to make those pie graphs into a dash widget, right? [17:27:49] atgo: I think we're going in circles with TY Matching Gifts. It's a production thing, they just need to add the paragraph again. [17:27:57] sigh. ok [17:28:12] oops, I gotta go to SoS. pizzzacat, let's pick this up right after that. [17:28:38] ejegg not that I'm aware of? [17:28:40] ejegg: I think pizzzacat wants the class warface pie for the report, not the dash (yet) [17:28:43] ok talk to you after that [17:28:58] yeah it's for the annual report. [17:29:11] atgo: my only suggestion was to use the URL directly and not do double-substitution. That will not have any effect on translation memory. [17:29:14] oh, gotcha! sorry, thought we were dash-ing [17:29:22] no worries :) [17:29:50] we totally [17:29:59] can make it a widget though! [17:33:06] yeah we can make it a widget fo sho [17:33:13] at some point [18:00:35] SoS going long, will jump into tech standup as soon as I can [18:53:16] awight: ejegg Where can I get info and maybe access to infrastructure needed to answer the questions from ewulczyn____ 's email? [18:53:36] hmm, let me take another look at the email [18:53:39] awight sorry if I'm being a pain about the query but do you think it's possible to get the correction before tomorrow? I sorta promised Megan I'd have the pie chart created by then so they can all look it over and come back with any corrections before releasing Friday [18:53:49] * AndyRussG is pestering as promised [18:55:16] ejegg: thanks!! basically I'd like to understand how data on banner impressions are pulled and I guess take a look at the scripts, maybe try out some queries... [18:55:36] also would be nice to learn, for my own edification :) [18:57:31] dunno if there's some doc on that somewhere [19:01:19] (CR) Ejegg: [C: 2] "Thanks for informative commit message! Was going to ask why we needed this." [extensions/DonationInterface] (drupal) - https://gerrit.wikimedia.org/r/165126 (owner: Awight) [19:06:35] hey awight [19:06:38] are you remote today? [19:12:22] AndyRussG: Sorry, I guess I'm not clear on the banner stats myself. I had some idea that Special:RecordImpression was actually adding things to a table, but that's totally not the case. The answer may lie in this tool: [19:12:26] https://git.wikimedia.org/summary/?r=wikimedia/fundraising/tools/DjangoBannerStats.git [19:12:32] yep [19:12:41] I have a lot of input on this topic ;) [19:13:17] AndyRussG: ejegg: don't worry too much about that repo, I'm planning to slash and burn within the month. [19:13:27] hmm, ok [19:13:36] But I'd like to chat with anyone who is interested about how to write the replacement. [19:13:46] did strike me as odd to use django for a comman-line job [19:13:56] just for the ORM, i guess? [19:14:11] Long story short, we have unsampled logs in Hive. The old job was pulling from Squid UDP logs, sampled 1:100 [19:14:26] Yeah that Django thing was originally supposed to be a dashboard, I hear. [19:14:54] ejegg: awight ah cool... What and where is Hive? [19:15:43] Hive is on stat1003, lemme look for docs... [19:16:00] ahh, dashboard would make more sense [19:16:46] stat1003 is a machine on the cluster I guess... [19:17:00] * AndyRussG loves rewrites heh heh [19:20:37] ejegg: AndyRussG: We're doing the SQL brownbag at the moment, but the big question I want to pose to you all is, whether we should export from Hive as the raw entries, or in aggregate form. And if we should store that intermediate thing for auditing, or just dump it straight into MySQL. [19:22:05] awight: thanks! OK I don't know much about all that but I'd love to learn, happy to give unlearned innocent opinions anytime... [19:22:18] let me commune with the Hive mind/docs, will let you know if i have useful input [19:23:15] ejegg: thanks much! Yeah if u have a link to docs that'd also be great :) [19:23:30] back in a short little while (I'll get backscroll tho) thanks again :) [19:26:20] AndyRussG|school: I believe it's this apache project: https://hive.apache.org/ [19:26:30] I'm leaning towards doing the computation in Hive. [19:26:47] awight: definitely sounds more fun that way! [19:27:12] lol +1 [19:55:51] awight: ejegg ah thanks [19:56:00] Mmm OK a data warehouse [19:56:20] It's where the varnish logs are sent [19:56:22] configuration? setup? [19:56:30] OK [19:56:37] I have heard varnish a few times before... [19:58:05] http://www.mediawiki.org/wiki/Manual:Varnish_caching [20:04:04] the reasons for not showing logs won't be too detailed, from what I can see in bannerController.js [20:04:54] it's either 'empty' (nothing came back from Special:BannerRandom) [20:05:07] 'preload': the mixin decided not to show it [20:05:33] 'cookie': there was either a hide cookie or a close cookie for that category [20:05:57] or no value at all, which happens when the 'alterImpressionData' callback nixes the banner [20:06:30] D'oh. I meant *not showing banners* 5 lines back [20:07:18] We should probably record different reasons for close cookies and donate cookies, now that we can tell the difference [20:29:36] yeah I think we should keep all URL parameters in their own column [20:29:41] no reason to destroy data at this point... [20:33:47] awight: ejegg: Ah OK I see [20:34:02] So the issue isn't with the infrastructure for pulling info from the logs [20:34:11] It's with the way banner impressions are recorded [20:34:24] awight: what did you mean about destroying data? [20:35:16] In DjangoBannerStats I see the bannerimpression_raw table [20:38:31] AndyRussG: there are a few issues in the air. One is that we need to be able to get "hide" numbers, which the current django thing is not doing. [20:38:52] The bigger issue though is that the django thing, which is the infrastructure for pulling info from logs into mysql, needs to die quickly, soon. [20:39:11] Did we used to use squid but now use varnish? Ist that why the tables in DjangoBannerStats talk about squid? [20:39:15] And while we're doing that, we should switch data sources to use the unsampled hive numbers [20:39:18] yes [20:39:22] OK [20:39:39] whew! figured one thing out correctly \o/ [20:39:42] I looked at the code with ewulczyn____ yesterday, and we realized that none of the bannerimpressions_raw code is running. [20:39:54] so, there was some hide code (which I do not trust), but it isn't in use. [20:40:34] awight: did you get chance to look at how I was looking for hidden impressions in the csvs? [20:40:37] to enable that, we'd have to also switch to the other aggregation job in that code base, which has outstanding bugs [20:40:44] ewulczyn____: not yet! [20:40:53] pizzzacat: that is insane. [20:41:04] pizzzacat: the count is good, at least :( [20:41:36] pizzzacat: oh I think I see. Where is the > $1,000,000 row? [20:42:29] ewulczyn____: emailed, I think I see the issue. [20:42:45] awight: I'm just trying to get a picture of the whole infrastructure chain in my brain, I think I'm correctly imagining bits but I fear it's not all there... Would you have 10 minutes or so to walk me thru it sometime today maybe? thx in advance... [20:42:51] * awight plays little engine soundtrack [20:42:57] (I think I can) [20:43:11] AndyRussG: yeah now is good if you'd like [20:43:15] jitsi? [20:43:26] ah yes one sec :) [20:44:28] ejegg: you want in? ^^ [20:45:08] sure, one sec! [20:46:48] ok, got a link? [20:46:49] awight yeah it always seems to be a little off [20:47:05] because of some gotchas that someone with more history/background might know? [20:47:15] pizzzacat: no I had the answer above :) [20:47:21] we're missing the last query [20:47:27] a few > $1M donations [20:47:31] ohh [20:47:40] I don't know how to read :-/ [20:47:49] ejegg: I think AndyRussG has to create the link cos he's the one with an account [20:48:17] awight: ejegg I was just gonna to the anonymous chrome way, just trying to turn on my silly webcam [20:48:19] so, are those in civi? [20:48:22] awight? [20:48:29] pizzzacat: yes [20:48:36] ok [20:49:03] pizzzacat: the issue was that this query is missing from your data: [20:49:06] SELECT -1, SUM(total_amount), AVG(total_amount), COUNT(total_amount) FROM civicrm_contribution WHERE (receive_date BETWEEN @START AND @END) AND total_amount > 1000000; [20:49:16] I think gmail snipped that off as if it were my signature :) [20:49:18] https://meet.jit.si/HearIdeaFootball [20:49:45] awww no FF, so much for the FLOSS solution :p [20:50:15] chromium seems to do it [20:50:19] awight: ejegg anyone else interested in CN banner impressions (ewulczyn____?) [20:50:20] ^ [20:50:33] awight thanks! [20:51:37] AndyRussG: do I have a picture or voice? I hear silence and blackness. [20:52:06] awight: You appear to be silence and blackness on this end too [20:52:20] Maybe if I kick you out? [20:52:24] (jk) [20:53:56] awight it's still off by about $300K but much closer on both numbers. [20:54:24] pizzzacat: $300k too high, right? that's just refunds. [20:54:46] AndyRussG: ejegg: well, how about https://plus.google.com/hangouts/_/grcboifngm34o2dqiy63lf5irqa for now [20:54:52] I liked jitsi when it worked. [20:55:10] this is probably something chrome does to prevent competition ;) [20:55:18] K going there [20:55:48] (PS1) Ejegg: Give RecordImpression different reasons for different cases [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/165607 [20:57:14] awight do you have a way to filter out refunds? [21:02:25] (PS2) Ejegg: Give RecordImpression different reasons for different cases [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/165607 [21:02:41] buh [21:02:57] How's... the things? Going. [21:05:51] K4-713 the things? [21:06:00] are you speaking to the room? :) [21:06:01] pizzzacat: Yes! Those. [21:06:07] How are they? [21:06:18] the things are great [21:06:22] Yaaaaaaaaaaaay. [21:06:33] how are the things of yours? [21:06:41] I've just been talking to people for five hours straight. [21:06:49] whoa [21:06:51] * K4-713 blinks exactly six times [21:07:09] I think this is my life now. [21:07:12] I didn't know that was humanly possible to do! *laughs* *nervously adjusts batcape* [21:07:27] I need lozenges. [21:07:29] welcome to management? [21:07:40] haaar. [21:07:46] I brought bananas today. [21:08:01] I saw at standup [21:08:01] ...said something about bananagement this morning. [21:08:06] bahahahaha [21:08:28] I wish I was there to put a dollar in the Pundraising jar [21:09:00] If I get a banana costume and a tablet with a calendar on it, I can be a bananager for halloween. [21:09:38] I feel like that's doable haha [21:09:45] ...but seriously though, the Things are good? [21:10:31] How is Berlin? [21:11:32] it's good. I spend most of it working so far but I go out in the middle of the day and explore [21:11:56] I'm next to a really beautiful canal that goes on for miles and has all kinds of serene shit by it [21:12:05] eeehehe [21:12:25] drink all the coffee, stare at all the stuff [21:12:48] I could use some serene shit, too. [21:13:01] oh you'd love it here.. it's like Seattle but not cold [21:13:04] (yet) [21:13:07] ...woah. [21:13:10] yeah. [21:13:25] and there's this thing in the sky like half the time - the "sun" [21:13:37] I don't know how you translate it to English [21:13:42] but it's pretty [21:13:52] According to Ancestry.com, about a quarter of me is from all over there. [21:14:02] * K4-713 shrugs [21:14:02] orly [21:14:14] like zero of me is from here :( [21:14:44] I am the direct descendant of a surprising number of Heinrich Horns. [21:14:52] ...which is a RIDICULOUS NAME. [21:15:05] I love it. [21:15:29] But, yeah. They did that to, like, every other kid for a while. [21:16:24] great name [21:16:59] if you get another guinea pig, you should definitely use that [21:17:07] I was just thinking that exact thing. [21:17:40] Heinrich Horn the... 14th or something. [21:17:49] haha [21:18:00] sounds so distinguished. [21:18:17] ...for a ball of fur. Yes. [21:18:34] Possibly not a departure. [21:20:16] and on that note, I'm out for the evening! or, day! afternoon? day..! [21:20:26] Enjoy! [21:20:40] enjoy your bananas! [21:23:13] awight: I think something you did to the unit testing made the groups go funny. [21:23:29] ...not sure anybody is using those right now, other than me. [21:24:35] DonationInterface group works just fine. WorldPay group can't find the right gateway class. [21:27:22] ejegg: awight thanks!! [21:28:15] K4-713: rrrgh. Yeah groups have never worked that well for me. [21:28:19] Aw. [21:28:32] I used to use those things a lot. [21:29:49] damage bananagement [21:30:02] bananaging expectorations [21:30:28] Don't forget risk bananagement. [21:30:38] OK i'll look at what I broke in groups. Can you paste the error from --group WorldPay ? [21:30:45] sounds like autoload fail. [21:30:53] ...bananage up. [21:31:04] This shouldn't be this funny. [21:31:08] slippery business [21:31:15] it's a slippery uphill slope. [21:31:27] the best part is the downhill WHEE [21:32:05] Did you ever see Transylvania 6-5000? [21:32:10] No [21:32:19] There's this amazing banana peel scene. [21:32:33] that does not have the stars. [21:33:03] No. But they're foold. [21:33:07] Er. Fools. [21:33:27] Perfect bad movie night submission. [21:33:37] can't wait :p [21:34:02] Particularly if we start this month. [22:42:59] K4-713: something pretty hilarious on my screen... [22:43:10] Looks like civi. [22:43:17] ...that's not funny. [22:43:32] funny like I'm some kinda clown, here to amuse you? [22:43:38] * K4-713 looks closer [22:43:41] whaddyu MEAN, funny? [22:45:47] a donation from Hugh Jass? I.P. Freely? [22:56:09] hey awight [22:57:48] atgo-away: bonsoir [22:57:54] what's this CN thing you sent? [22:57:57] hehe @goaway [22:58:07] whaat [22:58:10] where? [22:58:17] I'm sending CN things all day, it seems. [22:58:26] oh the meeting an hour ago? [22:59:33] ejegg: there was some amuzement in the Civi IRC channel, is all [22:59:58] you're onto me :P [22:59:59] apparently "fudmergency" was misread as "f u and your emergency" by someone who had announced an emergency a few seconds earlier :) [23:00:03] oic [23:00:08] atgo: the meeting? [23:00:27] the CN mess up? [23:00:35] whaaat are u referring to? [23:01:16] oh kay euphemisms. yeah the email I sent linking to GlobalAllocation [23:01:44] ewulczyn____ is trying to do some banner numbers stuff, and we noticed that the allocations are very wrong when mobile and desktop campaigns overlap. [23:01:52] Thanks, that should be a card. [23:02:27] It's very likely that the allocations are being calculated wrong and we're dramatically curtailing everyone's traffic under these circumstances. [23:02:34] i see [23:02:34] now, meating [23:02:44] yeah, please make a card [23:08:41] * K4-713 giggles quietly at backscroll [23:10:23] ejegg: I can't seem to find Mr. Jass anywhere in civi. Which is strange. [23:10:30] You'd think that after seven years... haha [23:18:23] Gah. [23:22:40] heh [23:30:10] * K4-713 frowns [23:30:21] This is a lot like pulling a thread on a sweater. [23:32:45] That's one sleeve gone. [23:32:56] Awesome tank-top sweater ftw. [23:37:12] awight|mtg: CN allocations wrong? [23:39:36] AndyRussG: awight|mtg is... well, yes. In a meeting. [23:39:51] Ah I thought it was a mtg [23:40:23] Mango Tang. [23:40:39] But. [23:40:52] Mustard-topped garlic! [23:40:53] We do have reason to believe that we are undercutting our own traffic. [23:40:58] ooo. [23:41:02] K4-713: hmmm [23:41:28] In the allocation log... there were three campaigns. [23:41:46] One was for desktop, one mobile, one... tablets or something. [23:41:53] And they all said 33%, instead of 100. [23:42:14] Ah right [23:42:17] That makes perfect sense [23:42:22] Does it? [23:42:25] Yes [23:42:32] That there is a problem? [23:42:35] Because device targeting is a banner criteria [23:42:55] ...not campaign level. [23:42:58] I get it. [23:43:00] So I imagine each campaign gets sent to 33% of the users, regardles of the device [23:43:11] Awesome. [23:43:23] Then when it reaches the device, it only shows to the users that have a device compatible with that campaign [23:43:29] mhm [23:43:41] Not 100% sure that's what would happen but I can do a test case locally [23:43:46] Please do. :) [23:44:06] since I've been working on allocations fits in nicely with my work evening [23:44:09] K u bet! [23:44:15] Thanks! [23:44:20] np :) [23:45:07] In other news, I suddenly want to rewrite half of DonationInterface. [23:45:34] ... [23:45:39] I should probably not do that right now. [23:50:32] K4-713: didn't you not like rewriting things? [23:53:15] AndyRussG: Actually... I love it. When I have the time, which is never. [23:54:02] Heheh right good point [23:54:24] It's probably about time to burn at least half of this to the ground. [23:54:42] We know more things than we did before. [23:54:58] Hmm I know the feeling [23:54:59] I don't know much about DonationInterface actually... [23:55:58] I have this sinking suspicion that anybody who gets in there long enough to really understand the globalcollect integration, can read my mind for three months afterward.