[00:34:06] #1283: (K) Tech Task #1283 ALL Nightly Audit Scripts: Alter queue message generation to include transaction source data: O:K|TS:B|P:SH|TT:CT|NT:Y|NV:Y|IU:N|TA:NR|BG:#Maef|BC:#Iit|G-TS:#A|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1283 [00:41:06] #1284: (K) Tech Task #1284 Alter the queue consumer for final consumption of transaction source data: O:K|TS:B|P:SH|TT:CT|NT:Y|NV:Y|IU:N|TA:C|BG:#Maef|BC:#Iit|G-TS:#A|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1284 [07:30:40] #1276: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276 [07:33:40] #1276: (AW) Cool. I'm imagining a small step beyond this, ... -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276 [08:15:40] #1276: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276 [08:30:40] #1285: (AW) Tech Task #1285 Reconstruct source metadata from our logs: O:AW|TS:B|P:NtH|TT:CT|NT:Y|NV:Y|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285 [08:31:40] #1285: (AW) ->Sprint 36 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285 [08:31:40] #1286: (AW) Tech Task #1286 Spike: how many people need to be thanked?: O:AW|TS:B|P:E|TT:S|NT:Y|NV:Y|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1286 [08:32:40] #1286: (AW) ->Sprint 36 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1286 [08:34:32] (PS1) Adamw: New schema for transaction source metadata [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/105149 [08:35:41] #1286: (AW) TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1286 [08:35:41] #1240: (AW) O:AW|TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1240 [08:35:41] #632: (AW) O:AW|TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/632 [08:35:41] #631: (AW) O:AW|TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/631 [08:36:41] #1285: (AW) TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285 [08:36:41] #1278: (AW) ->Sprint 36 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1278 [08:36:41] #1267: (AW) TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1267 [08:37:40] #994: (AW) TS:DR|MtIDo:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/994 [08:41:39] #1278: (AW) Consider tradeoffs in normalizing some source i... -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1278 [10:48:53] (PS1) Adamw: WIP: (FR #1285) Munge logs for source metadata [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105159 [11:33:00] (PS2) Adamw: WIP: (FR #1285) Munge logs for source metadata [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105159 [15:38:05] jessicarobell: #wikimedia-translation too? [15:39:10] @PiRSquared: sure! just joined! thanks [19:25:04] (CR) Mwalker: [C: 2] Higher-level move function, to cross filesystems [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/104760 (owner: Adamw) [20:29:26] (PS1) Adamw: Output another copy of stats in JSON format [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105254 [20:44:40] #1287: (AW) Tech Task #1287 Civi export using saved mapping is causing OoM error: O:AW|TS:B|P:SH|TT:CT|NT:Y|NV:Y|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1287 [20:45:14] pizzzacat: what format do you need the json data in? [20:45:29] it seems to me that awight's patch is going to be very verbose [20:45:46] mwalker: pizzzacat: yeah i can supply example generated files if that's helpful [20:45:57] and... yes. json is gonna be hell of verbose [20:47:00] mwalker: are you implying we should do something like CSV wrapped in json? [20:47:03] I'm wondering if we want to tear out some of the code and make it output a hash of date: value (for the single value files) [20:47:27] ah [20:47:34] or date: [value, value, value] [20:51:42] I need an array of objects [20:52:08] pizzzacat: & you care about the format of each object? [20:52:15] like [ { [20:52:15] "date":"2012-12-01T00:00:00+0", [20:52:15] "country":"AE", [20:52:15] "max":21.60604, [20:52:15] "sum":21.60604, [20:52:16] "avg":21.60604, [20:52:16] "donations":1, [20:52:17] "ytdsum":21.60604 [20:52:17] }, [20:52:18] { [20:52:18] "date":"2012-12-01T00:00:00+0", [20:52:19] "country":"AU", [20:52:19] "max":104.1, [20:52:20] "sum":4973.03, [20:52:20] oh [20:52:31] yes that's what i made with the code in CR [20:52:32] pivoted. is this a pain? [20:52:37] no [20:52:51] but mwalker has a good point that this will increase file size by... 2x [20:53:05] it's a matter of where we want the pain to be [20:53:45] well, 2x filesize is edging towards client-side evil [20:53:59] we can deliver using transparent .gz most of the time... [20:54:00] but... [20:54:07] client memory might be affected [20:54:16] that's another thing u might want to profile when comparing the formats [20:56:01] * awight rubs eyes. *don't* google for that [20:56:11] hah too late [20:56:36] yes. we are going to be using quite a bit of data here. [20:57:57] I wish... there were a good way to shard this data and load as needed [20:58:07] yeah client side [20:58:11] hrm [20:58:15] hrmmm [20:59:42] hmm yeah two of those files are 2MB [21:00:28] transmission time is prolly OK, cos they compress.gz to about 300KB [21:01:34] probably loading 2MB into memory is pretty far below the threshold of, users start to notice something wicked? [21:21:56] * mwalker points to the visual editor [21:34:21] mwalker: I think 1MB of javascript is quite a bit different than just data structs [21:35:27] mwalker: hey check out this lib that pizza is using: http://harvesthq.github.io/chosen/ [21:36:29] yep [21:36:41] a while ago someone tried to get that into MW and failed [21:36:47] mostly because of bikeshedding [21:37:03] ugh. it kicks the crap outta jquery.multiselect or whatever [21:37:14] you mean that one that I'm using in CN? [21:37:21] yeah [21:37:30] I chose that one because it was better for managing large lists of things [21:37:49] really? Chosen seems to do that pretty well [21:37:51] at least, imo [21:37:55] what was the issue? [21:38:23] Maybe once you have a large number of things selected? I see... [21:39:02] oh hah yeah it turns into a many-line tagcould clusterfuck [21:39:17] *nods* that being said; I'm not totally sold on the solution we have now [21:39:23] I just haven't found one I think is better [21:44:48] (CR) Mwalker: [C: 2] "Not sold on this being the best format; but it'll work for the immediate purposes." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105254 (owner: Adamw) [21:45:08] awight: if I update tools on al; is that going to cause splosions? [21:46:46] hmm... given that it's locked at a particular version [21:47:29] looks like the big thing is your "No longer suppress thank-yous" patch [21:47:32] that OK to go out? [21:50:15] mwalker: nope it's fine to update, now [21:53:18] !log updating fundraising-tools on al from b54854fcdd7a272ca2af8475affa033517456d07 (for statistics generation in json format) [21:53:19] there anybody out there [21:53:30] !help [21:53:36] Logged the message, Master [21:53:59] wait... do wm-bot and morebots both respond to !log!? [21:54:12] wm-bot: help [21:54:18] @help [21:54:19] I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.2.0 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features [21:55:38] jeremyb: would you happen to know why wm-bot would have said "there anyone out there" when I !log'd? [21:58:58] mwalker: K4|food: jfyi, pizzzacat is kicking down doors. I accidentally unveiled the dashboard and there was immediately a cheering crowd. The seriously. [21:59:08] interesting awight... it doesn't seem to have actually create the json files [21:59:16] oooh; is there a demo I can cheer at? [21:59:30] awight: Aw, you did the big reveal, huh? [21:59:53] Who caught it? [22:00:04] huh? [22:00:20] ... [22:00:25] interesting... [22:00:30] bahahahaha [22:00:48] Wait wait, where are the cheerable things [22:00:55] +1 [22:01:19] GAH! So much for surprise shiny. [22:01:20] K4-713: it was terrible. I was surrounded by designers for a minute ;) [22:01:40] it'll still be wowable. [22:01:45] K4-713: I know! It was an accident: i read the README and... suddenly there was something amazing actually running on my computer. [22:01:47] there's other tricks up my sleeve ;) [22:01:49] it *worked* [22:01:58] haa [22:02:03] Oh, I'm sure it will. I was just hoping to keep people off your back for as long as possible, pizzzacat. ;) [22:02:15] YOU HEAR THAT, PEOPLE? :p [22:02:22] me too…THANKS awight!!@)#(*@ [22:02:24] mwalker: lol that's cos the code was not updated [22:02:35] pizzzacat: nice Q-bert cuss-out [22:02:48] BTW. fire alarm. [22:02:56] Ah, there it goes. Enjoy. [22:03:17] !log tools updated from b54854fcdd7a272ca2af8475affa033517456d07 to 0df36fd9bc55b24ffad49c0dbd0db67d67a2b1f7 [22:03:17] there anybody out there [22:03:29] wm-bot: go away [22:03:34] Logged the message, Master [22:03:37] haaarrr [22:03:59] mwalker: fwiw, frcodeup syntax for "unlock from revision" is -p PROJ=head [22:04:07] ya [22:04:16] I realized that I had to do that after you said it [22:04:32] ah yeah. pretty soon that will all be over [22:05:41] Wellp, I have to run out again for who knows how long, in a few minutes. [22:05:47] pizzzacat: New .json files have been uploaded to fr-data [22:06:01] Don't do anything too interesting while I'm.... at all. ;) [22:06:07] beheeehehe [22:06:15] * awight swings down from the curtainstrings [22:06:57] 'course, if you want to send me a SEKRIT DEMO LINK while I'm out... haha [22:07:06] !log [22:07:06] there anybody out there [22:07:09] * marktraceur jelly [22:07:12] !log del [22:07:12] Successfully removed log [22:07:16] !log [22:07:16] wha [22:07:27] mwalker [22:07:30] Logged the message, Master [22:07:41] awight: were you stalking me in jenkins? [22:07:46] mwalker: it seems [22:07:47] and jeremyb; what did you just do!? [22:09:09] mwalker: :-) :D :D [22:09:22] * jeremyb runs away [22:09:52] K4-713: oh... hehe, uthere? https://gerrit.wikimedia.org/r/105159 [22:10:41] awight: orly? [22:10:54] does it take a few mins to show up? [22:11:15] pizzzacat: ya; I just updated the script on aluminium and ran it [22:11:22] but it does update every 15 minutes normally [22:11:32] pizzzacat: they are uploaded, u don't see? [22:12:05] awight: Did you torch "server"? [22:12:26] pizzzacat: samarium:/var/www/*.json [22:12:42] K4-713: no, i did make some small tweaks to the supercard description of data needed tho [22:13:07] K4-713: you're right, server can be reconstructed from logs [22:13:18] K4-713: but you want me working on this or not? [22:13:20] it's really crazy [22:13:25] but sort of fun and very easy [22:13:32] I love multi-line regexen [22:13:57] * awight is now worried about future employers stumbling across that quote ;) [22:14:06] hahaha [22:14:16] Too late. The internet has it now. [22:14:20] talk about dig my own grave and fill it with raw sewage. [22:14:35] awight: I guess it's not super-important to me to backfill, so much as it is to get all the new messages in. But possibly I'm misunderstanding where this code goes. [22:14:45] I LOVE IT. [22:14:57] ...anyway. [22:15:00] K4-713: hehehe now I know you're just set to "replay" [22:15:11] ok. I will drop the backfill [22:15:23] I mean, it seems fairly simple. Ish. [22:15:38] ...now that you've basically done it. [22:15:40] :p [22:15:40] K4-713: I had this vision that we could actually audit how messages moved through the system [22:15:55] Hmmmmmm. [22:16:12] like, one msg came in through the frontend, took 3 seconds to inject into the queue, 150 seconds until dequeued, by function X in the CRM [22:16:22] oho [22:16:26] That's... interesting. [22:16:27] then, a second message came in through IPN and 100 seconds later, it was rejected as a duplicate [22:16:30] yeah [22:16:38] So, definitely write that down somewhere. [22:17:07] https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285 -ish [22:17:15] At the moment, though, I'm mostly just interested in being able to identify procs misbehaving in realtime, more or less instantaneously from wherever we see the bad data. [22:17:35] um so not live? sorry. [22:17:46] And, the happy side-effect of being able to tell if the safety nets are being overused. [22:17:52] K4-713: there are other things to discuss before I add the metadata fields, IMO [22:18:16] anyway, I'll add notes to the cards for now [22:18:20] I'm sure there are. Most of that was sort of off the top of my head. [22:18:36] We should totally talk. I'll probably just agree with you. [22:18:51] Okay, now I really have to go. [22:19:02] Back in [some number of some other time units]. [22:21:13] pizzzacat: the files are live; just take the usual URLs and replace .csv with .json [22:25:40] #1276: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276 [22:33:07] * mwalker hates django [22:33:13] wonders how to turn it into a son [22:33:15] *song [22:34:38] you must have not met the guy [22:35:10] He spent all his guitar earnings throw parties for a few hundred "family" members [22:36:14] ... not finding links right away, but I think they burned down a few cafes with raucous partying [22:37:24] heh; never heard of this guy before [22:37:39] but I love how the photo on wiki has him causually smoking a cigarette whilst playing the guitar [22:37:58] yeah sorry i was trying to defuse the rising ire that Django the python framework inspires [22:38:31] as frameworks go it could be far worse actually [22:38:55] it's just that... what is the framework for, again? [22:39:17] something incredibly sketchy like creating normalized database rows on-the-fly [22:41:22] well; ish -- it provides a full HTTP framework in python [22:41:31] but... we don't use that? [22:41:36] and it tacks on database stuff [22:41:39] and no; we don't use it [22:41:46] peter intended to [22:41:54] if it comes with an ice cream maker, I'm sold [22:42:09] I'm sure somewhere there is a logo for it that looks like one [22:42:23] hehe that is a unicorn, not a spilled cone [22:42:25] speaking of which, kaycie got me a waffle maker for xmax -- soooo many waffles [22:47:12] * awight hiccup [22:55:36] (PS1) Ssmith: changed stackchart, fixed formatting and reload/resize/redraw issues [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105395 [22:55:37] (PS1) Ssmith: refactored for modularity and got real data into method breakdown. [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105396 [22:55:38] (PS1) Ssmith: fix pie chart, enlarge [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105397 [22:55:39] (PS1) Ssmith: refactor into more modular and scalable framework [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105398 [22:55:40] (PS1) Ssmith: dir restructure for better review [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105399 [23:00:33] (CR) Ssmith: "thanks for this!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105254 (owner: Adamw) [23:02:22] (CR) Adamw: [C: 2] changed stackchart, fixed formatting and reload/resize/redraw issues [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105395 (owner: Ssmith) [23:56:56] (CR) Mwalker: [C: 2] Schema migration for source metadata [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/104167 (owner: Adamw) [23:57:26] ^^ crap. I meant to say, WIP [23:59:02] oops [23:59:07] well... I can revert it? [23:59:13] (CR) Adamw: [C: 2] refactored for modularity and got real data into method breakdown. (4 comments) [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105396 (owner: Ssmith) [23:59:50] mwalker: nah, just as long as we don't accidentally run it!