[00:34:06] <slander>	 #1283: (K) Tech Task #1283 ALL Nightly Audit Scripts: Alter queue message generation to include transaction source data: O:K|TS:B|P:SH|TT:CT|NT:Y|NV:Y|IU:N|TA:NR|BG:#Maef|BC:#Iit|G-TS:#A|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1283
[00:41:06] <slander>	 #1284: (K) Tech Task #1284 Alter the queue consumer for final consumption of transaction source data: O:K|TS:B|P:SH|TT:CT|NT:Y|NV:Y|IU:N|TA:C|BG:#Maef|BC:#Iit|G-TS:#A|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1284
[07:30:40] <slander>	 #1276: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276
[07:33:40] <slander>	 #1276: (AW) Cool. I'm imagining a small step beyond this, ... -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276
[08:15:40] <slander>	 #1276: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276
[08:30:40] <slander>	 #1285: (AW) Tech Task #1285 Reconstruct source metadata from our logs: O:AW|TS:B|P:NtH|TT:CT|NT:Y|NV:Y|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285
[08:31:40] <slander>	 #1285: (AW) ->Sprint 36 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285
[08:31:40] <slander>	 #1286: (AW) Tech Task #1286 Spike: how many people need to be thanked?: O:AW|TS:B|P:E|TT:S|NT:Y|NV:Y|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1286
[08:32:40] <slander>	 #1286: (AW) ->Sprint 36 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1286
[08:34:32] <grrrit-wm>	 (PS1) Adamw: New schema for transaction source metadata [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/105149
[08:35:41] <slander>	 #1286: (AW) TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1286
[08:35:41] <slander>	 #1240: (AW) O:AW|TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1240
[08:35:41] <slander>	 #632: (AW) O:AW|TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/632
[08:35:41] <slander>	 #631: (AW) O:AW|TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/631
[08:36:41] <slander>	 #1285: (AW) TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285
[08:36:41] <slander>	 #1278: (AW) ->Sprint 36 -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1278
[08:36:41] <slander>	 #1267: (AW) TS:DR -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1267
[08:37:40] <slander>	 #994: (AW) TS:DR|MtIDo:(s -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/994
[08:41:39] <slander>	 #1278: (AW) Consider tradeoffs in normalizing some source i... -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1278
[10:48:53] <grrrit-wm>	 (PS1) Adamw: WIP: (FR #1285) Munge logs for source metadata [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105159
[11:33:00] <grrrit-wm>	 (PS2) Adamw: WIP: (FR #1285) Munge logs for source metadata [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105159
[15:38:05] <PiRSquared>	 jessicarobell: #wikimedia-translation too?
[15:39:10] <jessicarobell>	 @PiRSquared: sure! just joined! thanks
[19:25:04] <grrrit-wm>	 (CR) Mwalker: [C: 2] Higher-level move function, to cross filesystems [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/104760 (owner: Adamw)
[20:29:26] <grrrit-wm>	 (PS1) Adamw: Output another copy of stats in JSON format [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105254
[20:44:40] <slander>	 #1287: (AW) Tech Task #1287 Civi export using saved mapping is causing OoM error: O:AW|TS:B|P:SH|TT:CT|NT:Y|NV:Y|IU:N|T:TT Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1287
[20:45:14] <mwalker>	 pizzzacat: what format do you need the json data in?
[20:45:29] <mwalker>	 it seems to me that awight's patch is going to be very verbose
[20:45:46] <awight>	 mwalker: pizzzacat: yeah i can supply example generated files if that's helpful
[20:45:57] <awight>	 and... yes.  json is gonna be hell of verbose
[20:47:00] <awight>	 mwalker: are you implying we should do something like CSV wrapped in json?
[20:47:03] <mwalker>	 I'm wondering if we want to tear out some of the code and make it output a hash of date: value (for the single value files)
[20:47:27] <awight>	 ah
[20:47:34] <mwalker>	 or date: [value, value, value]
[20:51:42] <pizzzacat>	 I need an array of objects
[20:52:08] <awight>	 pizzzacat: & you care about the format of each object?
[20:52:15] <pizzzacat>	 like [  {
[20:52:15] <pizzzacat>	     "date":"2012-12-01T00:00:00+0",
[20:52:15] <pizzzacat>	     "country":"AE",
[20:52:15] <pizzzacat>	     "max":21.60604,
[20:52:15] <pizzzacat>	     "sum":21.60604,
[20:52:16] <pizzzacat>	     "avg":21.60604,
[20:52:16] <pizzzacat>	     "donations":1,
[20:52:17] <pizzzacat>	     "ytdsum":21.60604
[20:52:17] <pizzzacat>	   },
[20:52:18] <pizzzacat>	   {
[20:52:18] <pizzzacat>	     "date":"2012-12-01T00:00:00+0",
[20:52:19] <pizzzacat>	     "country":"AU",
[20:52:19] <pizzzacat>	     "max":104.1,
[20:52:20] <pizzzacat>	     "sum":4973.03,
[20:52:20] <awight>	 oh
[20:52:31] <awight>	 yes that's what i made with the code in CR
[20:52:32] <pizzzacat>	 pivoted. is this a pain?
[20:52:37] <awight>	 no
[20:52:51] <awight>	 but mwalker has a good point that this will increase file size by... 2x
[20:53:05] <pizzzacat>	 it's a matter of where we want the pain to be
[20:53:45] <awight>	 well, 2x filesize is edging towards client-side evil
[20:53:59] <awight>	 we can deliver using transparent .gz most of the time...
[20:54:00] <awight>	 but...
[20:54:07] <awight>	 client memory might be affected
[20:54:16] <awight>	 that's another thing u might want to profile when comparing the formats
[20:56:01] * awight  rubs eyes.  *don't* google for that

[20:56:11] <pizzzacat>	 hah too late
[20:56:36] <pizzzacat>	 yes. we are going to be using quite a bit of data here.
[20:57:57] <awight>	 I wish... there were a good way to shard this data and load as needed
[20:58:07] <pizzzacat>	 yeah client side
[20:58:11] <pizzzacat>	 hrm
[20:58:15] <pizzzacat>	 hrmmm
[20:59:42] <awight>	 hmm yeah two of those files are 2MB
[21:00:28] <awight>	 transmission time is prolly OK, cos they compress.gz to about 300KB
[21:01:34] <awight>	 probably loading 2MB into memory is pretty far below the threshold of, users start to notice something wicked?
[21:21:56] * mwalker  points to the visual editor

[21:34:21] <awight>	 mwalker: I think 1MB of javascript is quite a bit different than just data structs
[21:35:27] <awight>	 mwalker: hey check out this lib that pizza is using: http://harvesthq.github.io/chosen/
[21:36:29] <mwalker>	 yep
[21:36:41] <mwalker>	 a while ago someone tried to get that into MW and failed
[21:36:47] <mwalker>	 mostly because of bikeshedding
[21:37:03] <awight>	 ugh.  it kicks the crap outta jquery.multiselect or whatever
[21:37:14] <mwalker>	 you mean that one that I'm using in CN?
[21:37:21] <awight>	 yeah
[21:37:30] <mwalker>	 I chose that one because it was better for managing large lists of things
[21:37:49] <awight>	 really?  Chosen seems to do that pretty well
[21:37:51] <mwalker>	 at least, imo
[21:37:55] <awight>	 what was the issue?
[21:38:23] <awight>	 Maybe once you have a large number of things selected?  I see...
[21:39:02] <awight>	 oh hah yeah it turns into a many-line tagcould clusterfuck
[21:39:17] <mwalker>	 *nods* that being said; I'm not totally sold on the solution we have now
[21:39:23] <mwalker>	 I just haven't found one I think is better
[21:44:48] <grrrit-wm>	 (CR) Mwalker: [C: 2] "Not sold on this being the best format; but it'll work for the immediate purposes." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105254 (owner: Adamw)
[21:45:08] <mwalker>	 awight: if I update tools on al; is that going to cause splosions?
[21:46:46] <mwalker>	 hmm... given that it's locked at a particular version
[21:47:29] <mwalker>	 looks like the big thing is your "No longer suppress thank-yous" patch
[21:47:32] <mwalker>	 that OK to go out?
[21:50:15] <awight>	 mwalker: nope it's fine to update, now
[21:53:18] <mwalker>	 !log updating fundraising-tools on al from b54854fcdd7a272ca2af8475affa033517456d07 (for statistics generation in json format)
[21:53:19] <wm-bot>	 there anybody out there
[21:53:30] <mwalker>	 !help
[21:53:36] <morebots>	 Logged the message, Master
[21:53:59] <mwalker>	 wait... do wm-bot and morebots both respond to !log!?
[21:54:12] <mwalker>	 wm-bot: help
[21:54:18] <mwalker>	 @help
[21:54:19] <wm-bot>	 I am running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.20.2.0 my source code is licensed under GPL and located at https://github.com/benapetr/wikimedia-bot I will be very happy if you fix my bugs or implement new features
[21:55:38] <mwalker>	 jeremyb: would you happen to know why wm-bot would have said "there anyone out there" when I !log'd?
[21:58:58] <awight>	 mwalker: K4|food: jfyi, pizzzacat is kicking down doors.  I accidentally unveiled the dashboard and there was immediately a cheering crowd.  The seriously.
[21:59:08] <mwalker>	 interesting awight... it doesn't seem to have actually create the json files
[21:59:16] <mwalker>	 oooh; is there a demo I can cheer at?
[21:59:30] <K4|food>	 awight: Aw, you did the big reveal, huh?
[21:59:53] <K4|food>	 Who caught it?
[22:00:04] <huh>	 huh?
[22:00:20] <K4-713>	 ...
[22:00:25] <mwalker>	 interesting...
[22:00:30] <K4-713>	 bahahahaha
[22:00:48] <marktraceur>	 Wait wait, where are the cheerable things
[22:00:55] <Nemo_bis>	 +1
[22:01:19] <K4-713>	 GAH! So much for surprise shiny.
[22:01:20] <awight>	 K4-713: it was terrible.  I was surrounded by designers for a minute ;)
[22:01:40] <pizzzacat>	 it'll still be wowable.
[22:01:45] <awight>	 K4-713: I know!  It was an accident: i read the README and... suddenly there was something amazing actually running on my computer.
[22:01:47] <pizzzacat>	 there's other tricks up my sleeve ;)
[22:01:49] <awight>	 it *worked*
[22:01:58] <pizzzacat>	 haa
[22:02:03] <K4-713>	 Oh, I'm sure it will. I was just hoping to keep people off your back for as long as possible, pizzzacat. ;)
[22:02:15] <K4-713>	 YOU HEAR THAT, PEOPLE? :p
[22:02:22] <pizzzacat>	 me too…THANKS awight!!@)#(*@
[22:02:24] <awight>	 mwalker: lol that's cos the code was not updated
[22:02:35] <awight>	 pizzzacat: nice Q-bert cuss-out
[22:02:48] <awight>	 BTW.  fire alarm.
[22:02:56] <K4-713>	 Ah, there it goes. Enjoy.
[22:03:17] <awight>	 !log tools updated from b54854fcdd7a272ca2af8475affa033517456d07 to 0df36fd9bc55b24ffad49c0dbd0db67d67a2b1f7
[22:03:17] <wm-bot>	 there anybody out there
[22:03:29] <awight>	 wm-bot: go away
[22:03:34] <morebots>	 Logged the message, Master
[22:03:37] <awight>	 haaarrr
[22:03:59] <awight>	 mwalker: fwiw, frcodeup syntax for "unlock from revision" is -p PROJ=head
[22:04:07] <mwalker>	 ya
[22:04:16] <mwalker>	 I realized that I had to do that after you said it
[22:04:32] <awight>	 ah yeah.  pretty soon that will all be over
[22:05:41] <K4-713>	 Wellp, I have to run out again for who knows how long, in a few minutes.
[22:05:47] <awight>	 pizzzacat: New .json files have been uploaded to fr-data
[22:06:01] <K4-713>	 Don't do anything too interesting while I'm.... at all. ;)
[22:06:07] <awight>	 beheeehehe
[22:06:15] * awight  swings down from the curtainstrings

[22:06:57] <K4-713>	 'course, if you want to send me a SEKRIT DEMO LINK while I'm out... haha
[22:07:06] <jeremyb>	 !log
[22:07:06] <wm-bot>	 there anybody out there
[22:07:09] * marktraceur  jelly

[22:07:12] <jeremyb>	 !log del
[22:07:12] <wm-bot>	 Successfully removed log
[22:07:16] <jeremyb>	 !log
[22:07:16] <awight>	 wha
[22:07:27] <jeremyb>	 mwalker
[22:07:30] <morebots>	 Logged the message, Master
[22:07:41] <mwalker>	 awight: were you stalking me in jenkins?
[22:07:46] <awight>	 mwalker: it seems
[22:07:47] <mwalker>	 and jeremyb; what did you just do!?
[22:09:09] <jeremyb>	 mwalker: :-) :D :D
[22:09:22] * jeremyb  runs away

[22:09:52] <awight>	 K4-713: oh... hehe, uthere? https://gerrit.wikimedia.org/r/105159
[22:10:41] <pizzzacat>	 awight: orly?
[22:10:54] <pizzzacat>	 does it take a few mins to show up?
[22:11:15] <mwalker>	 pizzzacat: ya; I just updated the script on aluminium and ran it
[22:11:22] <mwalker>	 but it does update every 15 minutes normally
[22:11:32] <awight>	 pizzzacat: they are uploaded, u don't see?
[22:12:05] <K4-713>	 awight: Did you torch "server"?
[22:12:26] <awight>	 pizzzacat: samarium:/var/www/*.json
[22:12:42] <awight>	 K4-713: no, i did make some small tweaks to the supercard description of data needed tho
[22:13:07] <awight>	 K4-713: you're right, server can be reconstructed from logs
[22:13:18] <awight>	 K4-713: but you want me working on this or not?
[22:13:20] <awight>	 it's really crazy
[22:13:25] <awight>	 but sort of fun and very easy
[22:13:32] <awight>	 I love multi-line regexen
[22:13:57] * awight  is now worried about future employers stumbling across that quote ;)

[22:14:06] <K4-713>	 hahaha
[22:14:16] <K4-713>	 Too late. The internet has it now.
[22:14:20] <awight>	 talk about dig my own grave and fill it with raw sewage.
[22:14:35] <K4-713>	 awight: I guess it's not super-important to me to backfill, so much as it is to get all the new messages in. But possibly I'm misunderstanding where this code goes.
[22:14:45] <K4-713>	 I LOVE IT.
[22:14:57] <K4-713>	 ...anyway.
[22:15:00] <awight>	 K4-713: hehehe now I know you're just set to "replay"
[22:15:11] <awight>	 ok.  I will drop the backfill
[22:15:23] <K4-713>	 I mean, it seems fairly simple. Ish.
[22:15:38] <K4-713>	 ...now that you've basically done it.
[22:15:40] <K4-713>	 :p
[22:15:40] <awight>	 K4-713: I had this vision that we could actually audit how messages moved through the system
[22:15:55] <K4-713>	 Hmmmmmm.
[22:16:12] <awight>	 like, one msg came in through the frontend, took 3 seconds to inject into the queue, 150 seconds until dequeued, by function X in the CRM
[22:16:22] <K4-713>	 oho
[22:16:26] <K4-713>	 That's... interesting.
[22:16:27] <awight>	 then, a second message came in through IPN and 100 seconds later, it was rejected as a duplicate
[22:16:30] <awight>	 yeah
[22:16:38] <K4-713>	 So, definitely write that down somewhere.
[22:17:07] <awight>	 https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1285 -ish
[22:17:15] <K4-713>	 At the moment, though, I'm mostly just interested in being able to identify procs misbehaving in realtime, more or less instantaneously from wherever we see the bad data.
[22:17:35] <pizzzacat>	 um so not live? sorry.
[22:17:46] <K4-713>	 And, the happy side-effect of being able to tell if the safety nets are being overused.
[22:17:52] <awight>	 K4-713: there are other things to discuss before I add the metadata fields, IMO
[22:18:16] <awight>	 anyway, I'll add notes to the cards for now
[22:18:20] <K4-713>	 I'm sure there are. Most of that was sort of off the top of my head.
[22:18:36] <K4-713>	 We should totally talk. I'll probably just agree with you.
[22:18:51] <K4-713>	 Okay, now I really have to go.
[22:19:02] <K4-713>	 Back in [some number of some other time units].
[22:21:13] <mwalker>	 pizzzacat: the files are live; just take the usual URLs and replace .csv with .json
[22:25:40] <slander>	 #1276: (AW) Description changed -- https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1276
[22:33:07] * mwalker  hates django

[22:33:13] <mwalker>	 wonders how to turn it into a son
[22:33:15] <mwalker>	 *song
[22:34:38] <awight>	 you must have not met the guy
[22:35:10] <awight>	 He spent all his guitar earnings throw parties for a few hundred "family" members
[22:36:14] <awight>	 ... not finding links right away, but I think they burned down a few cafes with raucous partying
[22:37:24] <mwalker>	 heh; never heard of this guy before
[22:37:39] <mwalker>	 but I love how the photo on wiki has him causually smoking a cigarette whilst playing the guitar
[22:37:58] <awight>	 yeah sorry i was trying to defuse the rising ire that Django the python framework inspires
[22:38:31] <mwalker>	 as frameworks go it could be far worse actually
[22:38:55] <awight>	 it's just that... what is the framework for, again?
[22:39:17] <awight>	 something incredibly sketchy like creating normalized database rows on-the-fly
[22:41:22] <mwalker>	 well; ish -- it provides a full HTTP framework in python
[22:41:31] <awight>	 but... we don't use that?
[22:41:36] <mwalker>	 and it tacks on database stuff
[22:41:39] <mwalker>	 and no; we don't use it
[22:41:46] <mwalker>	 peter intended to
[22:41:54] <awight>	 if it comes with an ice cream maker, I'm sold
[22:42:09] <mwalker>	 I'm sure somewhere there is a logo for it that looks like one
[22:42:23] <awight>	 hehe that is a unicorn, not a spilled cone
[22:42:25] <mwalker>	 speaking of which, kaycie got me a waffle maker for xmax -- soooo many waffles
[22:47:12] * awight  hiccup

[22:55:36] <grrrit-wm>	 (PS1) Ssmith: changed stackchart, fixed formatting and reload/resize/redraw issues [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105395
[22:55:37] <grrrit-wm>	 (PS1) Ssmith: refactored for modularity and got real data into method breakdown. [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105396
[22:55:38] <grrrit-wm>	 (PS1) Ssmith: fix pie chart, enlarge [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105397
[22:55:39] <grrrit-wm>	 (PS1) Ssmith: refactor into more modular and scalable framework [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105398
[22:55:40] <grrrit-wm>	 (PS1) Ssmith: dir restructure for better review [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105399
[23:00:33] <grrrit-wm>	 (CR) Ssmith: "thanks for this!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/105254 (owner: Adamw)
[23:02:22] <grrrit-wm>	 (CR) Adamw: [C: 2] changed stackchart, fixed formatting and reload/resize/redraw issues [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105395 (owner: Ssmith)
[23:56:56] <grrrit-wm>	 (CR) Mwalker: [C: 2] Schema migration for source metadata [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/104167 (owner: Adamw)
[23:57:26] <awight>	 ^^ crap.  I meant to say, WIP
[23:59:02] <mwalker>	 oops
[23:59:07] <mwalker>	 well... I can revert it?
[23:59:13] <grrrit-wm>	 (CR) Adamw: [C: 2] refactored for modularity and got real data into method breakdown. (4 comments) [wikimedia/fundraising/DataDashboard] - https://gerrit.wikimedia.org/r/105396 (owner: Ssmith)
[23:59:50] <awight>	 mwalker: nah, just as long as we don't accidentally run it!