[04:09:11] (CR) Ejegg: [C: 1] "I can confirm this causes no errors when the code is deployed without the schema being updated first (edited compaign settings, added camp" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/153385 (https://bugzilla.wikimedia.org/31689) (owner: AndyRussG) [04:09:32] ejegg|away: nice, thanks much! :) [17:19:33] ccogdill: fyi, I realized that I sent Monday's recurring GC letters without the apologetic paragraph... [17:19:43] hah [17:19:52] well, we got absolutely 0 responses about it [17:20:00] so I guess that’s fine [17:20:03] I know... sorry to squander your time in negotiating the paragraph contents! [17:20:10] haha, no worries [17:20:19] ultimately it’s a really small group of people [17:20:37] Yeah, and it's not the worst thing. Much different than "restarting" a lapsed subscription [17:20:59] yeah, much more fallout from that sort of thing [17:21:07] speaking of which — have we gotten anywhere with deciding what to do re the TY letters that never got sent? [17:21:08] We might consider just sending the other languages... [17:21:17] I’m okay with that [17:21:23] ccogdill: no, that's where we left it on Monday [17:21:24] probably not worth translators’ time [17:21:34] okay, I’m going to resurrect that thread [17:21:37] ccogdill: yep, ok I can take that paragraph down [17:21:39] thx! [17:21:42] okay [17:21:53] can you let Jessica know? [17:21:59] sure! [17:22:03] I suppose she should get to veto if she wants... [17:25:43] AndyRussG, atgo, awight: anything you want me to ask/share at the scrum of scrums? [17:26:16] ejegg: I haven't been to such a thing, so don't know the scope [17:27:09] do you need anything from other teams, or will your upcoming plans affect them? [17:27:13] Hey ejegg... Thanks, nothing that I can think of (and I'm also pretty ignorant about it) [17:27:36] Ah hmmm, only my evil secret plans, so no, no need to share :) [17:27:42] heh, ok [17:28:22] ccogdill: If we make the text slightly more general, it would be worth keeping around, I guess... "You were charged on {date}, but we never sent you the tax receipt." rather than the "recurring / 2014" stuff. [17:28:53] ejegg: ah, you make it sound downright useful! [17:28:56] hmmm [17:29:20] Hey, I confess I'm mostly in it to remember what ppl look like [17:29:31] I don’t think I prefer that to a better explanation. It just makes us sound careless [17:29:54] let’s email Jessica and see what she thinks, considering the lack of response from the biggest contingent of donors (English) [17:30:27] ccogdill: well, can we make it general without sounding careless? [17:30:54] I don't think "small technical issue which caused some monthly recurring donations to receive no tax receipt" sounds like we're that much more on top of things [17:31:02] fair [17:31:28] I guess to me “issue that got resolved” sounds better than “this just happened" [17:31:33] know what I mean? [17:31:45] I mean, it's important that we're representing what actually happened, so the next... 10,000 or so should possibly reference that we found the issue during internal auditing... [17:32:04] ccogdill: that's true! [17:32:33] are you looking ahead to potentially sending to the 2013 and older people as well? [17:32:49] I'm definitely keeping it in mind [17:32:53] yeah [17:33:05] And noticed that this text is pretty close to what we need. [17:33:06] okay so we want to scale our response — I’m on board with that [17:33:20] (PS5) Qgil: [WIP] move getGatewayClass into GatewayAdapter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64873 (owner: Awight) [17:33:26] (CR) jenkins-bot: [V: -1] [WIP] move getGatewayClass into GatewayAdapter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64873 (owner: Awight) [17:34:24] (CR) Qgil: "This patch is now a year old. Can still be merged? Should it be abandoned?" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/63823 (owner: Awight) [17:34:38] like “We recently discovered during an internal audit that a small subset of donors did not receive a confirmation of their donation. This email serves as confirmation for your donation on [date]” [17:34:39] awight? [17:35:06] ccogdill: that would nail it for the next batch [17:35:35] okay I’ll send to Jessica [17:36:00] thanks for thinking about this. definitely makes sense to knock this out at once [17:36:25] yeah thanks for holding me back from just deleting, then regretting it in two days :D [17:36:40] haha I hardly held you back. We would’ve both been regretting it! [17:38:00] (CR) Qgil: "Based on mwalker's comments last year, should this review be in -1 while Adam replies??" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/86790 (owner: Awight) [17:42:42] (PS3) Awight: WIP tests for the return_value_map [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/86790 [17:42:44] (CR) Qgil: [C: -1] "The commit summary should explain the change made, not simply the existing problem." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/105731 (owner: Awight) [17:42:46] (CR) jenkins-bot: [V: -1] WIP tests for the return_value_map [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/86790 (owner: Awight) [17:55:58] pizzzacat: you've got a promise from analytics to look at your charts soon [17:56:59] (PS6) Qgil: move getGatewayClass into GatewayAdapter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64873 (owner: Awight) [17:57:04] (CR) jenkins-bot: [V: -1] move getGatewayClass into GatewayAdapter [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64873 (owner: Awight) [17:57:30] (CR) Qgil: "Turns out it's not a WIP commit. I should have asked before. Sorry for the noise. It was done with good intentions! Lesson learned." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/64873 (owner: Awight) [17:59:51] and awight/AndyRussG, sumana's doing an RFC followup meeting at 2 PST and wants to know if we've designated a single point person (scapegoat? whipping boy?) for the central notice caching overhaul [18:01:41] ejegg: oh dear. I can tie myself to the whipping post for now. [18:01:53] (PDT btw) [18:02:05] oh, right, summer! [18:02:13] stupid clocks [18:02:32] btw, the meeting's following up on a bunch of different RFCs, not just Central Notice [18:02:52] ejegg: ok sure. cos we really aren't ready for a followup meeting until we discuss internally [18:03:02] cool [18:03:57] My opinion fwiw is that we shouldn't bother with the varnish solution. Instead, we should fix the algorithm (AndyRussG is on that), and target a long-term ESI thing. [18:04:06] err s/varnish/node/ [18:05:28] hey awight, question from Lisa re the TY thing [18:05:37] Yeah? Fewer extra services are definitely good [18:06:00] ccogdill: yes [18:06:11] when you came up with those numbers, did you filter out check/BT/other donors who received a paper tax receipt? I don’t know how well Civi’s back end allows you to distinguish between them [18:07:31] ccogdill: anyone who we didn't expect to get a receipt should have been filtered out. For example, check donors who didn't provide an email have "anonymous" or "no contact information" in the no_thank_you reason. [18:07:56] okay cool [18:08:00] The list I pulled is everyone with a "null" no-thank-you reason, which are the contributions that are pulled in by the thank-you job. [18:08:10] so we’re pretty confident these are all people who should have received one and didn't [18:08:17] We should definitely filter further though to understand the composition of that group [18:08:29] I can't even say I'm pretty confident yet [18:08:33] hmm [18:08:38] so Lisa is going to talk to legal [18:08:46] do you think we should wait and get some more info? [18:08:55] No, talking to legal makes sense [18:08:57] IMO [18:08:59] it’s not like another few days/week will make a huge difference... [18:09:00] okay [18:09:54] It's possible that the thank-you job would have rejected some number of those cos they were missing email info, but the problem so far is that we didn't even try... [18:11:45] okay [18:12:03] is Rosie supposed to be setting the “no_thank_you” reason? [18:12:24] I’m wondering if maybe some of these are check and BTs. [18:13:37] ccogdill: I can get those answers easily, once we start working on the issue [18:14:00] But for now it's a very safe bet that there are a bunch of people from 2013 and earlier who are missing their TY... [18:14:05] okay [18:14:22] so I’ll tell Lisa we don’t have the final # but feel confident at least *some* did not receive a receipt, either paper or email [18:14:38] Yeah, something around 10,000 :) [18:14:46] sounds good [18:14:51] thank you! [18:15:15] I'll let you know as soon as we have more details! [18:15:35] I need to understand exactly what happened, so we can patch the hole going forwards. [18:15:44] definitely [18:18:28] hey Jeff_Green! [18:18:47] hey [18:18:50] here for my lutetium setup :) [18:18:56] ok! [18:19:01] my computer freaked out this morning and now i have a new one [18:19:08] so.. sorry i'm late to the table :) [18:19:13] no worries [18:19:13] ejegg: scrum^2 still on? It would be worth asking Ops about the Varnish upgrade to support ESI... [18:19:18] atgo: you're a mac user? [18:19:23] baaahahaha [18:19:24] Jeff_Green: yes [18:19:39] ahem. [18:19:45] atgo: are you familiar with ssh [18:19:59] awight, sorry, it's done [18:20:16] ejegg: no worries! [18:20:27] Jeff_Green: in theory, though in practice not so much [18:20:40] ejegg: I was just reading all the Zero stuff, https://www.mediawiki.org/wiki/Requests_for_comment/Zero_architecture [18:20:53] ejegg: they really wanted to use ESI but are waiting for... support [18:20:59] atgo: ok, well for starters you'll need to generate an SSH keypair [18:21:15] so on your mac fire up a terminal [18:21:33] k [18:21:35] and there enter: "ssh-keygen -b 2048" [18:22:03] it will ask you for the file to save, default should be id_rsa which is fine [18:22:13] so just hit enter? [18:22:16] yep [18:22:25] next it will ask for a passphrase [18:22:40] which i can make whatever? [18:22:41] it's going to encrypt the file, and this passphrase is the key to open it [18:22:44] ahh cool [18:22:49] sec [18:22:52] Huh. There was an update from the wikipedia zero people at the SoS, but they didn't mention varnish upgrades [18:24:12] atgo: ok, so do the passphrase and hit enter, and type it again and enter again [18:24:16] (if you haven't already) [18:24:19] ok [18:26:17] done? [18:26:41] yes [18:26:57] ejegg: hrm. Yeah they probably gave up on it for now. [18:27:25] There are notes saying the upgrade should be jan 2014, then another note that they are still waiting in June [18:28:22] What's the RT ticket under the varnish heading in that page? [18:34:05] Jeff_Green: what now? :) [18:35:42] ejegg: crazy other thing. Detecting the carrier based on IP. [18:36:11] oh,cool [18:36:22] there's also https://www.mediawiki.org/wiki/Requests_for_comment/Partial_page_caching [18:36:48] AndyRussG: do you have any exposure to / interest in ESI magic? [18:37:26] sorry, childcare chaos [18:37:33] no worries [18:38:09] awight: oh man, we started asking for it 3 yrs ago [18:38:30] atgo: ok so! there's now a new pair of files in the ssh config dir in your homedir on your mac [18:38:44] ejegg: looks like gwicke was working on it 10 years ago! [18:38:55] wow [18:39:16] atgo: in your terminal type this: cat ~/.ssh/id_rsa.pub [18:39:29] Still... kind of the only way we can dig out of this multiple-round-trip banner hell though. [18:39:32] Jeff_Green: alright [18:39:38] that will output your SSH public key, which will be a long string of text [18:39:40] yep [18:39:48] copy that to an email and send it to me [18:39:51] awight: yup, seems like it [18:40:07] sent! [18:40:17] and also post it to somewhere in officewiki (i.e. your user page) so I can cross-check it for tampering in email [18:40:33] we could really use numbers here... I think we're delaying page content load, but I would need concrete timing info before saying that out loud. [18:41:21] There is a phantomjs thing I wrote 2 years ago which might be useful. [18:42:17] Jeff_Green: done [18:43:11] (PS1) Ssmith: Fix checkbox padding [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/153860 [18:43:16] (CR) jenkins-bot: [V: -1] Fix checkbox padding [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/153860 (owner: Ssmith) [18:43:57] awight: Hey... I don't know about ESI or magic but I canl learn about them [18:44:31] AndyRussG: well, you might want to brace for a snowball, either way! [18:44:53] * AndyRussG ducks and braces [18:45:05] AndyRussG: I hear Hogwart's has a good program [18:45:10] * AndyRussG peers out over ledge [18:45:33] heheh [18:45:48] calling all owls! [18:45:50] What snowball am I bracing for? [18:46:15] Jeff_Green: i'm going to have to run in 15, just fyi [18:47:10] AndyRussG: There's a short RFC followup to mwalker|away's CentralNotice idea for faster load times [18:47:19] https://www.mediawiki.org/wiki/Architecture_meetings/RFC_review_2014-08-13 [18:48:45] awight: rightoh, I was "at" the last meeting, didn't know there was another one today, thanks for the headsup! [18:49:10] AndyRussG: I was saying a minute ago, I prefer an earlier idea mwalker|away had, that we serve using ESI, I don't think it's worthwhile to spend a lot of time on an intermediate solution... [18:49:15] I'm probably wrong [18:49:30] Hmmmm [18:49:46] Did you see the results of the previous meeting? The discussion was pretty intense [18:50:16] Yeah I did read it over [18:50:32] The job of followup was kindly given to us both by atgo, no? [18:50:37] hehe yessir [18:51:09] I though the last meeting was productive, but I interpreted the conclusions as roughly, don't do this in client javascript / node, it's gonna be a huge mess. [18:51:15] :P [18:51:35] AndyRussG awight did you see sumana's email from last night that asks who the point person is for this? [18:51:38] atgo: for context, ejegg is heads-upping about a CentralNotice followup today [18:51:51] atgo: ok, i'll get your account set up and send an email with info [18:51:55] thanks! [18:52:04] awight: cool. thanks [18:53:07] atgo: no I didn't, I'm a bit behind on my mailing lists, should try to get up 2 date [18:53:19] ok. just re-forwarded to you [18:53:51] thank you! [18:53:53] atgo: Thanks! [18:54:09] I guess that first point will take up most of the three hour discussion [18:55:37] atgo: AndyRussG: hehe yeah point person. I can face the tomatoes for a while. [18:56:15] I know K4-713 has been trying to push CN off of our plate, though. We need to have an internal discussion about this, soon. [18:56:43] awight: we have been having that convo with mwalker|away and terry.. which is part of why we have AndyRussG now :) [18:57:08] K4-713 = Katie? [18:57:16] I know! But this is will not be a lone dev project, it's gonna be a lot of work. [18:57:21] AndyRussG: yes :D [18:57:26] she is part cyborg [18:57:51] Ah no wonder she wants to push people onto plates (jk) [18:58:49] Hehe there is a video about this... not finding it immediately though. [18:59:52] IMO we should stay involved in CN, but making the site faster by fixing banner delivery should be everybody's business. [19:00:06] Hmm interesting [19:00:20] Like, if ESI is the blocker, 10 years later, someone should be assigned to making that work... [19:01:18] NEway, I'm easy about doing whatever needs doing, I undersetand it's more than a one-person job [19:02:11] I'll read up a bit before the meeting [19:03:55] AndyRussG: are you comfortable being point person on this, yet? [19:04:19] sure! [19:04:46] * AndyRussG innocently throws self to hungry wolves who know aaaaaaall about ESI [19:04:54] No really, np [19:07:12] baby attack... [19:08:17] AndyRussG: sorry. baby attack! [19:08:25] here she comes again... [19:09:04] Heheh quick neutralize her by numbing her brain with an android device! [19:09:33] baahaha [19:09:55] she somehow knows that trick. she says "bye-bye" and slams my laptop closed [19:10:11] papa, get brain back! [19:10:19] Oh no! That's supposed to be your move [19:10:54] OK I saw your reply in logs. It's been signed now :p [19:11:28] I can offer limited emotional and technical support, at least. [19:12:58] OK, thanks! If all else fails you can ask Mari to help by going around closing laptops at the office then, if you like [19:13:43] I wish! No kids, fun, or dogs these days. [19:14:30] really? all those things are now vorboten there? [19:14:58] It was damaging bad morale! [19:15:25] No, there was a minor dog pee incident ("Honeygate") that turned into policy fail. [19:20:25] awight: hmmm, not sure why that reflects on kids being allowed in the office [19:20:35] I mean, I've never, ever heard of a kid pee incident... [19:32:08] BTW I guess you also saw Sumana's ping on the Engineering list about the DI and data mapper RFCs? (not related to previous topic) [19:34:28] AndyRussG: oh I saw something go by, it looked very positive! When is that scheduled? [19:34:58] awight: no nothing scheduled, just a note to say the RFCs are there [19:40:07] AndyRussG: looking forward to (re)reading! I can imagine the discussion will eventually involve a lot of heuristic soul-searching: how do you know when your project could benefit from a data mapping layer... [19:41:14] ejegg: Have you seen these, btw? Awesome skunkworks project AndyRussG fit into a larger skunk thing. [19:41:22] https://www.mediawiki.org/wiki/Requests_for_comment/Data_mapper [19:41:26] https://www.mediawiki.org/wiki/Requests_for_comment/Dependency_injection [19:41:41] AndyRussG: I had to hold ejegg back from writing DI just the other day, actually :p [19:41:53] awight: ah haha, and thanks also [19:42:07] I guess folks just have to follow their gut on whether their inner data mapper is a shepard or a babysitter [19:44:45] niiice [19:44:54] how to make friends and influence people, through code [19:44:58] heh. /me might just be writing Silverpop importer's constructor with more DI in mind [19:45:22] * awight shields self from the light [19:46:49] AndyRussG: can I ask if you agree with my thoughts about skipping the node banner delivery and going straight for ESI? [19:47:12] Just that... before I run my keymouth, we should sound like we had SOME internal discussion at all :-/ [19:47:22] awight: u may ask... [19:47:31] lol well drn it i will [19:47:59] I still nead to reed more about it all before I have much to say [19:48:30] np with me if you'd like to throw any ideas out there of course [19:48:32] ok. I can withhold that judgement then, and toe the line that we have to discuss more [19:48:35] I mean, now's the time [19:48:44] yah szyzygy [19:48:54] might as well take advantage of the fact that it's on the table today [19:49:29] definitely <<12 min of time though [19:51:57] sounds like a legit question that it makes sense to clear up before we get too deep in any implementation of that RFC [19:52:10] Hmmm Silverpop isn't GPL [19:52:23] hehe [19:52:38] mercenaries! [19:53:26] heheh if it's not GPL it surely doesn't deserve to get DI [19:53:38] ejegg: u lack gerrit links for https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1861 [19:54:53] oh, will add that [19:55:41] (CR) Awight: [C: 2] "Just one typo to catch" (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/153385 (https://bugzilla.wikimedia.org/31689) (owner: AndyRussG) [19:55:51] (Merged) jenkins-bot: Add comments on changes, for logs and revisions [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/153385 (https://bugzilla.wikimedia.org/31689) (owner: AndyRussG) [20:00:01] awight: ejegg: should I submit a new change sans typo then? Or is it not worth it since it's just in the code for if the schema change is missing? [20:00:17] sorry about being slow on the typo-response [20:00:21] AndyRussG: meh a new patch perhaps [20:00:32] K one sec [20:00:47] Then I'll do a schema change bug report I guess [20:00:49] AndyRussG: It's pretty likely that people will hit your luxurious schema mismatch code, actually [20:00:56] this is used by a lot of third-parties [20:02:21] Really! interesting [20:02:58] awight: In that case should I make it log a warning or something? They'll be all, hmm, why doesn't it save the log messages? [20:04:44] The most hilarious was... sigh, it's offline it seems. But it was the LDS magazine using our banner code to fundraise above an article on why you can't trust Wikipedia. [20:04:54] Rich stuff! [20:05:12] AndyRussG: nah I wouldn't bother with a warning. But avoiding the hard fail was nice. [20:06:31] awight: lol, fantastic [20:11:12] (PS1) AndyRussG: Fix a typo in code that handles an old db schema [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/153882 [20:11:55] ejegg: what is Civi storing under activity "details"? [20:12:04] I see you're passing in the letter's html... [20:12:19] yeah, that's what it stores for individual mailings [20:12:39] It's easily viewable from the contact's activity records [20:12:47] ejegg: it's a "text" field, so... the whole thing up to 65k I guess? [20:12:50] and I figure it's nice to have the exact text [20:12:58] unless we're worried about space [20:13:06] That's really cool, but I'm a bit worried about storage. yeah have u estimated? [20:13:18] nope, let me see [20:13:21] k thx [20:16:08] ejegg: for reference, civicrm_contribution is currently the biggest table, at 1.3G [20:16:20] ok. templates are around 4k each. [20:16:30] just trying to get a count on ty letters per year [20:18:19] Jeff_Green: ^^ Anything we should know about adding a ton of data using text columns? [20:19:16] uh [20:19:19] more info I guess? [20:19:44] 4k text per ty letter (per contribution, pretty much) [20:19:46] ejegg: I was considering a slightly different approach when I wrote the stupid wmf_communication tables, my idea was to store each template once, but save template params for every letter sent out [20:19:51] not a huge win [20:20:00] ahh, could do [20:20:11] Jeff_Green: we're talking about saving every letter's contents in the civicrm_activity table [20:20:23] every letter ever? why? [20:20:25] I just don't know much about the pitfalls of text fields. [20:20:33] cos... it would be really luxurious for admins [20:20:45] It's what CiviMail does with other individually-sent mails [20:20:54] i see [20:20:57] For easy reference from a contact's activity list [20:21:01] how long do we expect that to scale? [20:21:19] err, we would probably expire the data at some point. donno. [20:21:43] donations counts are going up exponentially... hopefully we can put an end to that :p [20:21:55] ha [20:22:29] is this a native civicrm table? or something we're creating? [20:22:49] native one [20:22:50] wow, 2.6 million contributions just in the past year? [20:23:00] yeah, it's... growing [20:23:22] awight: it's tempting to explore table compression, but I haven't worked with it since the myisam days [20:23:32] so that's like 10GB / year [20:23:48] but it would be highly compressable! [20:24:00] that's true [20:24:07] do we ever intend to search that field? or just grab rows by key? [20:24:13] just grab [20:24:18] no fulltext nothing [20:24:25] correct [20:24:51] The table we'd be putting the data into (civicrm_activity) gets a lot of query action, fwiw [20:25:29] oh, so that would slow down some things... [20:26:01] shouldn't be a problem really [20:26:11] because the speed is all from the indexing [20:26:18] ah, i see, it's already got an activity for each contribution [20:27:24] ccogdill: not to ask a leading question, but is it true that having a log of actual emails sent to donors would be amazingly helpful? [20:27:41] yeah it would be awesome [20:27:52] I believe meganhernandez and the_wub would love it too [20:27:55] and jessicarobell [20:27:56] ejegg: Either way, we should talk to K4-713 cos she's been doing a lot of cogitating about data retention. [20:28:02] For bulk silverpop mails, I would still just store a pointer to the template [20:28:04] ccogdill: rad, thx for the input [20:28:14] it's just with the bulk mailing activity you only have a single datestamp [20:28:21] ejegg: sure--that could even be a local URL? [20:28:42] heh, or a URL to silverpop's displayTemplate action [20:28:54] currently, I copy + paste the url to the each email iteration in silverpop and put it on our testing calendar in gcal cause we don’t have a better place for it. so yes, makes my life easier! [20:29:24] It makes some sense to copy the template contents over to Civi as well, cos we don't all have access to Silverpop. [20:30:11] ejegg: for that matter, it would be awesome to include a link to the meta template for TY letters... [20:30:15] including the oldid... [20:30:23] Unfortunately with Silverpop's API, there's no exporting text of mailings that aren't saved as a template, so i really do have to screescrape [20:30:29] bblaaargh [20:30:36] awight: ahh, it will have the info comment! [20:30:51] yes, which we might dress up a bit... [20:31:21] I'll merge this patch for now, but u should talk to K4-713 before we deploy... about data retention. [20:31:40] yeah, JSON with source URL would be a good plan [20:33:07] I'm not sure. Does Civi already have a mechanism for decoding activity details based on type? [20:33:11] DI!!! [20:33:44] Oh, you mean an actual link to the mailing template in Civi. [20:33:46] Let me check [20:33:55] erm stuff like that [20:34:03] it does have a source entity ID column [20:34:19] which I believe points to the bulk mailing in bulk email activities [20:34:25] I do really like storing the actual TY though, so we can debug. [20:34:43] so yeah, i'm pretty sure it presents different activity types differently [20:35:04] Yeah I'm thinking about activity_type_id being used for DI [20:36:05] (CR) Awight: Add activity records for individual thank you emails (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/152208 (owner: Ejegg) [20:36:37] (CR) Awight: "We should have a conversation with the data retention expert, khorn." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/152208 (owner: Ejegg) [20:36:48] woohoo! [20:37:08] yeah, bulk email records show up very differently from individual email records. [20:37:59] even just in the activity list table [20:39:38] interesting, the details column in the shared bulk email activity record has a truncated message and a link to pop up the full template from the mailing table [20:39:50] Great. So we can extend that if we need to. Hrm, that's reasonable [20:40:05] (CR) Awight: [C: 2] Tests for CiviMail helpers (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/153426 (owner: Ejegg) [20:46:42] (CR) Awight: [C: 2] Add dates to banner-hiding cookies (server side) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/144991 (owner: Ejegg) [20:46:49] (PS7) Awight: Add dates to banner-hiding cookies (server side) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/144991 (owner: Ejegg) [20:47:25] ejegg: Are you comfortable with ^^ or would rather write tests before deployment? [20:48:13] ejegg: If you think it's ready to go, pls schedule us a deployment, maybe Monday? https://wikitech.wikimedia.org/wiki/Deployments [20:48:21] hmm, i should at least do some more local testing [20:48:59] do we want to roll this at the same time as AnyRussG's logging update? [20:49:06] sure! Seems like a tough one to test... [20:49:16] ejegg: yeah that makes sense [20:49:42] well, I can at least check that direct requests result in the headers I expect [20:49:47] ejegg: err, actually, we should get that schema change out first [20:50:01] ok [20:50:18] ejegg: awight: Did the banner filter thing already get deployed? How do I check that, btw? [20:50:36] ejegg: yeah the only thing I have a vague tingling sensation about is that cached mobile people can close banners [20:51:57] AndyRussG: no I don't think so [20:51:57] AndyRussG: but u'd check here: http://meta.wikimedia.org/wiki/Special:Version [20:53:40] ok, will locally revert the js and point my phone at my computer [20:55:42] awight: The branch to watch is wmf_deploy? Not used to not having a deploy train piloted by ops... [20:57:08] AndyRussG: no, we're in charge of CN deployments [20:57:38] Generally, Mingle should be up-to-date about what has been deployed [20:57:43] Ah hmmm [20:58:11] So when we're ready, we merge to wmf_deploy, then either wait for the train, schedule a SWAT deploy, or do it ourselves. [20:58:43] I would only suggest that we do the deployment in cases like this hide cookie patch, where we want to do a bunch of sanity testing, and revert quickly in case of stuff. [20:59:47] oh dang, that javascript won't get its wgNoticeCookieShortExpiry global [20:59:53] atgo: AndyRussG: ejegg: ok only a few seconds now, until AndyRussG volunteers to throw himself into the briars on #wikimedia-office :p [20:59:54] oops, and likely hasn't been... [21:01:04] well, the good news is there haven't been any more old-style requests caught in the log filter since those 2 on 8/6 [21:01:13] oh... heh ok then [21:01:26] feel free to make a note of that lag time somewhere on wikitech... [21:01:28] ack, wait, no [21:01:43] we left that one in the main config. [21:01:52] let me see that we're actually passing it to js still [21:01:56] lol welcome to CentralNotice [21:03:39] Didn't sumanah say something about how she was no longer chairing the RFC meetings... [21:03:47] I think she did [21:04:09] That *might* have bearing on the total silence [21:04:39] well... old habits... [21:07:21] hey awight, can I bother you with one more thing today? [21:08:03] ccogdill: for sure! [21:08:15] Yesterday I made a list of all active DD recurring donors [21:08:19] or at least I think I did [21:08:27] can you check my query and make sure I’m not missing people? [21:08:31] see mingle #1871 [21:08:32] https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1871 [21:08:44] I just need a little QA :) [21:09:29] ccogdill: and, apologies for being a jerk: re the thank-you issue! I was just hoping to postpone the work and explanations for a minute, but cat's already out of the bag ! [21:09:33] ok sure I can QA that [21:09:51] no problem! I can appreciate how I was making your life a little more difficult [21:09:59] drat. we really haven't been passing shortExpiry into the js globals since the deployment [21:10:08] ejegg: good catch [21:10:18] ejegg: what was the impact? [21:10:33] i need to figure that out [21:11:02] ccogdill: that looks like a solid query, I'll double-check in mysql [21:11:12] awsome, thank you [21:11:55] fwiw, I’m planning on taking some mysql classes soon [21:12:52] ccogdill: woot! I've been thinking that would make a great brownbag topic, also [21:13:27] I know I’m not the only one who could use some help [21:13:29] so yes! [21:14:24] hi [21:15:09] awight: [21:15:20] eileen: Hey, Eileen! Glad you survived and everything... [21:15:33] yeah - was a bit miserable [21:15:34] I lost a battle to simple cold, myself, but not as grave [21:15:44] my Mum is to blame [21:15:59] she got my kids into 'Andrew's twirl' basically eggnog [21:16:03] oooh dirty [21:16:04] & it involves raw eggs [21:17:05] salmonella [21:17:48] on the bright side I managed to blog about what I have learnt / done so far with Omnipay [21:17:51] oh god, sorry! [21:18:09] hehehe sounds somehow like an appropriate pasttime while ... suffering [21:18:41] also watched planet of the apes [21:18:44] (original) [21:20:05] https://civicrm.org/blogs/eileen/rethinking-payment-processing [21:21:16] so getting back to where I was before being run over by a chicken - I'll try to line up a time to talk with Tim - but just wondering when works for you - I see you have plans for a deployment on Monday [21:22:19] Oh, any time Monday is fine [21:22:43] The thing we're pushing out can be rescheduled, if need be [21:23:11] awight: what's going out monday? [21:23:25] well - I don't want to put you out - but yeah I'll suggest some time on Monday to him (my tuesday) [21:23:32] atgo: (maybe) the CentralNotice cookie thing [21:23:44] ahh... yeah that.. would be good to go out [21:24:15] eileen: the only point I noticed missing from your last email was that we'd like to make our existing payment processors truly open-source. [21:24:27] awight OK - cool [21:24:42] But I understand if you think this isn't high priority cos we are barely adding functionality to existing code [21:25:07] Perhaps our auditing code is worth integration though... [21:25:20] It seems that nobody in their right mind does such a thing yet? [21:25:33] I didn't leave it off on purpose as such [21:26:17] At the moment CiviCRM has some or your processors (well one in core & one sort of as an extension) [21:26:34] but global collect is no-where on it's horizon [21:26:37] eileen: we should probably have a meeting to organize WMF's priorities around the work you bring up. Not sure if it would be helpful to you, but you are welcome to attend! [21:26:48] sure [21:26:59] let me know when [21:27:16] eileen: yeah GlobalCollect is a processor aggregator, same with Adyen and WorldPay [21:27:28] eileen: will do [21:27:33] sorry re global collect - CiviCRM wouldn't be able to integrate global collect without having much better form builder capability [21:27:45] That sounds about right [21:28:02] It's probably the main reason our form generation is so screwy [21:28:03] Worldpay (as far as I understand so far) only presents one form variant [21:28:27] I don't know, actually [21:28:37] yeah - I'm trying to see the form generator as a standalone component [21:28:46] I think we're only using them for one payment method so far, so reading our code wouldn't give an answer [21:28:55] ah [21:28:56] eileen: great! I'd love to see that. [21:29:45] yeah - it seems like a pretty big task - I guess my rambling about metadata is basically 'what does the host app need aside from a slab of html to stick into it's form' [21:30:00] eileen: Hey, you must have run into this issue when writing CiviMigrate-- do you know of any libraries for data transformation, or decent standards around it? [21:30:41] I found something... I think it was by the OMG (stewards of UML, etc), but all it demonstrated for me was that there are a lot of edge cases. [21:31:11] not too much - in CiviMigrate I wrote some mappings for all the various ways one can spell 'Mr' [21:31:17] Their structure was to have a simple declarative core language for transformations, then a higher-level abstraction, and then a language for all the exceptions ;) [21:31:20] baahaha [21:31:36] That sounds like a full-time job! [21:32:15] eileen: Here are embarrassing notes on this subject, https://wikitech.wikimedia.org/wiki/Fundraising/tech/Transformer [21:32:22] Let alone Christchurch (hint ch & chr are inheritently mis-typable) [21:32:28] hehehe [21:33:05] My hunch is that we'd benefit from isolating all data transformation stuff in payment gateways into its own thing, then providing a component that would execute the transformations [21:33:14] Young-Jin always talks about Google Transform - but I think it's more a way of letting end users build up their rules [21:33:27] yeah, I saw that prez in 2012 [21:34:00] We have this concept of normalized donation message... I'm not sure it's useful... [21:34:01] the transforms you are thinking of are basically tidying up user entered data (what's my name type stuff?) [21:34:12] ? [21:34:22] normalized donation message as in? [21:34:30] no, it's stuff like moving data from the UI field names to normalized field names, then to API field names [21:34:37] with a little massaging in-between [21:35:07] awight: AndyRussG ejegg standup? [21:35:08] eileen: here's another nasty summary, of our normalized format, https://wikitech.wikimedia.org/wiki/Fundraising/Queue_messages [21:35:10] we're waiting for you [21:35:32] argh ok sorry eileen, this will take ~15 min. no worries if you have to take off! [21:35:37] One sec web cam hissues [21:35:41] right - your UI field names are often provided by global collect [21:35:57] no worries [22:01:38] eileen: whew! or half an our... [22:02:11] eileen: yeah the incoming data might come from the UI, or a listener, or audit scripts... [22:03:08] Does it make sense that we're using this intermediate "normalized" form, or would you have used civi contributions / contacts directly? [22:07:48] so you are normallising between stuff coming from the payment processor & CiviCRM [22:07:58] yah... [22:08:07] there are a lot of different intake paths [22:08:13] we serialize everything over the queue [22:08:18] yeah - it definitely makes sense to normalise! [22:08:42] We inherited this all, it's hard to get perspective on legacy assumptions... [22:08:50] oops I gotta tend baby for a minute [22:15:13] weird. she's okay just sitting in the stroller for a second. [22:15:24] atgo_: you want to discuss Phase 2 motivations? [22:15:48] is this the cucumber email? [22:15:53] yeah [22:15:59] alright let's do it [22:16:05] I think the sentences in italics are all motivation for the features [22:16:10] and they need a lot of work [22:16:19] hangouts? [22:16:30] yeah that's what shew as after [22:16:36] one sec. forgot my headset need to finda room [22:17:41] ok.. i'll call you awight [22:18:51] atgo: I'm on as phone and laptop, so mebbe just paste me the URL? [22:18:56] usually the phone takes precedence [22:19:16] yep that is happening [22:20:33] (PS8) Ejegg: Add dates to banner-hiding cookies (server side) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/144991 [22:21:22] awight: awight: took the special case for old JS back out ^^^ [22:21:46] derp. that was definitely addressed to awight [22:22:00] teehee [22:57:50] awight|afk: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1872 if you get a moment... rosie needs sooner rather than later [23:00:09] atgo: yes... I think the first priority is to get recurring GC back up though? [23:00:21] yep [23:00:31] I think I only have an hour left in the day... [23:00:38] yeah do what you have to. let's look at it friday [23:00:44] is there a chance GR could look at this one or not worth it? [23:00:52] not worth the start-up costs [23:01:14] Rosie can currently search by donor's campaign, I hope? [23:01:15] ok [23:01:27] not sure [23:01:30] they haven't sent the appeal yet [23:01:37] mebbe ejegg wants to look? [23:01:41] just the save the date [23:01:44] hehehe [23:01:57] missing campaign? I can try to figure it out [23:02:06] no guarantees! [23:02:14] ejegg: yeah it looks almost like a caching thing [23:02:37] or, the civi api must not be returning some campaigns? [23:02:42] oh, so maybe just drush cc all [23:02:46] I did try [23:02:48] ok. ejegg if this is too much start up cost for you, then let's wait until awight|afk has time on friday [23:02:56] plus, there really shouldn't be caching here [23:02:58] i'll spend a few minutes on it [23:03:02] thx! [23:03:04] cool [23:03:23] i've got to run (oy... today). but see you guys later/tomorrow [23:04:07] ok, see ya atgo [23:23:51] eileen: sorry it's been such a mess on our end! Here's some "light" documentation about our normalized format, https://wikitech.wikimedia.org/wiki/Fundraising/Queue_messages [23:26:40] (PS1) Awight: Revert "Transaction around recurring Globalcollect charge" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/153974 [23:26:52] (CR) Awight: [C: 2] Revert "Transaction around recurring Globalcollect charge" [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/153974 (owner: Awight) [23:48:49] (Abandoned) Awight: WIP remove synchronous thank-you functionality [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/151030 (owner: Awight) [23:53:40] blargh! The recurring globalcollect test claims to pass, unless I create a second, empty test function. [23:58:31] awight: great - some more light reading :-) [23:59:46] In this case, it really is light... there's almost no documentation! [23:59:49] :(