[14:53:31] jessicarobell: the-wub: jfyi, there is a fix for the automatic translation state changes which should go out in the next hour. I think I can test it sufficiently myself... I'll email to let you know how it goes! [14:54:30] ok! great awight. thanks! Let me know if you need any help. [14:54:37] great, thanks awight! [15:12:39] jessicarobell: the-wub: ok the fix is deployed. Looks like it worked, even! [15:13:24] I edited and reverted a paragraph in the sv TY translation, and the status was knocked back to "proofreading". [15:16:25] It did indeed! :) Thank you very much awight. Just tested it myself too. [15:16:38] wohoo! [15:18:04] jessicarobell: Finally! So, this should be the workflow we'd actually planned for. Holler if there are things we can improve now. [15:18:27] I will awight. Thank you!! [15:18:55] I'm hoping to drag the-wub into the fray, to help prepare the patches. [15:19:08] the-wub: if that sounds good? [15:19:40] the-wub: All you should need to do is set up a working CRM instance. Or do you have one already? [15:21:15] the-wub: the only setting that will require manual jiggling is the Twig directory, I can help with that any time. [15:22:26] the-wub: alternatively, u can use vagrant, which should give you a CRM instance, hands-free. [15:49:59] thanks awight. I have a vagrant installation now, and I hope to find time to get stuck into translations later this week [16:56:09] ejegg: hi... I have an unsurprise 4 u ;) [16:56:23] Would you like to have a bunch of CR? [16:56:48] Only the three "fixed "- patches I emailed out are critical... [16:57:25] hi awight [16:57:36] would love to look at some code! [16:58:25] I just updated card #47 with some possibilities for bounce handling if you want to weigh in. [16:58:27] hehe <_> >_> [16:58:33] oh excellent, will do [16:59:12] the-wub: Sorry I dropped off earlier, but I wanted to get a read from u on enthusiasm about helping prepare future TY translation patches. [17:02:09] ejegg: I like the first option the best. [17:02:24] hey awight, sorry I was afk. I have a vagrant setup now, and I'm hoping to get stuck into translations at some point this week [17:02:31] the-wub: wicked, thank you. [17:02:41] the-wub: just holler if I can help in any way. [17:03:26] the-wub: fwiw, https://collab.wikimedia.org/wiki/Fundraising/Engineering#Generate_new_Thank_You_email_templates [17:03:45] yes, seen that thank you :) [17:04:27] ejegg: the 4.4 upgrade involves moderate tech work, which is half-done, but we're still waiting for the quality to stabilize. [17:04:35] * awight checks in with #civicrm [17:05:55] oh, ok. I thought the delay was all on our side. [17:06:24] ejegg: it was, until I looked more closely at their release roadmap and accompanying rumors. [17:06:37] ooh, what kind of issues? [17:07:02] ejegg: I'm trying to figure that out, but afaict they are waiting to close all the "critical"-level bugs. [17:07:08] That sounds like a reasonable plan :D [17:07:14] hmm [17:07:36] (CR) Ejegg: [C: 2] "Hooray for dedupe" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/149777 (owner: Awight) [17:11:34] ejegg: meh, this is probably only the tip of the iceberg for blockers to Civi LTS: http://tinyurl.com/p89kw93 [17:12:14] oh, that's bad [17:14:06] ejegg: what's the deal with the Civi mailing API? I want to eventually merge our thank-you functionality into Civi, I was hoping it would simply be abstracting the template interface. [17:14:22] they basically didn't have one till 4.4 [17:15:32] Yeah, that rules out a civi "extension" for TY, which is probably the most correct way to integrate. [17:15:42] We could still do the work in Civi core, though. [17:15:46] For now anyway [17:15:49] As long as we have that templating abstraction. [17:18:22] Yeah, I guess so. If we're not using the API and messing about in core, I might just want to hack up the bounce processing [17:22:57] ejegg: that's definitely an option, but like you pointed out: not upstreamable [17:23:20] which is a pretty big issue, there've been many Civi upgrades with massive whitespace "cleanup" :( [17:24:46] Oh, for the sending you were hoping to just call core methods, not actually change core core. [17:25:00] ejegg: no, I did want to change core [17:25:13] Whatever TY functionality is missing from core... should exist [17:25:29] ah, good point [17:25:29] and we should be able to use either the Smarty garbage or our Twig garbage [17:26:03] I have a few attempted patches laying around which do this, actually, and now that Civi is headed towards Symfony2 there is a good chance of upstreaming. [17:26:12] and swap out things like unsubscribe link generation [17:26:26] hmm maybe not that one [17:26:38] cos we need to keep randos off of the CRM box [17:26:59] right, i mean civi would need to let us generate our own [17:27:34] I think we would be in full control of the templates, still [17:27:37] so templating logic + some token content calculation [17:27:44] yep [17:28:01] but... this might be a ton of work. Lemme poke around for a minute. [17:29:13] ejegg: Civi already has a "contribution receipt" workflow, which is all we need AFAICT [17:30:50] checking how it is fired off, I'm afraid it might be coupled to the "contribution pages" [17:31:27] oh, definitely a good place to hook it on [17:31:57] ejegg: yep, receipts are tightly coupled to contribution pages, which we aren't using. [17:32:09] i mean, not the page coupling, just that's the place to put it. [17:32:22] any different in 4.4? [17:34:12] ejegg: no, but it's no so bad: just CRM_Contribute_Form_AdditionalInfo::emailReceipt [17:34:24] Huh, they do have a recurring contibution receipt in 4.4, which would have to fire while the user's not on the site [17:34:27] We could do that from our import logic, I think that's fair [17:34:37] oh great point. /me looks [17:36:05] ejegg: yeah still not code we can hook into, but that static function I pasted just takes two params, will be easy to reuse. [17:36:47] but we'd still be waiting on 4.4 [17:37:15] ejegg: nah it's the same in 4.2. But, gross. Now I see that one param is a quickform object. [17:37:37] oh boy, display logic [17:38:14] ejegg: yep, and worse. So... I guess the highest-level thing we can use is CRM_Core_BAO_MessageTemplate::sendTemplate [17:38:41] which is not a huge win, it just causes us a bunch of template integration for basically no reason. sigh. [17:39:32] ejegg: how involved would you say it would be to reimplement bounce handling? [17:40:17] You said something about "categorize bounces", is that about hard vs soft, etc? [17:40:41] I like the idea of insulating Civi from incoming emails, actually. [17:40:42] yeah, exactly [17:41:04] too bad. thresholds and stuff... [17:41:39] Right, there's a bunch of logic in there for all the different mail server bounce replies [17:41:49] oof [17:42:07] well... we could write the Civi API for generic bounce processing? [17:42:46] Basically, your third option, but also writing the API. [17:43:09] actually, I think bounce processing is one of the few things in the 4.2 api [17:43:23] it's just tied to their VERP format [17:43:54] aha. array('job_id', 'event_queue_id', 'hash', 'body'), [17:43:54] I believe they are marking specific messages as bounced [17:44:08] yeah that is a terrible interface for us [17:44:52] You might want to discuss this on #civicrm... [17:45:06] Oh, good idea [17:45:22] BouncePattern does look like it's meant to cover our use case [17:45:37] yeah, the patterns and categorization is fine [17:46:20] But I assume that would not be enuf to teach Civi what we need? We would also need to replace the associations with jobs and mailings. [17:46:49] hmm, let me take another look [17:48:22] ejegg: omg have you looked in civicrm_mailing_bounce_pattern ? [17:48:53] uh, not much there [17:49:09] really? I see 124 regexen [17:49:19] ohhh, wrong file [17:49:34] ya in the db [17:50:14] apparently those are run against the message body, maybe including headers. [17:50:25] Seems incredibly fragile and English-dependent [17:50:35] yeah, that's some serious tribal lore [17:50:53] Each for a different mail server and situation [17:50:54] Maybe we should pipe to Trilogy... [17:51:00] ugh [17:51:12] Well, it's certainly not somthing I want to re-implement! [17:51:52] omg the string "not connected" anywhere in the message maps to "Unable to deliver to destintation mail server" [17:51:55] this is not good [17:52:09] [sic] [17:52:20] If we just break out the case: 'bounce' from Utils_Mail_EmailProcessor::_process, we can reuse the logic, flawed though it may be [17:52:24] hehe what if "not conected" is misspelled [17:52:39] destintation, huh? [17:53:30] I think we need to say an audit of all code paths touched by email processing is mandatory before we can turn this on, btw... [17:54:01] i wonder what kind of rules Google's ML algorithms have come up with for this categorization [17:54:35] I would love if we had a real MTA put some X- headers on incoming bounces that categorized for us [17:54:44] Jeff_Green: ^^ is that a thing? [17:54:55] Jeff_Green: we're noticing that the [17:55:04] that the Civi bounce processing is made of toothpicks [17:55:18] bah, that's not the bit we need [17:55:23] oh? [17:55:52] nvm [17:56:03] hehe that's how I feel [17:56:10] I wonder why this wasn't done years ago :p [17:56:32] awight: bounce mail makes more sense when you accept the fact that SMTP is made of toothpicks [17:56:53] Jeff_Green: yes but... ones that have been firmly glued to the sand [17:57:01] yeah, there's no actual standard for formatting bounce returns [17:57:02] civi bounce processing is probably crappy because there isn't really a concept of standardization in mail [17:57:30] ya exactly [17:57:39] Jeff_Green: is there a thing we can half-trust to feed us good guesses though? [17:57:58] well...no. :-) [17:58:01] I would definitely take Civi's categorizations over trying to write our own [17:58:01] I'm especially crept-out by the English-only regexes [17:58:08] ejegg: no question about it! [17:58:17] awight that is the reality of bounce processing [17:58:25] Like u say, those have been passed down around the campfire for generations [17:58:44] Yeah, those are based on what's out there in the default mail server configs around the world [17:59:40] Man I've really been under this convenient misassumption that anyone responding to the +bounce address must be bouncing, else it would be the other reply-to header. [18:05:25] so we can hook into categorization easily just with BouncePattern::match. Thresholds would mean adding records for each TY to at least the email table and the queue table. [18:07:00] ejegg: thresholds should be attached to individual messages, or to the address record? [18:13:35] ejegg|away: another factor to consider: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1842 [18:32:58] atgomez: I'm in the hangout... [18:33:08] K4-713 coming! [18:37:56] (CR) Awight: [C: 2] "Thanks!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/147705 (owner: PleaseStand) [18:37:59] awight: thresholds are per email address, not per message. Could just add a bounces count column to the email address table to track non-civi stuff. [18:38:09] (Merged) jenkins-bot: Remove usage of deprecated Xml::escapeJsString [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/147705 (owner: PleaseStand) [18:38:37] ejegg: want to video chat for a minute? [18:38:44] sure, one sec [18:40:29] ejegg: heading to a room... [18:40:45] ok, cool [18:47:00] atgomez: You're frozen and appear to be in pain. [18:47:55] And now you're gone. [18:47:55] K4-713 my laptop hates the sun [18:48:01] I get that. [18:48:05] I kind of do too. [18:48:13] ...unless I'm in the sea. [18:48:25] I don't think laptops like being in the sea, though. [18:48:36] i'll ask it [18:48:40] i'm back..? [18:48:47] Maybe? [19:55:35] hey awight ... question about where things are at with worldpay. [19:55:46] atgomez: yessir? [19:55:58] K4-713 and i seem to recall that we ran in belgium as the "it's so technically close to france that it should surface prolems" test [19:56:22] The auditing is still borken. We currently have 8 records from Belgium which came in through audit, and they are anonymous because of it. [19:56:30] hmmmmmmmmmm [19:56:30] Seems like a pretty big bug. [19:56:32] :( [19:56:46] yeah that's obviously not the best [19:56:57] but the ones that come in realtime work, right? [19:57:03] it's not like we got only 8 donations.... [19:57:07] atgo: Matt wrote this up on the 7th https://collab.wikimedia.org/wiki/Fundraising/Engineering/WorldPay_Handoff [19:57:08] Yeah the others work. [19:57:53] atgomez: yeah looking at those bugs it seems to be in bad shape. [19:58:07] yeah ejegg i wrote that, too :) [19:58:22] ok.. that makes sense [19:58:25] oh, gotcha. Was very informative to me at least! [19:58:26] are those bugs reported somewhere in mingle? [19:58:30] If there is a high volume of donations, we'll be flooded with issues we cannot handle because the info is missing. [19:58:42] atgomez: the audit bugs? yeah [19:58:42] * K4-713 opens mouth, closes mouth [19:59:00] i don't see this audit issue in the collab page [19:59:08] thanks ejegg for sharing though! [19:59:15] I was about to say "The first year I was here" in the style of "In *my* day" statements from old people... [19:59:24] It's been lurking the current Mingle sprints forever [19:59:25] hahaha K4-713 [19:59:31] ...and then realized that we should hella do better than that, forever. [19:59:38] https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1151 [19:59:53] https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1663 [20:00:09] ok. thanks [20:02:32] atgomez: I just created another card about something I consider a bug by design: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1846 [20:03:04] ok. we should discuss this one more and decide if it's a blocker for WP france the way the rest of these seem to be [20:03:11] here's my list for WP france: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/1821 [20:03:32] yep. [20:14:21] atgomez: fwiw, 1.2% of WorldPay donations have come in through auditing. [20:17:08] awight: ...Say. You know things about translations, right? [20:17:21] Like, way more than I do at this point. [20:17:50] K4-713: just... what's the question [20:18:16] I want to commit new labels and *not* cause people to waste time translating the dummy text in them. [20:18:27] WIP stuff. [20:18:39] oh hah [20:18:41] don't do it [20:18:46] Goddamnit. [20:18:57] maybe there's a stupid {{{identical}}} tag lemme see [20:19:02] This is a Problem. [20:19:07] ... [20:19:38] K4-713: https://translatewiki.net/wiki/Category:Identical_message_templates [20:20:01] Okay, #1 - I'm genuinely not sure what this gets me. [20:20:25] #2 - I hate that you don't have to pass any kind of CR to trigger translation action. [20:20:31] K4-713: I'm pretty sure if you said something like {{{Identical:Not available}}}, nobody translates. [20:20:46] Not Available, huh. [20:20:51] and it also turns into a string which almost makes sense for humans [20:21:04] yeah the identical thing: https://translatewiki.net/wiki/Template:Identical [20:21:17] means that there is already a translation, so don't waste time on retranslating [20:21:30] it's usually for things like {{{Identical:MediaWiki extension}}} ya know [20:21:37] huh [20:21:37] that are used across lots of projects [20:21:50] I feel like this would cause some kind of "computer missing" error. [20:22:04] beeehehe [20:22:11] Identical to NOTHING. So it's okay. [20:22:49] Guh. I would rather do something really stupid like reference the total wrong label until the real text gets approved. [20:23:27] K4-713: creativity will not be rewarded [20:23:45] No, but there's a chance that it will amuse me. [20:24:07] * K4-713 starts looking for obviously wrong label contents [20:24:32] K4-713: well I still recommend Identical cos translators do not translate [20:24:43] but there are lots of arguments to choose from [20:24:46] Where do you even put that? [20:24:53] * awight looking for an example [20:24:58] just as the message contents [20:25:48] So... set it as identical to something else we have that is stupid? [20:26:07] And, do I have to do something also stupid to qqq? [20:26:13] it has to be one of the Identicals that already exists in that list I pasted above [20:26:42] I really don't like that I can't find a usage. [20:26:50] I don't like that either. :D [20:27:45] K4-713: it must be deprecated and not documented as such. [20:28:03] Oh well. Thanks for playing, anyway. [20:28:07] I would say eff it, use just "Not available" and let translation memory do the heavy lifting [20:32:42] (CR) Ejegg: [C: 2] Fix and clean up recurring globalcollect [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/149650 (owner: Awight) [20:44:12] (CR) Ejegg: [C: 2] Transaction around recurring Globalcollect charge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/149613 (owner: Awight) [20:49:08] (PS3) Awight: WIP script to backfill missing recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/146364 [20:50:30] (CR) Ejegg: [C: 2] fix typo in recurring import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/149512 (owner: Awight) [20:52:59] (PS4) Awight: WIP script to backfill missing recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/146364 [20:53:01] (PS1) Awight: Split RGC message creation into its own function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/150038 [20:54:18] awight: that 'fix weak fail' throws the exception in order to work with the new transactional wrapping, correct? [20:55:27] ejegg: yeah it looks like that error was being suppressed previously [20:56:31] I think it would have taken a second logic error to trigger this bug, but... I can imagine that happening. [20:56:41] AndyRussG: bonjour! [20:58:20] (CR) Ejegg: [C: 2] fix weak fail on missing initial contribution [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/149614 (owner: Awight) [20:59:46] awight: Hey ho! [20:59:48] How's it going? [21:00:51] Great, I thought I should refactor one of our twitchiest components as everyone leaves for London ;) [21:01:19] don't worry, hte new guy is reviewing the changes. [21:01:30] what could possibly go wrong? [21:02:59] Luckily, the boss is not getting a SIM card for England, so we shouldn't have too much interference :D [21:03:14] heh [21:03:30] I do have a mifi, though. [21:03:37] * K4-713 wanders off again [21:04:00] I'm not worried. That thing probably doesn't have a sim card either. [21:17:31] ejegg: wow, thanks for the CR! [21:17:53] still a lot out there... [21:18:12] those were the blockers tho, the other stuff can wait til whenever [21:18:12] just looking at the 'phpunit tests pass' actually [21:18:43] (PS5) Awight: WIP script to backfill missing recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/146364 [21:19:40] and failing to run phpunit... Mind letting me know from where & what params you use to run it on drupal modules? [21:21:22] oooh [21:21:34] ejegg: well, go to the crm/ directory and run "phpunit" [21:21:55] if it doesn't work, hopefully that's due to missing config [21:22:28] You'll need at least: [21:22:40] drush vset wmf_common_twig_location "/yr/lib/twig/current" [21:23:28] drush vset standalone_globalcollect_adapter_path "/..../tools/payment_library/" [21:23:54] drush vset wmf_common_di_location ....DonationInterface [21:24:04] whew I think that's it for the tests we have so far [21:24:16] thanks! [21:26:46] (PS1) Awight: Merge remote-tracking branch 'origin/master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/150043 [21:27:36] K4-713: ejegg: fyi, that's the deployment I'm anxious about ^^ [21:27:51] awight: Er. Now? [21:27:58] My plan is, after meeting fest, to turn the QC and RGC down to 1, then attempt deployment [21:28:05] maybe 4pm [21:28:06] So: Now. [21:28:09] ha [21:28:11] er sort of [21:28:20] Maybe I wait until planes take off for London? [21:28:27] Ireland. [21:28:34] oh even better [21:28:39] ...schedule for 11am on Wednesday. [21:28:43] :D [21:28:45] offline and drunk [21:29:21] Yeah rollback looks hairy cos it's a merge commit, I'll just use jeff's deployment revlocking to back out. [21:29:40] Totally sure that won't be necessary, though :D [21:31:18] K4-713: actually, Wednesday AM would be reasonable... I just need the changes to go out by Friday, so we can accept the Coinbase audit. [21:31:31] mmmmmmmmmmhm. [21:31:32] K4-713: whatchu think? Probably doesn't matter to you directly? [21:31:38] I... [21:31:55] You're either under suitcases or under a cask of whiskey [21:32:04] :p jealous [21:32:11] Yeah, I don't know that I favor one way over another in a measurable way. [21:32:24] It's so close that I functionally don't care. [22:59:22] awight: all passing but one - ImportMessageTest is blowing up in setUp when it tries to create the contact. Somehow the globals aren't all set up and the api dies when it tries to switch to the funky TemporaryErrorScope. Is that test working for you? [23:00:56] sorry, you're probably neck-deep in that queue consumer deployment [23:05:31] ejegg: that's ringing a muffled bell... i had that happen.... [23:06:06] that reminds me, There are some drupal core patches I had to apply, which makes me nervous [23:06:18] wait, when? [23:06:22] ejegg: but, can you find the original error? [23:06:39] you might have to look in sites/default/files/civicrm/ConfigAndLog [23:06:57] the api always sets up the temp error scope before it does anything [23:07:02] ejegg: https://gerrit.wikimedia.org/r/#/c/149655/ [23:07:47] (CR) Awight: [C: 2] Merge remote-tracking branch 'origin/master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/150043 (owner: Awight) [23:08:41] hrm, yeah. Missing $GLOBALS [23:09:44] Looks like PHPUnit is pretty new in Drupal - like maybe only Drupal 8 [23:09:47] ejegg: ugh, well yeah u can apply my drupal patch [23:09:53] it seems to do the job, for now. [23:10:20] might help, but I'm actually missing different globals. _PEAR_default_error_mode [23:10:35] ejegg: ah. that one... argh i patched that as well [23:10:38] but apparently did not push [23:11:36] so very odd. plenty of other places tests use the api, or at least the fns they call do. and the globals are fully populated everywhere else [23:12:27] basically, that civi class where those globals are accessed, I faked out by returning an array of the same properties but =>null [23:12:41] thanks for trying to run the tests! [23:13:13] hey, why write em if we're not gonna use em? [23:13:28] ejegg: exciting that we might eventually be able to use them! [23:13:50] Having a testing framework to target has been a big deal, as u can see by my commits [23:14:19] Originally I was writing for simpletest cos it was the Drupal 7 standard, but it's utter garbage and lacks cmdline support [23:14:25] on top of other failings [23:14:37] eeww [23:15:07] sure, deploy to a server and run the tests via wget. then you'll know if you broke it all [23:15:42] or maybe the server's down, or your network is flaky, or ... [23:15:43] baahaha [23:15:59] yeah the failings of that test framework were so many... [23:16:21] probably accounts for a lot of Drupal module quality issues [23:17:08] ahh [23:17:25] well, it's nice that they're officially switching for version 8 [23:17:38] I hope... I basically read this as a rumor and chose to believe it. [23:17:54] Found the story buried in a bug report about adding phpunit support... [23:17:59] aww, was all happy about the "this will be removed in 4.3/4.4" comment on TemporaryErrorScope. [23:18:08] hah [23:18:09] then I checked github [23:18:57] ejegg: OK I'm throwing the switch on the CRM code... [23:19:03] woohoo [23:32:43] Geeze, I wander off for a while and you... fix. Everything. [23:33:12] K4-713: nope. so much phail [23:33:22] or... you must have not been talking to me [23:33:40] oops [23:34:09] In that case... where's my helmet. [23:59:06] (PS1) Awight: Make it more likely that the recurring effort id will become part of the trxn_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/150091