[00:45:08] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 240 min ago [critical = 240] [00:50:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 245 min ago [critical = 240] [00:55:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 250 min ago [critical = 240] [01:00:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 255 min ago [critical = 240] [01:05:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 260 min ago [critical = 240] [01:10:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 265 min ago [critical = 240] [01:11:21] ok so issue with merging that one contact is a little perturbing - a circular trigger action - probably the cause of some failmail too [01:15:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 270 min ago [critical = 240] [01:20:10] PROBLEM - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 275 min ago [critical = 240] [01:21:04] ACKNOWLEDGEMENT - check_impression_logs on alnitak is CRITICAL: CRITICAL landingpage-impressions.+.json.log rotated 275 min ago [critical = 240] Casey Dentinger looking into it [01:43:55] circular trigger? One of our wmf_ ones? [01:44:00] eileen: ^^ [01:44:13] ejegg: it's ok [01:44:17] it was a run-away query [01:44:35] not quite sure why it got triggered but I think it was me... yesterday [01:45:03] maybe I tested the fishing net rule on a set of 50 or so contacts - not sure [01:48:41] ejegg: I'm going to push up a few more tweaks - were you working on that button or shall I just include that? [02:04:15] eileen: oh, I made a teeny patch for that [02:04:22] sorry, didn't see the ping [02:04:27] lemme just rebase it [02:04:33] ejegg: ok cool - do you want to add it in - I haven't pushed anything yet [02:04:53] (PS2) Ejegg: Hide 'mark non-duplicate' button until things are found [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513219 [02:05:01] ^^^ [02:05:05] yay [02:05:49] ejegg: hmm we should squash it in or possibly merge them both so we can keep working with more visibility [02:06:00] (since this won't affect any existing screens etc) [02:08:10] might as well squash it! [02:10:26] (CR) Eileen: [C: +2] Hide 'mark non-duplicate' button until things are found [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513219 (owner: Ejegg) [02:13:16] ejegg: ok I'll do that [02:14:49] squishy-squashed [02:14:55] (PS12) Eileen: Dedupe url generator & batch mark excluded. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/512996 (https://phabricator.wikimedia.org/T217909) [02:15:15] (Abandoned) Eileen: Hide 'mark non-duplicate' button until things are found [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513219 (owner: Ejegg) [02:58:11] (CR) Legoktm: [C: +2] Add phan [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/511109 (owner: Umherirrender) [03:02:23] (Merged) jenkins-bot: Add phan [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/511109 (owner: Umherirrender) [04:37:07] Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): Make 'emailExplain' variant default behavior - https://phabricator.wikimedia.org/T218730 (Ejegg) p:Triage→Normal a:Ejegg [04:42:25] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising Sprint It Came From Out of Scope, and 5 others: CRM contribution tracking writing should have a switch to use the new infrastruct... - https://phabricator.wikimedia.org/T215463 [05:03:51] (PS1) Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513255 [05:20:37] (CR) Eileen: [C: +2] CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513255 (owner: Eileen) [05:24:56] (Merged) jenkins-bot: CiviCRM submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513255 (owner: Eileen) [05:28:45] (PS13) Eileen: Dedupe url generator & batch mark excluded. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/512996 (https://phabricator.wikimedia.org/T217909) [05:28:47] (PS1) Eileen: WIP - clean up search controller [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/513256 [05:29:30] (PS14) Eileen: Dedupe url generator & batch mark excluded. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/512996 (https://phabricator.wikimedia.org/T217909) [07:29:45] (CR) Vedmaka Wakalaka: "This change is ready for review." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/497611 (https://phabricator.wikimedia.org/T208125) (owner: Vedmaka Wakalaka) [09:03:41] (PS13) Vedmaka Wakalaka: Banner in-page preview mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 [09:27:19] (CR) jerkins-bot: [V: -1] Banner in-page preview mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [09:32:11] (CR) Vedmaka Wakalaka: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [09:55:43] (CR) jerkins-bot: [V: -1] Banner in-page preview mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [13:01:45] (CR) Vedmaka Wakalaka: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [13:06:03] (CR) Vedmaka Wakalaka: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/497611 (https://phabricator.wikimedia.org/T208125) (owner: Vedmaka Wakalaka) [13:06:22] (CR) Vedmaka Wakalaka: "Hm... It looks like CentralAuth is failing with and without CentralNotice, is something wrong with it at the moment?" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [13:19:27] (PS1) Jgleeson: Remove unused commented out test from employer field variant patch. [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513282 (https://phabricator.wikimedia.org/T219558) [13:25:56] (CR) jerkins-bot: [V: -1] Banner in-page preview mode [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [13:30:07] (CR) jerkins-bot: [V: -1] Banner live preview [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/497611 (https://phabricator.wikimedia.org/T208125) (owner: Vedmaka Wakalaka) [13:49:07] Fundraising Sprint Hansel and grep -l, Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog: Opt-in numbers in IBM not as expected - https://phabricator.wikimedia.org/T219401 (Pcoombe) Jus... [15:08:41] hi folks! [15:15:25] hey ejegg :) [15:15:46] hi jgleeson! [15:17:27] are we waiting for dstrine on the employer field countries? [15:18:08] oh, i guess it only makes sense for places covered by the matching gift database [15:18:34] ejegg, it looks like I've missed some requirements... again [15:18:51] there's more UI work to be done I think [15:18:55] oh shoot! what else was there? [15:19:19] the explanation? [15:19:37] look's like they're four variants in total, https://phabricator.wikimedia.org/T219558#5116621 [15:20:03] three* [15:20:11] all differing by the emailExplain-type text they show [15:20:40] I'm just looking into using your emailExplain variant behaviour [15:21:59] there are* [15:26:59] ohhh [15:28:56] i totally missed that bit. [15:29:09] me too [15:29:53] so yeah, the only way to add different messages based on a variant would be that emailExplain js route [15:32:45] I'm working on that now [15:33:26] with the variant names would a numeric increment be a problem? it might result in people testing out the other possibles [15:33:45] I know personally when I see increment-like query params, I become curiou [15:33:47] curious [15:45:34] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, User-Nikerabbit: CentralNotice: Saving banners with translatable messages may time out - https://phabricator.wikimedia.org/T224598 (Nikerabbit) [15:46:49] jgleeson it should be fine to use numbers [15:47:27] in the opt in page the fancy version we built first is wle_001 [15:47:50] (also passed in via a variant= param) [15:56:48] Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog, and 3 others: Viewing Special:CentralNoticeBanners subpage can throw fatal BannerExistenceExceptio... - https://phabricator.wikimedia.org/T149240 [16:10:58] (PS1) Jgleeson: WIP added ui explainer text for employer field [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 [16:13:16] Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog, and 2 others: Deprecation warning from CentralNotice: OutputPage::parseInline - https://phabricator.wikimedia.org/T213978 (AndyRussG) O... [16:14:20] Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Analysis, Fundraising-Backlog, and 2 others: CentralNoticeImpression occasionaly fails validation on device enum field - https://phabricator.wikimedia.org/T203597 (AndyRussG) @Ottomata, @Nuria,... [16:16:17] Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog: CentralNotice: Deploy accumulated changes (2019-05-27) - https://phabricator.wikimedia.org/T224444 (AndyRussG) [16:16:49] Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog: CentralNotice: Deploy accumulated changes (2019-05-27) - https://phabricator.wikimedia.org/T224444 (AndyRussG) Open→Resolved [16:58:43] (CR) Ejegg: [C: +1] "looks like a nice quick way to do an English-only test!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 (owner: Jgleeson) [16:59:50] jgleeson: that seems like a fine way to do a quick test. I guess the non-WIP would have a copy-pasted employerExplain2 module? [17:01:51] ejegg, the enlglish-only point is got me wondering if I need to handle the i18n aspects also [17:01:58] english* [17:02:16] jgleeson I think for today english-only is fine [17:02:33] unrelated: The internet here is insane. 76.1Mbps download 191.4Mbps upload [17:03:41] ejegg, re: emmployerExplain2, yep! [17:03:46] gonna add them now [17:07:30] jgleeson I think this is the db we'd be using: https://hepdata.com/hep-data/matching-gifts/ [17:08:15] page isn't loading for me in this cafe, but maybe with that nice bandwidth you can see if it shows the countries covered! [17:14:46] yeah the site is loading slow for me too ejegg [17:15:13] I can't see where it mentions the countries covered [17:15:15] finally loaded for me, and I don't see any country list [17:15:19] ditto [17:15:39] heh, ok, I guess we should ask the fr-non-tech folks where they plan to target [17:33:18] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, and 4 others: Payments wiki form variant with employer field - https://phabricator.wikimedia.org/T219558 (DStrine) @NNichols @... [17:45:32] Fundraising-Backlog: Threshold alert for Ingenico Timeouts - https://phabricator.wikimedia.org/T224683 (XenoRyet) [17:52:15] Fundraising-Backlog: Paypal subscriptions missing initial record - https://phabricator.wikimedia.org/T224684 (mepps) [17:54:09] Fundraising-Backlog, FR-Ingenico, FR-Smashpig: Should payment processor API retries use exponential backoff? - https://phabricator.wikimedia.org/T224685 (Ejegg) [18:03:27] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, and 4 others: Payments wiki form variant with employer field - https://phabricator.wikimedia.org/T219558 (CaitVirtue) Definitel... [18:04:16] Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.34-notes (1.34.0-wmf.8; 2019-06-04): Make 'emailExplain' variant default behavior - https://phabricator.wikimedia.org/T218730 (Ejegg) @Pcoombe, @spatton just a heads-up that this will be ro... [18:08:33] (PS2) Jgleeson: WIP added ui explainer text for employer field [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 [18:10:25] fr-tech I just pushed up a patch which adds in the explainer text variants I mentioned on standup. https://gerrit.wikimedia.org/r/513311. However I'm confused why my
is not being parsed here https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/DonationInterface/+/513311/2/modules/js/ext.donationInterface.employerExplainAlt.js@4 am I forgetting how xhtml works? or is it sanatising the text somewhere I'm missing? [18:11:29] back a little later fr-tech. gonna go pick up Roxy and the kids [18:11:51] if anyon has an answer to my problem above, I'll catch it in the chat logs when I get back, thanks in advance! [18:24:53] (CR) Vedmaka Wakalaka: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/497611 (https://phabricator.wikimedia.org/T208125) (owner: Vedmaka Wakalaka) [18:25:01] (CR) Vedmaka Wakalaka: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/508720 (owner: Vedmaka Wakalaka) [18:31:24] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, and 4 others: Payments wiki form variant with employer field - https://phabricator.wikimedia.org/T219558 (NNichols) No. There... [18:35:35] (CR) Ejegg: "maybe it's just the slash-orientation thwarting your br-parsing?" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 (owner: Jgleeson) [19:55:30] Fundraising Sprint Gravity wasn't always this pushy, Fundraising Sprint HTTP originally stood for Happy Turtle Transfer Protocol, Fundraising Sprint Ivory and eggshell white are the same color, Fundraising Sprint Junebugs prefer July, and 3 others: Ingenico... - https://phabricator.wikimedia.org/T190871 [19:59:09] Fundraising Sprint Pluto is still a planet, Fundraising-Backlog, Patch-For-Review: Fix mobile orientation of radio buttons on payments.wiki opt-in - https://phabricator.wikimedia.org/T201528 (XenoRyet) [19:59:45] Fundraising Sprint Sasquatches can't find us either, Fundraising Sprint They Live, Fundraising Sprint USB stands for underhanded socket bureaucracy, Fundraising-Backlog: Review emergency procedures for holidays - https://phabricator.wikimedia.org/T204763 (XenoRyet) [20:28:03] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 6 others: Upgrade PaymentsWiki to Mediawiki 1.31 (new LT... - https://phabricator.wikimedia.org/T184460 [20:28:38] Fundraising Sprint Winter Wanderland, Fundraising-Backlog, Patch-For-Review: set dash default to 51 million - https://phabricator.wikimedia.org/T183157 (Ejegg) p:Triage→Normal [20:31:15] Fundraising Sprint Window dressing is mostly olive oil, Fundraising Sprint XML ate my homework, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review: FormChooser routing not respecting gateway=adyen for US / USD (but it is for C... - https://phabricator.wikimedia.org/T209416 [20:32:23] cstone: lmk if you want to chat about that duplicate invoice ID problem. I'm available for the next few hrs [20:34:19] ejegg ok cool, I was doing some more investigation some of them are getting through [20:37:50] Wikimedia-Fundraising-CiviCRM, Continuous-Integration-Infrastructure (Slipway): Migrate wikimedia-fundraising-civicrm to a Docker container - https://phabricator.wikimedia.org/T210287 (Ejegg) Thanks @hashar, that sounds great! @Eileenmcnaughton is the local subject matter expert on CiviCRM and its build... [20:39:13] cstone interesting, so sometimes we DO retry with an incremented invoice ID? [20:39:30] or you just mean sometimes we have a good ID the first time around? [20:40:54] hmm i think it might be the good ID, let me find that log i was looking at again [20:41:03] i think on the 28th a bunch went through without error [20:49:52] in this recurring_smashpig_charge-20190428-002101.log.bz2 theres a success instead of an error then it lists 1238 ids [20:49:55] what would be happening there? [20:52:05] cstone: yeah, that's 1238 successful payments [20:52:28] ah okay [20:55:12] hmm, i do see a bunch of 'card expired' type errors. Lemme spot check a couple of those to make sure we're marking those failures in the contribution_recur table [20:55:54] yeah, all the errors for that batch are 'not authorized' or 'card expired' [20:56:10] cstone where were you seeing the big batch of bad invoice ID errors? [20:57:20] yesterday had 54 failures [20:57:35] recurring_smashpig_charge-20190529-012101.log.bz2 [20:58:45] wow, weird, I wonder why they would bunch up like that [20:59:07] so can you tell if the logic that generates the new invoice ID is good? [20:59:31] i.e. for one of those failures yesterday, can you see whether that invoice ID is one we already have stored? [21:00:13] hmm, maybe we should turn on debug logging for ingenico [21:00:25] at least on civi1001 [21:00:51] now, where would that end up? [21:03:54] hi jgleeson! [21:04:01] hey ejegg :) [21:04:10] Looks like English-only is a safe assumption for the matching gifts db [21:05:01] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Security: CentralNotice alerts: Add error handling - https://phabricator.wikimedia.org/T224699 (AndyRussG) [21:05:37] ah cool [21:05:40] ejegg to the generating invoice id question, one that went through correctly on 4/28 was .7 and then the one tried on 5/29 was .8 [21:05:52] ejegg, do we need to start thinking about wiring that up at this point? [21:06:12] cstone wait, for the same donor? [21:06:36] yeah [21:07:04] jgleeson: not connecting the db to our forms yet, just that they only want to test in countries that have employers in the db [21:07:17] cstone yikes! this just keeps getting worse [21:07:35] So we're successfully charging them, then trying to charge them again the very next day? [21:07:44] oh sorry not that [21:07:50] the next month [21:08:08] ohhhh, my bad, I didn't read your comment right [21:08:14] 4/28, then 5/29 [21:08:27] (CR) Jgleeson: WIP added ui explainer text for employer field (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 (owner: Jgleeson) [21:08:42] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: CentralNotice alerts: Document new CN Hook - https://phabricator.wikimedia.org/T224700 (AndyRussG) [21:08:48] ok, so somehow the 5/29 one had a duplicate invoice ID even though we hadn't used it last month. [21:09:08] yeah [21:09:30] so where the heck did it come from, I wonder [21:09:50] can you get into the console to search by that ID? [21:10:11] yeah thats how I saw he had that charge go through on 4/28 [21:10:18] (PS3) Jgleeson: added ui explainer text for employer field [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 [21:10:51] but if you search the ingenico payments console for the ID we got the duplicate error on, does anything show up? [21:11:07] (PS4) Jgleeson: added ui explainer text for employer field [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 [21:11:10] in their console, I believe the invoice id is called 'merchant reference' [21:11:23] jgleeson: oh boy, un-WIP'ed! [21:11:26] looking [21:14:55] ejegg hmm what is the console link again, somehow what i have saved just brings me to a page that says it works haha [21:16:22] cstone: https://wpc.gcsip.com/wpc [21:16:27] you may need to be on the VPN [21:19:26] are there multiple ingenico logins? [21:20:01] cstone yeah, the payments console is different from the hosted checkout config [21:20:20] hmm, usually I get MBeat to help with account stuff there, but he's out today [21:20:21] hmm okay im not sure if ive been in this one before then [21:20:55] dang, we need to get you set up there [21:21:10] And I'm afraid my account doesn't have the ability to make other accts [21:21:44] ok, how about if you grep for that invoice ID in ALL the logs for that date, not just the process-control logs? [21:21:55] ah okay sure [21:21:55] can you see if we're trying it multiple times? [21:27:31] hmm im not seeing the invoice id anywhere [21:28:10] hmm, lemme try searching one at random [21:30:31] (PS5) Ejegg: added ui explainer text for employer field [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 (owner: Jgleeson) [21:34:48] (CR) Ejegg: [C: +2] "This totally works!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 (owner: Jgleeson) [21:35:02] looking good jgleeson! [21:35:33] That CR message also has a note about an annoyance to keep in mind when making the variant files for production. [21:39:09] cstone you should see at least one line in fundraising-misc when you zgrep in the general log archive. Are those not showing up? [21:40:35] oh yep I just cant type hah [21:40:41] Fundraising Sprint Hansel and grep -l, Fundraising Sprint It Came From Out of Scope, Fundraising Sprint James Bond: Licensed to Share, Fundraising Sprint King Kong vs. Mozilla, Fundraising-Backlog: Opt-in numbers in IBM not as expected - https://phabricator.wikimedia.org/T219401 (XenoRyet) @C... [21:42:22] i just see it showing up once [21:42:51] hmm, yeah, i saw the same [21:44:19] hah so i just checked this mornings and it processed 375 [21:44:40] all successful? [21:45:32] yeah in the success array [21:45:58] ok, lemme try to search one of the supposed dupes from yesterday at the processor console [21:49:52] ok, searching one of the rejected ones i do see something in the payment console, and it's rejected! [21:50:33] so... cstone, want to review the code to see whether we're retrying when we shouldn't? [21:51:37] rejected like it was expired and we just keep trying it? [21:51:43] something like that [21:51:50] let's see if the console has more details on the reason [21:52:27] whoa, code 430330, type 'customer input error', INVALID_CARD_NUMBER [21:52:30] uhhhhhh [21:52:41] how did we ever get that tokenized in the first place? [21:52:51] hah what it saved the wrong number? [21:54:35] so weird. That's a donor we were able to charge before. They had a successful initial payment and one successful recurring payment, then this error [21:55:20] huh [21:55:46] ok, lemme peep another [21:57:15] same error for another donor who had 6 previous successful donations [21:57:27] maybe time to write to merchantservices! [21:57:40] fr-tech, what would be our typical englsh-speaking languages. I found this list here and wanted to confirm https://www.sheffield.ac.uk/international/english-speaking-countries [21:57:54] sorry, english speaking countries! [21:58:15] jgleeson: US, CA, GB, IE, AU, NZ [21:58:41] yeah this^^ [21:59:17] thanks! [22:01:16] ejegg, would the easiest way to work out what gateways service those countries be checking in the configs? [22:01:35] as I'm expecting we need a variant file for each like in test [22:01:59] jgleeson: I think we can get away with just ingenico for now [22:02:06] oh rly? [22:02:07] cool [22:02:10] that makes it simple [22:02:45] since we don't really collect donor info for paypal and amazon, it would be weird to have a form with just employer name [22:03:00] and we're mostly not using old-globalcollect [22:03:19] finally, adyen isn't default in any of those six afaik [22:04:58] cstone: want to write a message to merchantservices ? I can send you some of the examples I just looked up [22:05:18] sure [22:06:01] but also, it would be worth updating the code [22:06:08] to not retry on that error [22:06:33] or at least to update the invoice ID between tries [22:07:00] yeah, actually, maybe just always update the invoice ID when retrying! [22:07:21] it did look like it was updating the invoice id between tries or do you mean like every retry it would update? [22:08:02] cstone so when we get a failure code we retry, unless it's one of a specific list that the card networks fine us for retrying [22:08:13] i.e. we retry immediately [22:08:19] ahh okay [22:08:27] so not like trying the next day [22:08:28] and in that case, we should be incrementing the invoice ID [22:08:37] right, it's like with the timeouts [22:08:44] we're retrying up to 3x [22:09:14] ok, this cafe is super full now and the wifi is going to hell. Gonna head back to the apartment! [22:11:59] (Merged) jenkins-bot: added ui explainer text for employer field [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/513311 (owner: Jgleeson) [22:17:51] (PS1) Jgleeson: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/513523 [22:20:03] fr-tech, I think we need this patch https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/509103/ +2'ing also before we deploy the employer field stuff [22:20:57] does anyone have a few minutes to look it over [22:20:58] ? [22:21:11] it's a really small change [22:23:33] jgleeson: Yea, let me take a look. [22:27:13] (CR) XenoRyet: [C: +2] Added support for the new 'employer' field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/509103 (https://phabricator.wikimedia.org/T219558) (owner: Jgleeson) [22:27:36] Yea, pretty straight forward. [22:29:22] cstone OK, I'll look up a couple more of yesterday's dupe invoice failures and send you an email [22:29:35] ok thanks! [22:30:57] thanks XenoRyet ! [22:31:03] No problem [22:31:50] (Merged) jenkins-bot: Added support for the new 'employer' field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/509103 (https://phabricator.wikimedia.org/T219558) (owner: Jgleeson) [22:43:38] (PS1) Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/513525 [22:47:31] fr-tech, I'm about to deploy the employer field and optional field stuff. the release also includes an update to make the emailExplain variant behaviour default behaviour (without variant). Should I confirm with non-tech-fr whether or not any campaigns are using the email variant first? [22:47:48] phab task for the emailExplain default behaviour is here https://phabricator.wikimedia.org/T218730 [22:47:52] dstrine, ^ [22:50:44] cstone: ok, now I'm looking from the start of that list and seeing different types of failures. [22:51:03] so not all randomly bad card numbers? [22:51:26] jgleeson I put a heads-up on that ticket to let 'em know the emailExplain variant is going away [22:51:59] it doesn't look like pcoombe or spatton have responded [22:52:02] ejegg, [22:52:11] you think we're ok to proceed without? [22:52:32] I mean I could release first thing tomorrow, if it'd be better to wait [22:52:43] that way we have 8 hours to pickup any bugs [22:52:50] jgleeson: sure, let's just delete the emailExplain folder in form_variants [22:53:14] I know there's no problem if we supply a variant on the QS that has no corresponding folder [22:53:45] but I'm less sure of what happens if a variant specifies a UI module that we've deleted [22:53:58] let's go ahead today [22:54:06] ok cool [22:54:15] unless you want to get to bed at a reasonable hour ! [22:54:25] I'm matilda-sitting [22:54:28] and she's sleeping [22:54:30] just noticed it's almost midnight your-time [22:54:40] ok, cool, let us proceed! [22:54:51] so this is a pretty typical reasonable unreasonable hour :) [22:54:56] ok cool [22:55:02] just on the vagrant config files [22:55:14] I'm guessing we deploy them via localsettings? [22:55:26] emphasise /guessing/ [22:55:28] the variant config files are in a directory in the mediawiki root [22:55:37] yeah [22:55:39] so you'd just put a patch in gerrit for that [22:55:44] oh I see [22:55:50] we actually expose 'em to the world [22:56:25] ok I'll add a patch for mw [22:56:53] cstone but there are some error codes in there we shouldn't be retrying [22:57:19] jgleeson: ejegg hey sorry just saw jgleeson 's question. NO one is running any fancy test or weird variants at the moment. [22:57:34] thanks dstrine [22:57:44] the regular default donation page should not show the employer field [22:57:59] jgleeson: ejegg I want to confirm this^ [22:58:07] dstrine: I think folks had put variant=emailExplain on all the places with an opt-in field, because they wanted it as default behavior [22:58:39] dstrine: yeah, default donation page won't have that field [22:59:11] dstrine: anyway, if we delete the emailExplain variant at the same time we deploy the tweak to make it default behavior [22:59:28] they won't see any errors if they keep sending that on the querystring [23:00:30] ejegg: I'm down for what you just typed I guess I didn't understand that variant=emailExplain had to be specified [23:02:24] dstrine until today! [23:02:39] ok [23:03:04] but you are saying that fr-online won't have to do anything special moving forward for the email text? [23:03:32] dstrine: so, you can only specify one variant at once [23:03:57] and I thought they might want to try the employer variants in places that are currently using the variant parameter for emailExplain [23:04:07] ok got it [23:04:26] that's why I pulled in T218730 [23:04:27] T218730: Make 'emailExplain' variant default behavior - https://phabricator.wikimedia.org/T218730 [23:04:45] ok go it [23:04:49] proceed [23:04:58] no need for my silly questions [23:04:58] yeah, after today's deploy that explainer will show up whenever someone comes in with opt_in=0 [23:04:59] :) [23:12:09] (CR) Jgleeson: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/513523 (owner: Jgleeson) [23:12:58] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/513523 (owner: Jgleeson) [23:15:33] I'm going to run for the day. Good luck with the release. [23:15:34] fr-tech, it looks like they're after a few bundled updates to civi from eileen in this release https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/513525/-1..1 I'm wondering if it's worth cherry-picking the employer stuff and just releasing that alone to avoid any problems [23:15:53] jgleeson: it's all really safe stuff [23:16:01] thanks eileen [23:17:25] cstone ok, i'm seeing a lot more 'our problem' error codes. Are you digging into that recurring payment code to see why we might be retrying when we shouldn't? [23:18:50] yeah I was looking at that smashpig class [23:21:09] https://phabricator.wikimedia.org/diffusion/WFCG/browse/master/sites/default/civicrm/extensions/org.wikimedia.smashpig/CRM/Core/Payment/SmashPigRecurringProcessor.php$335 [23:21:48] but I just see it going to the handleException where it should be incrementing on 300620 but is something happening so it never gets there then? [23:23:12] seems like it [23:23:26] we [23:23:33] 've got a test case that should cover that [23:26:06] cstone there's some code to say DON'T retry on certain error codes, right? [23:26:21] (PS1) Jgleeson: Added employer field variants [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 [23:26:38] (CR) jerkins-bot: [V: -1] Added employer field variants [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 (owner: Jgleeson) [23:27:10] ejegg in this one its just mentioned 300620 [23:27:11] https://phabricator.wikimedia.org/diffusion/WFCG/browse/master/sites/default/civicrm/extensions/org.wikimedia.smashpig/CRM/Core/Payment/SmashPigRecurringProcessor.php$360 [23:30:14] (PS2) Jgleeson: Added employer field variants [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 [23:30:15] cstone: oh shoot! [23:30:39] hmm, let's see if the SmashPig library is aware of any error codes [23:33:46] cstone ah shoot, I didn't check to see whether those failures were from the same day [23:34:08] so maybe we're not retrying in that loop, but retrying the next day [23:35:12] the ids listed when it says success or failure what are those referring to [23:35:34] cstone I think those are the contribution_recur.id values [23:36:19] aha, this one shows a failure in the ingenico console on 5/28, then we try again on 5/29 with the same invoice ID [23:36:36] so... we should treat some error codes as non-retryable [23:36:48] there are two retry loops [23:37:09] the one that retries timeouts or (should) retry the 'invoice ID already used' error [23:37:31] and the one that retries failed charged for three days [23:38:08] See https://phabricator.wikimedia.org/diffusion/WFCG/browse/master/sites/default/civicrm/extensions/org.wikimedia.smashpig/CRM/Core/Payment/SmashPigRecurringProcessor.php$195 [23:40:00] so when a payment fails for one of these non-retryable reasons we should go directly to 'Cancelled' [23:40:21] and not just increment the failure_count [23:41:06] oh hey: T100536 [23:41:06] T100536: Stop retrying certain types of GC recurring failures - https://phabricator.wikimedia.org/T100536 [23:41:34] (PS3) Jgleeson: Added employer field variants [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 [23:43:04] right I think we're good to go [23:43:20] I didn't see any errors when looking at the logs that weren't the 300620 one, are those caught somewhere else? [23:43:49] (CR) Ejegg: [C: -1] "country_fields should be based on this: https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/globalcollect_gateway/config/countr" (3 comments) [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 (owner: Jgleeson) [23:44:42] oooh ok ignore my last statement [23:45:25] cstone: ok, I'mma quit looking up stuff and send you the email now [23:48:53] (PS4) Jgleeson: Added employer field variants [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 [23:49:16] (CR) Jgleeson: Added employer field variants (3 comments) [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 (owner: Jgleeson) [23:51:05] (PS2) Ejegg: WIP switch for using contribution tracking queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/510757 (https://phabricator.wikimedia.org/T215463) [23:51:58] ejegg, looks like I might run out of time tonight :\ Matilda is due a bottle in 10 minutes before bed. I've updated that patch and can check tomorrow and release then if we're ok with that? Alternatively someone else could push it out if we'd like to get it out today [23:52:04] (CR) Ejegg: [C: +2] "Good stuff!" [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 (owner: Jgleeson) [23:52:09] !!! [23:52:15] jgleeson: sure thing [23:52:28] actually looks like it's all ready to go [23:53:37] just gotta wait for jenkins to merge that last patch [23:54:07] (Merged) jenkins-bot: Added employer field variants [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/513531 (owner: Jgleeson) [23:54:51] ok.. I'm gonna deploy civi first then payments wiki [23:59:46] (CR) Jgleeson: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/513525 (owner: Jgleeson)