[01:09:15] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog: Investigate why Ingenico donation did not recur on 6/14 - https://phabricator.wikimedia.org/T199331 (Ejegg) @MBeat33 oh gosh, it looks like this is affecting ALL recurring credit card donations started since 4/24. I think the issue... [01:44:21] (PS1) Ejegg: Fix processor_id for old-style recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446710 (https://phabricator.wikimedia.org/T199331) [01:54:55] (PS2) Ejegg: Fix processor_id for old-style recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446710 (https://phabricator.wikimedia.org/T199331) [01:55:34] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog, Patch-For-Review: Investigate why Ingenico donation did not recur on 6/14 - https://phabricator.wikimedia.org/T199331 (Ejegg) p:Triage>Unbreak! a:Ejegg [02:00:05] (CR) jerkins-bot: [V: -1] Fix processor_id for old-style recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446710 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [02:01:30] ejegg: just looking at your PR - I agree the change ‘does what it says it does’ - but how does our code handled the 1 [02:19:07] eileen: the old-style GC recurring uses a weird method to identify payments [02:19:20] it has the same 'order_id' for every payment in the subscription [02:19:28] ejegg: ok - did you see the test failed? [02:19:31] but increments another field, 'effort_id' [02:19:41] ah, I'll try to fix that test [02:19:47] (or the code!) [02:20:07] anyway, we were using processor_id to track effort_id [02:21:04] now that it's something non-numeric, the recurring_globalcollect code is interpreting it as zero, adding 1 [02:21:28] and sending effort_id = 1 with the first recurred installment [02:21:42] and the old globalcollect API is rejecting that [02:22:02] since effort_id 1 was already used for the donor's initial payment [02:22:44] oh derp, property of non-object [02:22:53] should that be array access? [02:37:01] ejegg: that all makes sense to me - I’m just not sure what I can & should to do idependily verify it [02:40:36] Although I guess logically the 1 was removed in cleanup so it should be ‘safe’ to put back [02:41:49] (PS3) Ejegg: Fix processor_id for old-style recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446710 (https://phabricator.wikimedia.org/T199331) [02:42:34] yeah... it's a heck of a thing to replicate [02:42:48] I think that test should be fairly true to life though [02:43:16] ejegg: so I guess if I can confirm [02:43:32] 1) we are saving 1 when we don’t have better info [02:43:41] & 2) we aren’t saving it when we do [02:44:00] & 3) it was just removed in clean up not deliberately [02:44:05] right, it's not so much 'better info' as interpreting that field differently for different APIs [02:44:27] it had a very specific meaning for the old recurring logic [02:44:40] (perhaps better captured by the 'installments' field) [02:45:02] so in ‘old’ we are talking about ones started before a certain date & newer ones are different? [02:45:32] old meaning the ones created via the GlobalCollect WebCollect API [02:45:47] which is the main API we've been using for CC processing all along [02:46:04] the new Ingenico Connect API uses tokenized recurring [02:46:55] ok but the ‘old’ one is still our main current one? [02:47:01] yeah [02:47:04] :-) [02:47:18] we have successfully tested the new recurring logic though! [02:47:40] so since 4/24, the only new cc donations that HAVE recurred are the ones from the new API 1-hour tests [02:48:30] I wanna say ‘nice’ but that is not really is it - because they other ones haven't [02:49:02] yeah... pretty bad [02:50:06] it was totally hubris on my part, too... I saw the processor_id field being used for that and immediately thought the previous programmers were doing some collossally dumb hack, when they weren't at all. [02:50:55] then even when I saw that there was another field called payment_processor_id, I didn't take the processor_id field seriously enough to make sure it was still being populated correctly [02:52:05] lol [02:52:20] So at the moment it’s being set to $gateway_subscr_id [02:52:31] & that seems to be always set to ‘something’ [02:52:41] yeah, which makes sense for the new API [02:52:51] whereas the refactor that removed the 1 left the possibility it might be NULL [02:53:30] ie 'payment_token_id' => isset( $payment_token_id ) ? $payment_token_id : null, [02:55:02] ah no - that is payment_token_id [02:55:34] so I guess the part that feels a bit ‘magiic’ is the if-else-else [02:55:40] so, it right now is being set to the same thing as the trxn_id [02:55:43] ie. if (!empty($msg['payment_processor_id']) && !empty($msg['payment_token_id'])) { [02:55:53] elseif (!empty($msg['recurring_payment_token']) && $msg['gateway']) { [02:56:11] OK & trxn-id makes sense sometimes or never? [02:56:22] makes sense for the new API [02:56:28] ok [02:56:39] but the recur logic for the old API depends on it starting out as 1 [02:56:52] so I guess the only risk I can see is if ‘something that shouldn’t’ got picked up in the last part of the 'else' [02:57:10] ie the first 2 parts are no change for anything that hits them [02:57:24] the last part will apply to anything that doesn’t hit the first 2 parts [02:58:00] right, we're assuming anything without either a token or a token ID is old-style recurring [02:58:22] ejegg: so that’s the assumption that I can’t evaluate off my own experience [02:58:38] but you feel confident in it? [02:58:41] I think it should be fine [02:59:04] because the tokenized recurring doesn't actually depend on that value [02:59:24] right & this is only ingenico hitting this? [02:59:26] we could go back to setting it to 1 for everything and the tokenized recurring would still work [02:59:31] ok [03:00:21] yeah, we get recurring donation from paypal, which handles the scheduling on their side and just tells us when we have a new installment [03:00:55] and then from Ingenico, where we have a drupal module to recur the old-API subscriptions [03:01:05] and the civi extension to recur the new-API ones [03:01:24] nothing else recurs as of now [03:01:49] ok - I feel OK to +2 now - I’m gonnat paste above as my comment :-) [03:02:05] :) thanks! [03:02:57] (CR) Eileen: [C: 2] "This is a simple code change & it seems to 'do what it says it does'." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446710 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [03:04:34] ejegg: also your thank you letters patch - is this just a case of running something locally to see if I get the same output? [03:05:39] eileen: nah, it's more like proofreading, or at least making sure the markup doesn't look horribly broken [03:05:49] pcoombe often helps with that [03:06:07] it's not urgent by any means [03:06:45] actually, I could send those out to some of the fr-creative team for them to review in their email clients. [03:07:56] ejegg: yeah that sounds like a good plan [03:08:07] I expect they probably prefer to be involved in that review [03:08:13] (Merged) jenkins-bot: Fix processor_id for old-style recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446710 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [03:11:45] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog, Patch-For-Review: Investigate why Ingenico donation did not recur on 6/14 - https://phabricator.wikimedia.org/T199331 (Ejegg) @MBeat33 the job really should be automatically reporting these failure numbers someplace: https://pha... [03:16:41] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446713 [03:17:20] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446713 (owner: Ejegg) [03:18:42] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446713 (owner: Ejegg) [03:20:14] !log updated CiviCRM from b21d783e7f to 6f311f48d9 [03:20:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:20:57] That deployed included the ts() override [03:21:27] hmm, doesn't seem visibly slower [03:31:41] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog, Patch-For-Review: Investigate why Ingenico donation did not recur on 6/14 - https://phabricator.wikimedia.org/T199331 (Ejegg) OK @MBeat33 , I deployed step 1) of the fix, so old-GC-API / WebCollect subscriptions created from now... [03:33:58] (PS1) Ejegg: Fix impossible test for recurring [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446714 [04:25:45] (CR) Eileen: [C: 1] "So, I agree that the line as is will catch nothing at all AND that your fix makes it do what the writer intended it to do." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446714 (owner: Ejegg) [09:06:21] Wikimedia-Fundraising-Banners: All banner images should be inline SVG where possible - https://phabricator.wikimedia.org/T199147 (Pcoombe) Open>Resolved Thanks! [11:33:39] (CR) Steinsplitter: [C: 1] Add an extra message to allow users to add custom navbar [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/446698 (https://phabricator.wikimedia.org/T138284) (owner: Brian Wolff) [13:52:56] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog, Patch-For-Review: Investigate why Ingenico donation did not recur on 6/14 - https://phabricator.wikimedia.org/T199331 (MBeat33) Great, many thanks for the update and the fixes, @Ejegg [14:16:33] Fundraising Sprint Karma chameleons hide amongst us, Fundraising Sprint Lactose is unusually tolerant, Fundraising Sprint Matt Damon to head up Space Force, Fundraising Sprint Naming Sprints Is Not Important, and 2 others: Civi: enable Force Merge Selected ... - https://phabricator.wikimedia.org/T193674 [14:18:48] Fundraising-Backlog, Fr-CiviCRM-dedupe-FY2017/18: Civi: enable Force Merge All Duplicates - https://phabricator.wikimedia.org/T200012 (MBeat33) [14:19:02] Fundraising-Backlog, Fr-CiviCRM-dedupe-FY2017/18: Civi: enable Force Merge All Duplicates - https://phabricator.wikimedia.org/T200012 (MBeat33) p:Triage>Normal [14:47:18] (CR) Pcoombe: [C: 2] Update Thank You letters [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445721 (https://phabricator.wikimedia.org/T198870) (owner: Ejegg) [14:52:29] (Merged) jenkins-bot: Update Thank You letters [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445721 (https://phabricator.wikimedia.org/T198870) (owner: Ejegg) [15:20:21] morning ejegg [15:21:09] hi jgleeson! [15:21:20] how's your day going? [15:21:47] pretty good, been testing out the import for the opt_in field, took a little working out to get an end to end test but got there in the end [15:22:07] I've also looked over your recurring fix, will plus 1 that shortly [15:22:38] actually I guess I could +2 it [15:25:52] jgleeson: oh, thanks! eileen +2ed the more pressing one last night [15:26:07] Now I need to do a data fix [15:26:17] ahh cool [15:26:20] I think that one will be pretty straightforward [15:34:12] (PS1) Ejegg: Fix undefined variable in exception constructor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446860 [16:00:35] fr-tech, be there in 1 minute! [16:02:08] ejegg standup? [16:20:34] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042 (Milimetric) @DStrine: ok, let us know when you have a solid plan for it and how it fits in with other work. [16:26:25] If you hold a UNIX shell to your ear, can you hear the C? [16:26:39] - copied from someone else :) [16:30:55] fr-tech are we doing tech talk? [16:31:06] I believe so, yea. [16:31:16] Was just finishing an email then jumping in. [16:31:36] figure we can pick up on the list of scheduled jobs [17:30:59] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Import log files not including all skipped rows - https://phabricator.wikimedia.org/T200031 (Ejegg) [17:51:18] PROBLEM - check_mysql on payments1002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [17:56:18] RECOVERY - check_mysql on payments1002 is OK: Uptime: 366 Threads: 2 Questions: 1906 Slow queries: 0 Opens: 26 Flush tables: 1 Open tables: 89 Queries per second avg: 5.207 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [18:01:36] (PS1) Jgleeson: Added support for new 'opt_in' field bug: T199278 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 [18:02:28] (PS2) Jgleeson: Added support for new 'opt_in' field bug: T199278 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 [18:04:20] (PS3) Jgleeson: Added support for new 'opt_in' field bug: T199278 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 [18:06:49] (PS8) Mepps: WIP Refactor of ConfirmCreditCard [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/444311 (https://phabricator.wikimedia.org/T194517) [18:06:51] (PS2) Mepps: Consolidate flags [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445534 (https://phabricator.wikimedia.org/T194517) [18:06:53] (PS7) Mepps: WIP Move to problem array, split out functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445535 (https://phabricator.wikimedia.org/T194517) [18:06:55] (PS5) Mepps: Trying to simplify logic [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446187 [18:08:42] (CR) jerkins-bot: [V: -1] WIP Refactor of ConfirmCreditCard [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/444311 (https://phabricator.wikimedia.org/T194517) (owner: Mepps) [18:08:58] (CR) jerkins-bot: [V: -1] Trying to simplify logic [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446187 (owner: Mepps) [18:09:28] (CR) jerkins-bot: [V: -1] WIP Move to problem array, split out functions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445535 (https://phabricator.wikimedia.org/T194517) (owner: Mepps) [18:09:35] ejegg this looks like a missing translation string again ^^ [18:09:43] (CR) jerkins-bot: [V: -1] Added support for new 'opt_in' field bug: T199278 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 (owner: Jgleeson) [18:12:08] fr-tech, when we make schema updates do they get run in CI? [18:12:14] to civi [18:16:03] jgleeson: they should! [18:16:37] ah ok [18:16:49] jgleeson: ohh, you may have to add a call to your XXX_update() function to the XXX_install() function at the top of the file [18:18:17] hmmmm I'm seeing some geocode import test failures although I only appended the cases array with 3 of my own so no sure how I've broken existing ones [18:18:28] oh weird! [18:18:43] some of those tests are fragile [18:18:56] and they don't all clean up after themselves very well [18:19:06] (CR) jerkins-bot: [V: -1] Consolidate flags [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/445534 (https://phabricator.wikimedia.org/T194517) (owner: Mepps) [18:19:09] We have a CiviFixtures class that's supposed to help with that [18:19:34] i think the geocode thing has tripped me up before [18:19:53] my best guess is it's something to do with regional differences [18:20:09] between my dev env and everywhere else? [18:20:14] but that's just a shot in the dark [18:20:45] ejegg, it appears there's a few missing from the XXX_install() method, should I add them all or just my own [18:20:53] jgleeson: just your own [18:20:59] ok cool [18:21:10] a lot of those missing ones will be data fixes that wouldn't apply to an empty db [18:21:29] got it [18:21:32] for instance the one I'm about to write to fix the recur records [18:22:45] my env is really flakey at the mo, I had to comment out your schema update with the comment 'Tokenize recurring payments that we were unable to tokenize up front' as it wouldn't run for me [18:23:21] flaky* [18:23:23] jgleeson: ahh, maybe missing the standalone 'smashpig' db that holds the damaged table [18:23:40] or doesn't have all the connection info set up [18:24:01] yeah it was something db connections related [18:26:05] (PS4) Jgleeson: Added support for new 'opt_in' field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 (https://phabricator.wikimedia.org/T199278) [18:32:23] jgleeson: it worked! [18:32:59] just one teeny request - can we make the inputs in the test use '1' and '0' for yes and no as well, since that's what ends up in the queue messages? [18:33:21] ahhhh I wanted to ask about that [18:33:33] I was confused by the 'Y' [18:33:44] I assumed they were being mapped in the test world due to them working [18:34:53] I'm sure the 1 and 0 will work too [18:35:03] I think some of the spreadsheet imports use Y and N [18:35:26] yeah definitely makes sense to use the real values [18:35:33] And I think it might even be the Civi API that does the bool normalization for us [18:35:41] ahh [18:36:58] (PS5) Jgleeson: Added support for new 'opt_in' field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 (https://phabricator.wikimedia.org/T199278) [18:37:07] ok updated the fields and pushed [18:37:12] cool! [18:37:34] I might add some notes for testing [18:37:39] as it's a bit of a hack [18:37:42] as explained on standup [18:37:48] atm at least [18:37:56] doesn't look so hacky to me... [18:38:07] not the unit tests, the manual [18:38:09] Oh, you mean getting the field in a queue message [18:38:11] from the DI side [18:38:12] ye [18:38:33] I ended up resorting to redis dump and restore to make it easier for repeat donations [18:38:47] or just add the opt_in: true to another country in country_fields.yaml [18:38:49] but I guess everyone will have their own shortcuts for that type of thing [18:39:22] yeah I did that, and then effectively snapshot the message so I could replay it at the queue level using `redis restore` [18:39:42] to save repeat donations [18:39:54] ah cool [18:40:15] I mostly just use the smashpig scripts to dump to file and load from file [18:40:41] never actually tried the redis dump and restore tools [18:40:48] what format do they work with? [18:41:38] (CR) Ejegg: [C: 2] "Looks good!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 (https://phabricator.wikimedia.org/T199278) (owner: Jgleeson) [18:42:09] jgleeson I guess the next step in the backend will be to get the field into the silverpop export [18:42:27] that'll be in our tools repo, under the silverpop dir [18:42:50] redis dump is it's own wacky serialised binary [18:42:51] all of the queries are in update_table.sql [18:42:54] ahh [18:43:02] oops, that's the silverpop_export dir [18:43:04] but if you capture it and restore as-is, it's fine [18:43:26] I think you can export a readable form of it with flags [18:43:45] ok cool, shall I move on to that then? [18:43:50] OK, I'mma do one quick fix for the recurring donations that have a bad value in the processor_id field, but have not yet been canceled [18:44:12] jgleeson: yeah, want to take a look around that dir and let me know if you have any questions? [18:44:29] the update.py script is the entry point [18:44:41] I can't just now, I'm about to head off. I should of been down 30 minutes ago ha [18:44:45] I can do first thing [18:44:54] oh right, go play with yr kid! [18:45:02] and make a go at it, noting anything for when your online [18:45:06] but yeah, would be a good thing for tomorrow [18:45:31] cool, I'll jump on it. Catch you tomorrow! [18:45:34] cya [18:45:38] have a good evening fr-tech! [18:48:56] (Merged) jenkins-bot: Added support for new 'opt_in' field [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446876 (https://phabricator.wikimedia.org/T199278) (owner: Jgleeson) [19:04:30] ejegg, you around? [19:06:28] (PS1) Ejegg: Fix processor_id for subscriptions not yet canceled [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446883 (https://phabricator.wikimedia.org/T199331) [19:06:40] saurabhbatra: yeah! [19:06:54] Just dealing with a little snafu in recurring payments right now though [19:07:08] fr-tech anyone about to review that last patch ^^^^ ? [19:07:25] Yea, I can take a look. [19:07:26] The quicker we can get it out, the better. [19:07:30] thanks! [19:08:10] It'll fix up the ~55% of badly recorded donations which haven't yet been canceled [19:08:41] no probs, fixed my problem :-) [19:08:53] saurabhbatra: oh nice! [19:11:16] XenoRyet: the payment_token_id IS NULL is just a belt-and-suspenders double-check thing [19:12:21] cool [19:13:25] (CR) XenoRyet: [C: 2] Fix processor_id for subscriptions not yet canceled [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446883 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [19:13:32] thanks! [19:13:40] No worries [19:14:12] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446886 [19:14:21] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446886 (owner: Ejegg) [19:18:41] (Merged) jenkins-bot: Fix processor_id for subscriptions not yet canceled [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446883 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [19:18:43] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446886 (owner: Ejegg) [19:21:03] !log updated CiviCRM from 6f311f48d9 to 2bb0d0f9d0 [19:21:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:24:04] hi MBeat [19:24:14] hi saurabhbatra [19:24:22] o/ [19:25:01] MBeat not sure if you remember but you shared some stats with me quite some time back [19:25:08] saurabhbatra: I was gonna say, the limited sample of fraud sounds problematic, because I suspect we’re systematically only catching certain types of fraud. MBeattie can review the sample you found perhaps. [19:25:36] For example, if we only have examples of fraud on a certain payment processor, it’s not very relevant training data for other types of fraud. [19:25:56] i think we're only catching frauds that got through [19:26:12] sure thing. I have docs that track the GC600s stopped by our filters, and also the ones we manually settle after review, let meknow if you’d like links saurabhbatra [19:26:39] saurabhbatra: We might have to dig into the logs to get details about other types of fraud fwiw [19:26:46] we have some data for Adyen as well, but less transactions [19:27:03] yeah, chargeback metrics! so do you guys get the data off of the fr civi instance? [19:27:20] or is fraud information stored in other places as well [19:27:57] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog, Patch-For-Review: Investigate why Ingenico donation did not recur on 6/14 - https://phabricator.wikimedia.org/T199331 (Ejegg) @Mbeat33 all the WebCollect subscriptions created in the past month (not yet canceled) have been fixed... [19:27:57] I'm guessing transaction information which is stopped by minfraud or by the payment processor marked as possible fraud never makes it to civi [19:30:10] saurabhbatra: the Civi instance does have chargeback data [19:30:16] awight: do you mean the table log_civicrm_contribution or text logs? [19:30:26] Text logs :-[ [19:30:45] in the civicrm_contribution table, the contribution_status_id column has a value that means chargeback [19:30:49] ejegg: yup, I was able to get ~1500 rows of chargeback data out of the Civi instance [19:30:54] ah, great [19:30:57] for chargebacks metrics we d/l transactions from the Ingenico console periodically, but we also use the Ethoca alert system, which tips us off about impending fraud. We get to proactively refund them to prevent chargebacks, but for tracking we count them as chargebacks anyway. let me email you the links. And yeah, the stuff that gets stopped does not reach Civi as transactions (though some things show in vivafredge tables, like ejegg [19:31:19] MBeat: ahh, and those would not show in civi as chargebacks [19:31:35] since civi's status goes off the status in the audit file [19:31:49] yes, good point [19:32:05] i wonder if I'll get more results if I use contrib_status_id [19:32:27] I’m pretty sure most fraud doesn’t even hit Civi [19:32:28] i was using something like - select * from civicrm.civicrm_contribution where cancel_reason='fraudulent donation' and is_test=0; [19:32:47] unless we’re only looking for chargebacks, and I missed that? [19:32:57] probably the union of that + contribution_status_id 13 [19:34:01] Oops, just caught up with backscroll. Sorry to be redundant :-) [19:34:07] awight: we're getting a healthy mix of payment gateways though, i see a lot of paypal, cc (credit card) and amazon [19:34:26] That’s interesting! But all chargebacks, right? [19:34:56] yes, all chargebacks [19:35:43] Maybe likely chargebacks should have their own model? I’d be curious to know if they overlap well with fraud that gets stopped earlier... [19:35:44] i can show you a sample of the data, maybe over OTR? [19:35:54] naw I’m okay for now [19:35:55] :) [19:35:57] Actually... [19:36:12] * awight fumbled for yubikey [19:36:56] oh or you can run the query for yourself, i'll pm you [19:39:07] Fundraising Sprint Naming Sprints Is Not Important, Fundraising-Backlog, Patch-For-Review: Deploy new thank you email translations - https://phabricator.wikimedia.org/T198870 (Ejegg) OK, these are deployed! If you'd like to fire off some test-spam, the shooting range is thisaway: https://civicrm.wiki... [19:39:20] fundraising-tech-ops: encrypt fundraising mariadb replication - https://phabricator.wikimedia.org/T170320 (Jgreen) paymentsdb replication is encrypted as of today... [19:41:34] cwd: Mind if I PM keys [19:42:45] i think Casey's vacationing right now [19:44:21] oh, good for him! [19:47:59] fundraising-tech-ops: encrypt fundraising mariadb replication - https://phabricator.wikimedia.org/T170320 (Jgreen) For the fundraisingdb's mysql needs to be restarted to enable SSL support, then enable it for replication like so (adjusted): change master to master_host='payments2001.frack.codfw.wmnet', mast... [19:51:13] Jeff_Green: I don’t know if you’re in on the scheme to grant me staging access, but I have my yubikey at the ready if so. [19:53:26] fr-tech just added an explanation of the failmail from 1 hr ago to https://www.mediawiki.org/wiki/Fundraising_tech/Failmail_zoo [19:53:59] Nice. And thanks for the ping. [19:54:05] hi awight, ya we can do that [19:54:22] awight: is it the same key as before, or a new one? [19:57:24] Jeff_Green: Same key. Glad to know it’s still on file :-) [19:57:26] relocating... [19:58:11] awight: we should have it in git, do you still have your FR ssh key too? [19:58:50] that… might be burnt aside from deep backups [19:59:20] oho, > AAAAB3NzaC1yc2EAAAADAQABAAACAQDIElKCAKHvdAGf7hZi [19:59:21] ? [20:00:15] i gotta dig through git...i'll ping you here when I know more [20:01:29] :) happy to upload to office if the digging gets tough [20:07:41] Jeff_Green: https://office.wikimedia.org/wiki/User:Awight/ssh_key_fr [20:09:42] got 'em both from git! [20:10:36] :) [20:10:57] Good for my credibility [20:11:08] true [20:11:57] (CR) Mepps: "Hmm this seems to have disappeared again." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446657 (owner: L10n-bot) [20:12:59] awight: did casey already fill you in on bastion config? [20:14:17] Jeff_Green: Not yet [20:14:34] okeedokee [20:15:20] it's a little simpler now, configwise, maybe [20:15:34] I'll email [20:15:45] :100%: [20:17:48] ok sent [20:24:11] awight: you should have access now, let me know how it goes [20:24:50] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042 (LilyOfTheWest) @Milimetric Hashing out the first step of what's needed from the community organizer end is relatively straightforwa... [20:27:56] (CR) Eileen: [C: 2] Fix undefined variable in exception constructor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446860 (owner: Ejegg) [20:33:28] (Merged) jenkins-bot: Fix undefined variable in exception constructor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446860 (owner: Ejegg) [20:33:49] awight: catch you Monday, off to sleep now! [20:42:46] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042 (DStrine) There are a bunch of privacy issues. This requires a lengthy research project and some level of community involvement. @Js... [20:54:00] Jeff_Green: nice work, I’m in! [20:54:06] woot! [20:54:18] Seems that frdev1001 is the current staging box? [20:54:56] yes, that's right [20:55:03] it replaced lutetium [20:56:23] Access denied for mysql, is there any .my.cnf I should do? [20:57:03] oh, forgot about that... [20:57:06] looking [20:57:07] fr-tech, are we on the right track to use payments_fraud.validation_action = ‘reject’ to find fraud? [20:57:23] It seems to be mostly contribution_status_id = 1 [20:58:01] awight: I guess those would be the ones where our filters got it wrong [20:58:13] and MBeat pushed it through at the console [20:58:36] mm there are 106 rows of that… seems excessive. [20:58:42] 106k* [20:58:54] oh jeez, that's a lot of work for DS [20:59:03] but if it has validation_action=reject [20:59:15] we wouldn't have automatically captured it [20:59:16] * awight scratches head [20:59:32] well, this is the interesting point where saurabhbatra and I are stuck, for now [20:59:34] is the source 'audit' for those? [20:59:44] Good Q, I haven’t done any queries myself [21:00:10] I should have mysql access shortly, though, I hear elves tinkering as we speak [21:00:54] not the silent, deadly elves of Tolkein then [21:01:57] * awight pulls an arrow from SSH public key [21:06:45] mepps or eileen_: I want to make it possible to tag ContributionRecur entries with RecurringRestarted. Can I make the existing RecurringRestarted tag apply to two different entities, or do I need a whole new tag? [21:07:36] hmm, looks like used_for is a 1:1 relation [21:07:48] tag has ‘used for' [21:08:54] awight: ok try now? [21:10:03] Jeff_Green: Chilling experience of déjà vu, this simulation of working for fundraising is very realistic. [21:10:15] ha [21:10:21] eileen_: ok, i'm going to try to create a second tag with name RecurringRestarted but used_for = 'civicrm_contribution_recur'; [21:10:23] just wait until something catches fire :-) [21:10:34] * awight shaves hair to prevent it catching [21:10:48] Yes thank you for the warning [21:11:04] * Jeff_Green checks if I still have your phone# for 3AM SMS's about payments gone awry [21:11:47] Hey it’s not really like that. When I was on the team we would just fire up the time machine at opening hours, to take care of things in the past like that. [21:11:57] :-) [21:11:59] ahem. /me runs away crying [21:13:43] ejegg: Have you talked any local theaters into screening Sorry to Bother You? [21:14:06] What's that? [21:14:25] Boots Riley’s first film, it’s gonna be amazeballs [21:14:54] ooh, cool! [21:15:09] There's a dope cinema / bar that might be up for it [21:15:18] I bet :) [21:15:38] oh right, it's the name of a Coup album too [21:16:58] Gross! confirmed that contribution_status_id = 1 is "completed" [21:16:59] not cool. [21:17:17] Now I have no idea why there are so many rejected validation actions, tooo complicated [21:17:34] what's the source_type breakdown on those? [21:18:06] k checking [21:18:20] * awight fingers trembling with writing a civi query [21:20:49] Argh, where is contribution_tracking ID in Civi? It’s only one-way from contribution_tracking -> contribution_id? [21:24:46] +----------+-------------+ [21:24:47] | count(*) | source_type | [21:24:48] +----------+-------------+ [21:24:50] | 160838 | NULL | [21:24:52] | 9754 | audit | [21:24:53] | 13 | direct | [21:24:54] | 168 | listener | [21:24:55] | 38286 | payments | [21:24:56] +----------+-------------+ [21:25:06] using horrific query, [21:25:08] select [21:25:09] count(*), [21:25:10] e.source_type [21:25:12] from fredge.payments_fraud f [21:25:13] left join drupal.contribution_tracking ct on ct.id = f.contribution_tracking_id [21:25:15] left join civicrm.civicrm_contribution c on c.id = ct.contribution_id [21:25:16] left join civicrm.wmf_contribution_extra e on e.entity_id = c.id [21:25:17] where [21:25:18] f.validation_action = 'reject' group by e.source_type [21:25:40] totally different results than Saurabh got, actually [21:25:49] I’ll look at his query [21:26:28] oh nbd, he was joining contribution extra ID against contribution tracking ID [21:35:13] awight: with all those left joins, the NULLs maybe have no contribution record at all? [21:35:33] i.e. actually were rejected correctly [21:35:33] that’s what I was expecting to see... [21:35:36] yah [21:35:47] the 'payments' ones confuse me [21:35:49] But I’m thinking these should be included in our training... [21:46:29] (PS1) Ejegg: Add RecurringRestartedUncharged tag [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446979 (https://phabricator.wikimedia.org/T199331) [21:51:25] (PS1) Ejegg: Generalize get_tag_names util, move to wmf_civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446980 (https://phabricator.wikimedia.org/T199331) [22:01:06] Fundraising-Backlog, Analytics, MediaWiki-extensions-CentralNotice: Make banner impression counts available somewhere public - https://phabricator.wikimedia.org/T115042 (LilyOfTheWest) @DStrine what are the timelines you're looking into for setting aside time for this? Knowing how long you expect to... [22:12:47] (PS1) Ejegg: Use RecurringRestartedUncharged tag in charge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446983 (https://phabricator.wikimedia.org/T199331) [22:14:42] XenoRyet: ^^^ this chain of patches is most of what we need to get the rest of those donations fixed [22:14:54] 10-4, I'll take a look [22:15:07] I should write a test for that last one, but the first two should be ready to go! [22:18:29] (CR) jerkins-bot: [V: -1] Use RecurringRestartedUncharged tag in charge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446983 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [22:23:06] hmm, didn't even need a new test to fail out... [22:37:23] boo: CiviCRM_API3_Exception: 'civicrm_contribution_recur' is not a valid option for field entity_table [22:37:37] ohhh, shouldn't have the civicrm_ [22:38:00] hmm, no, the rest do... [22:38:50] grr [22:50:09] oh, silly me, I didn't updb on my phpunit database! [22:50:16] that should actually work... [22:54:28] (PS2) Ejegg: Generalize get_tag_names util, move to wmf_civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446980 (https://phabricator.wikimedia.org/T199331) [22:54:30] (PS2) Ejegg: Use RecurringRestartedUncharged tag in charge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446983 (https://phabricator.wikimedia.org/T199331) [22:54:34] OK, now with a test, and toxic code made less so ^^^ [22:54:57] 10-4 [22:55:50] final step is to actually mark the mistakenly canceled subscriptions with that tag, uncancel them, and fix their processor_id [22:56:02] makes sense. [23:00:49] (CR) XenoRyet: [C: 2] Add RecurringRestartedUncharged tag [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446979 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:02:36] (CR) XenoRyet: [C: 2] Generalize get_tag_names util, move to wmf_civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446980 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:04:59] (CR) XenoRyet: [C: 2] Use RecurringRestartedUncharged tag in charge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446983 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:05:03] Yea, looks good. [23:07:08] (Merged) jenkins-bot: Add RecurringRestartedUncharged tag [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446979 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:09:48] (Merged) jenkins-bot: Generalize get_tag_names util, move to wmf_civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446980 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:11:20] (Merged) jenkins-bot: Use RecurringRestartedUncharged tag in charge [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446983 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:17:32] (PS1) Ejegg: Fix remaining misrecorded contribution_recur records [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446993 (https://phabricator.wikimedia.org/T199331) [23:19:28] XenoRyet: I think that'll do it for the final step ^^^ [23:19:37] looking [23:19:44] I should just mock up a broken _recur record locally to test [23:20:07] Good call. Let me know how it goes. [23:22:39] SQL syntax error :P [23:26:22] (PS2) Ejegg: Fix remaining misrecorded contribution_recur records [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446993 (https://phabricator.wikimedia.org/T199331) [23:26:29] Ahh, cancel(l)ed status needed a double l [23:26:36] hah [23:26:40] XenoRyet: ^^^ that version worked for me locally [23:27:33] just to be superstitious, I'mma deploy the existing stuff first [23:27:43] Sounds like a plan. [23:27:56] I'll give this one a try local as well [23:28:14] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446994 [23:28:49] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446994 (owner: Ejegg) [23:29:52] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446994 (owner: Ejegg) [23:31:43] !log updated CiviCRM from 2bb0d0f9d0 to 08680be68e [23:31:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:32:01] (CR) XenoRyet: [C: 2] Fix remaining misrecorded contribution_recur records [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446993 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:33:19] Thanks! [23:33:46] No worries. Seemed important to get this handled quickly. [23:34:02] yeah, don't want to skip any more installments if possible [23:34:24] I could calculate the exact cost of that bug, but it would probably make me feel bad [23:34:28] *worse [23:34:49] Yea, no real utility in knowing that number. [23:35:06] ...and wouldn't include this wasted time, or the time taken by DS [23:38:43] (Merged) jenkins-bot: Fix remaining misrecorded contribution_recur records [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446993 (https://phabricator.wikimedia.org/T199331) (owner: Ejegg) [23:40:33] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446999 [23:40:42] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446999 (owner: Ejegg) [23:42:03] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/446999 (owner: Ejegg) [23:43:13] !log updated CiviCRM from 08680be68e to 22a8c47668 [23:43:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:44:40] ejegg: no worries from DS-land. the complexity of what y’all handle never fails to impress me [23:44:52] Thanks MBeat! [23:45:18] XenoRyet: ah drat, I might need to tweak the next_sched_recur_date on those too... hmm. [23:45:36] Think so? [23:46:47] phooey, yeah, at least for the ones that were cancelled more than a week ago [23:46:59] well, at least they've got that tag to find them by [23:47:16] though I don't seem to see it in the UI [23:50:44] It's on the tag management screen [23:52:04] yeah, I just don't see it connected to the actual recurring record that the db says it's on [23:53:22] hrmph, well, at least july & august have 31 days, so I can use simple logic to get the next date [23:56:43] Gotta call it a day. I'll catch you tomorrow. [23:56:50] ok, have a good evening