[00:00:44] !log process-control config revision is 0098b7a118 - adjust dedupe rule [00:00:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:03:48] ejegg: I managed to get the contact import patch merged - yay [00:03:56] upstream [00:18:56] awesome! [00:19:14] ok, I'mma see about those missing opt-ins [00:36:58] well, it's certainly not the most efficiant way to opt them all in [00:37:02] but it seems to be working [00:37:37] that queue consumer seems to slow down as it runs [00:37:56] hmm, or it's just slow on certain contacts [01:37:28] Fundraising Sprint Da Vinci Coder, Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, and 3 others: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (Eileenmcnaughton) @CCogdill_... [01:37:56] Fundraising Sprint Da Vinci Coder, Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, and 3 others: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (Eileenmcnaughton) @Ejegg @CC... [01:40:57] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506335 [01:41:49] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506335 (owner: Ejegg) [01:42:20] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506335 (owner: Ejegg) [01:43:57] !log updated fundraising CiviCRM from 468f85e524 to 519fe8028e [01:44:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:44:32] ejegg|afk: you are not having any ssh probs then? [01:44:42] I’l lost all ssh to our servers [01:44:49] no, I'm not [01:44:57] hmm, what are you seeing for an error? [01:45:49] well I lost all my sessions [01:46:01] & when I try to ssh back in I think the bastion is ignoring me [01:46:13] hmph, that's annoying [01:46:30] I've got a job running under screen, but it's still scrolling away [01:46:40] yeah .... [01:46:46] and when I connected again to deploy that audit patch it went through fine [01:46:57] hmm - it’s me…. [01:48:00] what I wanted to do was to run the dupe rule specifically against a group (542) to get Leanne’s ones merged [01:49:10] so I can copy the dedupe job to the one_off and edit the command line [01:49:18] yep [01:49:21] let's see what that would be [01:49:35] if you start from the mg one that would give group id param [01:49:37] --group_id=542 [01:49:43] ah yeah [01:50:57] I guess I might as well wait until I can catch cwd instead of logging a phab since it will probably be randomly fixed by the time he sees it if I do [01:51:31] any particular rule_group_id ? [01:51:57] 13 [01:52:53] so like this: civicrm-merge --batch=1000000 --group_id=542 --rule_group_id=13 [01:53:00] (with all the usual drush args first) [01:53:16] yep [01:55:56] ejegg: maybe we could increase either dedupe frequency or batch size in the main job too - it’s doing 6k pretty quick at the number it’s on because most are dupes [01:56:27] sure, sounds good [01:56:41] I’d kinda like to let it catch up as it will clean out a few - but also I don’t want MBeat to be dealing with recent ones that would be merged if we were up to date [01:56:41] maybe I'll wait to do that till after the opt-in queue is cleared out [01:56:47] ejegg: good thinking! [01:57:57] do you know what to make of the ubsubscribe bug? [01:58:15] eileen: which unsubscribe bug is that? [01:58:21] new failmail [01:58:52] oh, invalid string is wierd [01:59:30] so, we are running basically all the people that have opted in back through the opt-in queue consumer, since the audit parsers were ignoring that flag till now [01:59:49] Which means I've got a loop running the queue consumer job 400 times in a row [02:00:15] ouch [02:00:31] heh, it was easier than the alternative [02:01:01] :-) [02:01:02] that queue can take messages that are just a single property, so dead simple to parse out of log files and transform with a couple lines of sed [02:01:22] and we have a maintenance script that can dump things into queues really easily [02:01:30] ah that’s cool [02:01:34] when will that dedupe line run [02:01:43] ok, that latest failmail is what I expected [02:01:51] oh, I need to run that dedupe one manually, one sec [02:02:07] ah [02:02:07] I can pause the opt-in thing for a minute [02:02:31] this ssh thing is a pain :-( [02:02:47] oops, pressed ctrl-c a bunch of times, maybe that'll trigger a failmail storm [02:02:53] :-) [02:03:42] ok, running the merge job [02:04:34] oh, did that need to be scheduled to run various times? [02:04:47] I guess so, with that batch limit [02:05:52] ejegg: nah [02:06:01] the ‘true’ limit is about 10k contacts [02:06:07] as that is how many are in the group [02:06:25] ah, got it [02:09:27] ok the number is dropping [02:09:56] just passed CID 10 000 000 [02:13:43] :-) [02:14:16] yeah about 800 are merged now [02:20:27] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (Eileenmcnaughton) @leannes on digging it turns out we were using the name + email rathe... [02:25:12] ejegg:just looking at cancel_reason & processor id [02:25:20] the processor one is Ok I think? https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501128/ [02:25:49] right, so how about we come up with a set of standard cancel reasons [02:26:09] like card failed, donor cancelled, etc [02:26:16] that are processor-agnostic [02:26:53] then the job that normalizes IPN messages can decide on one of those [02:27:09] ok, looking at the processor ID one [02:28:13] looks good! [02:28:59] so let me just convince myself that's not going to have any unintended consequences [02:29:14] ejegg: ok - makes sense [02:29:19] I was worried we were misusing that field for recurring globalcollect, but I may have been mistaken [02:29:57] so on the fail - we perhaps need a generic prefix like automated [02:30:03] or something [02:31:25] ok, so recurring globalcollect uses 'processor_id' to record the number of times the thing has been charged so far [02:31:36] but your patch touches payment_processor_id [02:31:47] not processor_id [02:31:52] i THINK that should be good [02:32:53] ah yeah confusing huh [02:33:09] so we only handle one form of cancel at the moment from the looks case 'recurring_payment_profile_cancel': [02:33:10] $message['txn_type'] = 'subscr_cancel'; [02:33:10] hmm, and the new ingenico ones already have a payment_processor_id. So where are we setting that? [02:33:27] are they coming through the extension? [02:33:55] the extension is just making new monthly charges based on rows that already exist there [02:35:58] ok yep- 21367 of them [02:36:48] aha, it's the new code at the start of wmf_civicrm_contribution_message_import [02:37:11] the new ingenico ones come in with recurring_payment_token set [02:38:33] ffs I had a file mask on - no wonder I couldn’t find stuff [02:38:49] err, no, that stuff at the start is only to associate the record with EXISTING recurs that have that token [02:38:53] it's in wmf_civicrm_message_contribution_recur_insert [02:39:34] line 95 of recurring.inc [02:39:52] creates a recurring payment token based on the gateway field [02:40:08] then pulls out token and payment processor ids from that result [02:40:37] ok this is kinda creepy - for those ones there is no cancel link but there is an edit link [02:40:42] ok, so that's fine, because we're actually only getting new recur records for ingenico via the donations queue [02:41:02] (for the others the edit link is broken :-( [02:41:04] on the edit screen can you cancel? [02:41:29] not at the moment [02:41:40] oh jeez, that's bad :( [02:41:42] I think short term I’ll find a way to bring back the edit link [02:42:00] but I think this is the real fix https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501131/ [02:42:22] ah, ok [02:42:30] actually - I think I have an idea about the missing cancel link [02:42:34] so... let's see if we need to change other code to account for that [02:43:30] so, Pending vs In Progress? [02:43:50] is In Progress to mark ones that we're in the middle of processing a new monthly payment for? [02:44:00] So they don't get double charged? [02:44:17] ejegg: yep [02:45:47] There is a function updateOnNewPayment that will set to complete once done [02:46:40] you mean once the monthly payment has been charged? [02:47:28] So, our process may be a bit silly, in that we usually insert the recur record after we've already charged the first monthly payment [02:48:03] so if 'Pending' means a recurring contrib that has zero payments so far [02:48:29] we would be correct in creating ours as 'Completed' since we generally create them with 1 payment already in the bag [02:49:01] ejegg: no that would be ‘in progress' [02:49:12] completed is ‘all received nothing to see here' [02:49:47] as in, we're never going to charge this person again? [02:50:01] yep - or not on this authorisation [02:50:17] I was wrong about the cancel link being missing [02:50:45] sorry, 'not on this authorization' ? [02:50:47] update - with the payment processor id filled in 1) the edit link works, 2) the cancel link works as well as on the others [02:51:01] i like that update! [02:51:02] ie. they could authorise us to do another 50 payments [02:51:17] & we’d create a new contribution_recur record [02:51:19] ohh, so it's assuming recurring is always for a limited time [02:51:44] whereas we're treating it as indefinite, until their card fails 3x in a row [02:51:50] then we mark it Failed [02:51:58] or until they tell us to stop [02:52:03] and we mark it Canceled [02:52:09] well if we have installments as ‘0’ then it will never be 'completed' [02:52:28] oh shoot, now I'm abusing the installments field [02:52:42] oh [02:52:54] i'm incrementing it each time we charge a new installment [02:53:02] the thing is we are currently hacking the core code to see completed as cancellable [02:53:35] ok, if we want to align with core I need to stop abusing that field, and I guess get a count of associated contribs instead [02:53:47] what do we use it for [02:54:27] i forget right now [02:54:50] sorry, it's getting pretty late here, I think I need to call it quits for tonight! [02:55:06] sure [02:55:18] that dedupe job looks like it's getting near the end [03:06:33] all done! [03:06:40] have a good one [03:46:33] Fundraising-Backlog, MediaWiki-Vagrant, Patch-For-Review: Add mediawiki-vagrant php7.2 xdebug support - https://phabricator.wikimedia.org/T220406 (bd808) >>! In T220406#5134091, @Mainframe98 wrote: > The change above results in the message `Cannot load Xdebug - it was already loaded` whenever I run a... [05:46:25] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, and 2 others: Search for name and org name in a 'the-agnositc way' in Civi - https://phabricator.wikimedia.org/T115536 (Eileenmcnaughton) Email sent out Hi al... [06:18:11] Fundraising Sprint Da Vinci Coder, Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, and 3 others: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (CCogdill_WMF) Hey @Eileenmcn... [06:51:55] Fundraising Sprint Da Vinci Coder, Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, and 3 others: Re-run omnirecipient repair to catch the few missed ones - https://phabricator.wikimedia.org/T215865 (Eileenmcnaughton) @CCogdill_... [07:39:18] (PS2) Eileen: Update extended report [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506049 (https://phabricator.wikimedia.org/T218616) [08:06:49] Fundraising-Backlog, MediaWiki-Vagrant, Patch-For-Review: Add mediawiki-vagrant php7.2 xdebug support - https://phabricator.wikimedia.org/T220406 (Mainframe98) It was the latter, I updated because of an unrelated reason (needed rMWVAe7f3fd), and after provisioning, I noticed the message. The thing,... [10:06:10] (CR) Ejegg: "The failing test is looking at the case when there is a donor with a non-primary email that matches the email to be opted in. The desired " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/505686 (https://phabricator.wikimedia.org/T217710) (owner: Cstone) [12:56:58] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (LeanneS) Wonderful - thank you!! Looks like the count has gone down a lot to ~1600. I'm... [13:36:46] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog: Payments wiki form variant with employer field - https://phabricator.wikimedia.org/T219558 (NNichols) Thanks Sam! Thoughts in your Qs: 1. In an ideal world it would be great if the information went to the... [14:22:37] fundraising-tech-ops, Operations, decommission, ops-codfw, Patch-For-Review: decommission betelgeuse.frack.codfw.wmnet - https://phabricator.wikimedia.org/T206870 (Papaul) [14:52:44] hey jgleeson_ [14:52:47] you around? [14:52:56] yeah just on 1:1 with mepps [14:53:03] I think I figured out the google sheets issue [14:53:47] go to sharing on the master exchange rate doc and switch it to "anyone with the link can edit" that should solve the permissions issue [14:54:01] for copies of the spreadsheet [14:54:44] it might also work with "anyone with the link can view" [14:55:32] yup I just confirmed. view works too [14:56:23] no rush sorry to interrupt 1:1 [14:59:29] ah interesting dstrine [14:59:42] if I remember rightly I think it was set to "anyone can view" [14:59:50] I'll try it out [14:59:52] I have 2 test spreadsheets I can show you at standup [14:59:53] thanks! [15:00:21] I was just testing out an alternative approach using caching to stem the number of google requests but if your idea works that will be even better [15:07:40] dstrine, still asking for auth :( [15:07:48] :/ [15:08:00] https://docs.google.com/spreadsheets/d/1UtDrb4GldCt4XJ5kd4gkYHDFWMz8yuKO8bstNqtWBuY/edit#gid=1483315145 [15:08:03] that's the copy [15:08:13] I didin't want to interrupt if you are in a meeting. Let me know when you are free [15:08:26] and the master doc is here https://docs.google.com/spreadsheets/d/1ys-6YCbmQKbeoNrwyj2uVAxKSGoYAdCptPLjMLmODj4/edit#gid=1943832082 [15:08:32] I've finished my 1:1 [15:08:37] free now [15:09:16] ok one second [15:11:09] jgleeson_: wanna meet in the standup hangout? [15:11:15] sure be 1 min [15:27:48] (PS3) Krinkle: Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) [15:28:06] (PS4) Krinkle: Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) [15:30:29] (CR) jerkins-bot: [V: -1] Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) (owner: Krinkle) [15:39:34] brb [15:44:44] (PS5) Krinkle: Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) [15:48:24] (CR) jerkins-bot: [V: -1] Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) (owner: Krinkle) [16:08:04] awww man, dstrine we were deceived by our eureka moment. turns out that the copies worked, because we had previously auth'd the original document and google is clever enough to know when you make new copies of things that require previously granted permissions not to prompt you again. However, this still means new users of the expense form will have to auth the importrange at least once :( [16:08:18] some of that is covered in the notes here: https://support.google.com/docs/answer/3093340 [16:09:58] so it looked like once we'd auth'd the first client document to the intermediate, and created susequent copies, we had cracked it. What was actually happening is that google knew we'd established the link already and is reusing it :\ [16:10:27] but each person who copies the document will have to do that at least once from what I understand [16:10:44] from google, Spreadsheets must be explicitly granted permission to pull data from other spreadsheets using IMPORTRANGE. The first time the destination sheet pulls data from a new source sheet, the user will be prompted to grant permission. [16:11:35] all is not lost, I think the caching approach might actually work after briefly testing it just now. [16:12:03] jgleeson: but by that reasoning, if we authorize the template expense report, anyone who makes a copy of the template should be fine? [16:12:38] the auth is individual to the user, e.g. me and you, not the doc [16:13:04] so once we auth it, we can create multiple copies and it works [16:13:14] but each new person will have to establish that first link [16:13:21] does that make sense? [16:13:44] ah yeah so we need to add instructions to the expense report [16:13:56] this is still way easier than last year [16:14:08] if you compare the older process [16:14:34] hmm I think that is one option but let me try out the caching of the API call results, which is this works will mean that googles background calls DON'T hit Oanda and instead just pull a cached result from the original call [16:14:58] ok [16:15:06] but yeah, if the cache options doesn't work, we could ask people to auth and dress that up to make it easy to do [16:15:22] which if* [16:17:01] (CR) Hashar: [C: +2] Add phan [extensions/FundraisingTranslateWorkflow] - https://gerrit.wikimedia.org/r/505840 (owner: Umherirrender) [16:29:33] fr-tech i'll be 3 min late [16:32:57] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, and 3 others: Update opt-in queue consumer to be able to create new donor records - https://phabricator.wikimedia.org/T217710 (XenoRyet) a:... [16:46:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Add ability to search Mailing Events in Civi - https://phabricator.wikimedia.org/T221874 (RLewis) [17:00:39] ok fr-tech, I've added some instructions to the 3rd row here https://docs.google.com/spreadsheets/d/1E-9Emnx9wAMUTXl0_LcBwPXmLGwCeiXCP-OU8lETw70 [17:00:44] does that work for you all? [17:01:54] I think you'll all see 'REF#' or something in the boxes affected, until you auth [17:01:56] could you color in that box? [17:02:17] better? [17:02:42] Looks like it works fine for me. [17:02:49] you should now be able to edit the doc mepps [17:02:57] colour to your hearts content :) [17:03:10] (I'm colour blind..) [17:05:02] okay cool, done [17:05:12] awesome! [17:05:13] thanks [17:05:23] could we make the font on the instructions a little bigger too? or was that a request from finance [17:05:31] it's probably okay at this point actually [17:05:43] thanks jgleeson [17:06:12] I haven't played with the colours or fonts, Finance have looked after that but I'm sure they'd be happy to make it more prominent to avoid user confusion [17:07:09] I'll wait for dstrine to test it out and if we're all happy, I'll start putting together the various pieces into a patch commit message to allow review [17:07:15] okay i just increased the font a little bit [17:07:22] and around that time I guess we can reach back out to finance [17:08:01] mepps, hmm that orange cell now hides some of the instructions for me [17:08:08] not for you? [17:08:41] "we need a bigger arrow" [17:08:53] feels like that's a line from moby dick [17:09:09] s/arrow/harpoon/ [17:09:21] or jaws? [17:09:28] I digress ... [17:09:29] how about now jgleeson? [17:09:36] perfect mepps ! [17:09:37] i edited a text a tiny bit, hope that's okay! [17:10:09] great [17:10:53] we're gonna need a bigger boat... [17:10:56] that was it [17:11:28] cool, thanks mepps and XenoRyet. I'll check back in later and if everyone is happy I'll wrap it up [17:52:19] PROBLEM - check_procs on frdb1001 is CRITICAL: PROCS CRITICAL: 1114 processes [17:57:23] RECOVERY - check_procs on frdb1001 is OK: PROCS OK: 249 processes [18:01:19] (Merged) jenkins-bot: Add phan [extensions/FundraisingTranslateWorkflow] - https://gerrit.wikimedia.org/r/505840 (owner: Umherirrender) [18:02:30] (CR) jenkins-bot: Add phan [extensions/FundraisingTranslateWorkflow] - https://gerrit.wikimedia.org/r/505840 (owner: Umherirrender) [18:18:00] fundraising-tech-ops, Operations, decommission, ops-codfw, Patch-For-Review: decommission betelgeuse.frack.codfw.wmnet - https://phabricator.wikimedia.org/T206870 (Dzahn) [18:22:29] fundraising-tech-ops, Operations, decommission, ops-codfw, Patch-For-Review: decom rigel.frack.codfw.wmnet - https://phabricator.wikimedia.org/T202535 (Dzahn) [x] confirmed gone from puppet repo [x] remove prod DNS [x] remove mgmt DNS [18:28:13] fundraising-tech-ops: replace betelgeuse, for it is ancient - https://phabricator.wikimedia.org/T203484 (Papaul) [18:28:20] fundraising-tech-ops, Operations, decommission, ops-codfw, Patch-For-Review: decommission betelgeuse.frack.codfw.wmnet - https://phabricator.wikimedia.org/T206870 (Papaul) Open→Resolved complete [18:28:55] fundraising-tech-ops, Operations, ops-codfw, Patch-For-Review: Rack/Setup frbast2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T196417 (Papaul) [18:29:00] fundraising-tech-ops, Operations, decommission, ops-codfw, Patch-For-Review: decom rigel.frack.codfw.wmnet - https://phabricator.wikimedia.org/T202535 (Papaul) Open→Resolved complete [18:47:06] Fundraising-Backlog: donate.wiki: fix visible template for recurring link - https://phabricator.wikimedia.org/T221885 (MBeat33) [18:58:40] (CR) Krinkle: [C: -1] "-1 for display.js issue." (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [19:18:27] (PS7) D3r1ck01: CentralNoticeHooks: Bundle configuration vars into JS that needs it [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) [19:21:59] (CR) D3r1ck01: CentralNoticeHooks: Bundle configuration vars into JS that needs it (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [19:43:18] (PS6) Krinkle: Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) [19:44:01] (CR) Krinkle: CentralNoticeHooks: Bundle configuration vars into JS that needs it (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [19:44:44] (PS5) Ejegg: Start recording the payment processor id for recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501128 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [19:46:15] (CR) Ejegg: [C: +2] "This is great! Let's make another little change in the other place we add these records, wmf_civicrm_message_contribution_recur_insert in " [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501128 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [19:50:42] (Merged) jenkins-bot: Start recording the payment processor id for recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501128 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [21:39:45] (CR) jenkins-bot: Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T217565) (owner: AndyRussG) [21:41:18] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 8 others: Reduce recurring TY emails - https://phabricator.wikimedia.org/T213209 (Ejegg) Looks l... [21:47:33] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 8 others: Reduce recurring TY emails - https://phabricator.wikimedia.org/T213209 (TSkaff) Waitin... [21:57:18] Fundraising-Backlog: Unfindable donor-facing error ref #s - https://phabricator.wikimedia.org/T221386 (Ejegg) Looks like a combination of the IP velocity filter and the utm_campaign filter hit them. The associated IP address seems to belong to a government agency, and shows up a lot of times in the logs for... [22:00:12] (CR) Ejegg: [C: +1] "Looks fine, but you're not using getCanonicalUrl anywhere. Do you have plans for it?" (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/497223 (https://phabricator.wikimedia.org/T217565) (owner: AndyRussG) [22:07:35] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 8 others: Reduce recurring TY emails - https://phabricator.wikimedia.org/T213209 (Ejegg) Sure th... [22:22:47] ejegg: I’ll look at those recurrings again soon - will start with ‘cancel_reason’ & see if I can pass it through - I’d ‘like’ to clean up the completed too but perhaps the installments part pushes it into too hard? [22:47:21] MBeat: we made some changes to the dedupe script which caused it to pick up a few more but to move the pointer back - it moved back to contact id 26.5m yesterday & it’s now on about 27.5m - I can reset the pointer to 31.8m - which is about what it should be - but it’s probably picking up an handful each run & if we let it catch up naturally might merge a low 4-figure number of dupes. - probably by early next week it would have d [22:47:22] that - but it means it’s not doing the very latest in the meantime - how does that trade-off work for you? [22:48:13] hi eileen is this script-related only, or would it affect the manual, query-based workflow as well? [22:48:49] it’s the script that runs in the back ground [22:49:06] normally when you load a page it has already merged any the script can ‘ just do' [22:49:35] but if the script is a bit behind then that might not be the case (although doing a batch merge in the UI has the same impact) [22:49:49] On a side note I’m noticed something [22:50:05] how long would you estimate until the script resumes having a go at the newest CIDs? [22:50:20] well it got through about 1m yesterday [22:50:44] so my best guess is 4-5 days [22:51:57] i think that’s fine, we can treat it like a test and see if the bulk email complaint from recent NL donors gets a bump - ty for the heads up [22:52:23] OK - if it causes probs & think I could do some things to speed it up [22:52:36] but it’s catching a small number of extras [22:59:21] eileen: sorry, just saw your message. So the completed bit doesn't seem that hard, really [22:59:47] ejegg: ok - it would be nice to do it - it has caused us those upgrade issues in the past [22:59:59] I guess we just need to fix the two places we're searching for recurring donations to charge [23:00:09] which would be the smashpig extension [23:00:15] and the recurring_globalcollect module [23:02:51] and... it looks like we ARE incrementing the installments each time, in the smashpig extension, but we're not actually depending on it for the API call. [23:03:15] So I think we can just stop doing that, and reset installments to zero [23:08:20] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: New dedupe blocking issue - on hold - https://phabricator.wikimedia.org/T221914 (Eileenmcnaughton) [23:11:00] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (Eileenmcnaughton) @LeanneS I gave it another kick -only ones the script can't merge rem... [23:15:37] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (Eileenmcnaughton) Also I see 750 in the other list civicrm/contact/dedupefind?reset=1&r... [23:17:20] ejegg: OK - I’m digging now - any chance you can check out this extension https://phabricator.wikimedia.org/T115536 [23:17:26] (PS3) Ejegg: Update extended report [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506049 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [23:17:41] sure thing eileen ! [23:18:08] (CR) Ejegg: [C: +2] Update extended report [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506049 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [23:19:16] ejegg: I love the drama of the original phab by adam - https://phabricator.wikimedia.org/T115536 [23:19:43] someone despite it absolutely killing our users it languished for years... [23:19:45] eileen: ok, the getPaymentsToCharge fn in the SmashPig extension is fine with charging 'Pending' [23:19:59] haha [23:20:06] ejegg: cool - I guess it should really be In Progress per yesterday [23:21:06] the extn does set the contribution_recur row to 'In Progress' when it's actually making the API calls to charge the next monthly payment [23:21:20] ejegg: ah fine [23:21:23] but let's see, I guess it sets it back to 'Completed' once it's done [23:21:29] not so fine [23:21:34] So we should change that to set it back to 'Pending' [23:21:43] I guess it’s trying to prevent a double charge though [23:21:58] yep [23:21:59] ‘In Progress’ means ‘one or more payments has been received’ [23:22:18] So perhaps we set it from In Progress to Pending when we are trying another payment? [23:22:18] so that status change would be in the 'run' function of SmashPigRecurringProcessor.php [23:22:43] wait, now I'm confused, what does 'Pending' mean? [23:23:27] (Merged) jenkins-bot: Update extended report [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506049 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [23:25:09] ejegg: so in core Pending means ‘recurring created, no payment received yet' [23:25:27] In progress means ‘one or more payments received but more expected’ [23:25:38] completed means ‘no more expected' [23:25:49] ‘cancelled means ‘user cancelled or we’ve given up’ [23:26:03] Hmm, any status for "we're charging a payment right now, don't try charging another one till we're done"? [23:26:43] not really ... [23:26:59] I suspect the right answer is to move next_sched_contribution_date first [23:27:13] & perhaps use the failure_retry_date if we are thinking about that [23:28:41] hmm, looks like we haven't used that yet, but that would be a good one [23:28:58] yeah - the other option might be to add a ‘Processing’ status [23:29:11] That would be cool [23:29:36] OK, so for failures, it looks like we just bump the next_sched_contribution_date up by a day or so [23:29:41] I don’t think there is anything stopping us from adding that via SmashPig - although I would try to upstream it [23:29:51] after the fact [23:30:28] so we could bump the sched date first & then if we record a fail populate failure retry [23:30:39] & process those in the same job or a separate one [23:31:13] a separate one might be good in that if there was something weird going on we could turn if off separately [23:36:43] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506572 [23:37:54] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506572 (owner: Eileen) [23:38:33] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506572 (owner: Eileen) [23:38:54] (PS9) Ejegg: Add extension to cleanup sort name for orgs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) (owner: Eileen) [23:39:48] !log civicrm revision changed from 519fe8028e to 88736c7c11, config revision is 2119df9495 - deployed patch to start recording payment_processor_id on recurring [23:39:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:40:26] (CR) Ejegg: [C: +2] "What a beautiful box! Lovely ribbons. And such pretty tissue inside. It's... a _pre hook!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) (owner: Eileen) [23:41:13] ejegg: lol - are you saying ‘that’s a lot of code for just one hook’ :-) [23:41:19] Srsly, though, I imagine that extn will be useful to plenty of other orgs :) [23:41:50] well I also wondered if we would later extend to strip suffixes [23:42:04] because it allows sort-name deduping [23:42:04] Ah yeah, like 'Inc' [23:42:32] And the configurability will let folks use it for other languages [23:42:35] other orgs might find that the search is ‘killing us’ :-) [23:43:26] maybe I should do a mini blog announcement on it [23:43:41] once we’ve kicked the prod tyres on it [23:44:17] It had to wait on the civi update because I had to patch core to not overwrite the sort_name again [23:44:30] (I put a unit test on core to stop it from reverting too) [23:45:21] nice [23:45:39] (Merged) jenkins-bot: Add extension to cleanup sort name for orgs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) (owner: Eileen) [23:46:57] Fundraising Sprint A series of unfortunate event handlers, Fundraising Sprint Bert and Ernie's Excellent Adventure, Fundraising Sprint Casino Royale With Cheese, Fundraising Sprint Da Vinci Coder, and 10 others: find another way to compile the donor list fo... - https://phabricator.wikimedia.org/T118822 [23:47:18] eileen: that getoptions call knows to filter out the test processors? [23:47:39] I think it prepends ‘test’ to the name [23:47:55] oh, that's not the case for me locally [23:48:03] hmm [23:48:59] or.. you mean the getoptions call does that internally? [23:49:21] I was just looking in the civicrm_payment_processor table and noticed that it has two of each name [23:50:18] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506577 [23:50:27] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506577 (owner: Eileen) [23:50:43] ejegg: but getoptions is not adding in the prefix [23:51:13] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506577 (owner: Eileen) [23:51:41] https://github.com/civicrm/civicrm-core/pull/13698 [23:53:08] ok, just trying the api call from drush, it seems to only return the non-test ones [23:55:02] (PS3) Eileen: Update existing recurring records to hold payment processor ids [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501132 (https://phabricator.wikimedia.org/T218616) [23:58:04] tbh - I’m not sure that’s the intention - but given it’s just an upgrade patch I won’t dig further at this stage - see https://github.com/civicrm/civicrm-core/pull/13698/files [23:59:37] (CR) jerkins-bot: [V: -1] Update existing recurring records to hold payment processor ids [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501132 (https://phabricator.wikimedia.org/T218616) (owner: Eileen)