[00:01:54] Fundraising Sprint Deferential Equations, Fundraising-Backlog: Update fr-tech job roles on wiki - https://phabricator.wikimedia.org/T158710#3066232 (ggellerman) a:ggellerman>cwdent [00:06:23] Fundraising-Backlog, FR-PayPal-ExpressCheckout: IPN listener should normalize new PayPal recurring messages - https://phabricator.wikimedia.org/T157074#3066239 (Ejegg) [00:06:54] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Wikimedia-General-or-Unknown, Patch-For-Review, WMF-deploy-2017-03-07_(1.29.0-wmf.15): donationinterface_langonly removal from WMF cluster/CommonSettings.php - https://phabricator.wikimedia.org/T159098#3056705 (awight) Open>R... [00:07:27] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3047164 (DStrine) p:High>Normal [00:12:37] Fundraising-Backlog: Create fr-tech deadlines calendar - https://phabricator.wikimedia.org/T159177#3066251 (Ejegg) Things to track: - mediawiki version lifecycles - processor API deprecation [00:21:42] Fundraising Sprint E 2017, Fundraising-Backlog: Create fr-tech deadlines calendar - https://phabricator.wikimedia.org/T159177#3066261 (DStrine) [00:21:44] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog: PHP-Queue: remove IndexedFifo and KeyValue interfaces - https://phabricator.wikimedia.org/T159175#3066262 (DStrine) [00:21:46] Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3066263 (DStrine) [00:21:49] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog: Update fr-tech job roles on wiki - https://phabricator.wikimedia.org/T158710#3066264 (DStrine) [00:21:52] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: We still have contacts whose merge is blocked on mis-rounded geocodes - https://phabricator.wikimedia.org/T158271#3066265 (DStrine) [00:21:54] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: Stop creating addresses N0NE Provided & delete them where we have them - https://phabricator.wikimedia.org/T158268#3066266 (DStrine) [00:21:56] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Unplanned-Sprint-Work: Review fr-tech data retention guidelines - https://phabricator.wikimedia.org/T156317#3066269 (DStrine) [00:21:58] Fundraising Sprint E 2017, Fundraising-Backlog, FR-PayPal-ExpressCheckout: IPN listener should normalize new PayPal recurring messages - https://phabricator.wikimedia.org/T157074#3066268 (DStrine) [00:22:00] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, MediaWiki-Cache, and 2 others: Mediawiki namespace pages, including CentralNotice banners, are slow to save - https://phabricator.wikimedia.org/T158084#3066267 (DStrine) [00:22:03] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog: fill out PCI SAQ-A form for 2017 - https://phabricator.wikimedia.org/T155779#3066270 (DStrine) [00:22:05] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, and 3 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3066272 (DStrine) [00:22:10] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 5 others: [Spike] investigate contribution tracking data (was Engage import fail... - https://phabricator.wikimedia.org/T146295#3066275 [00:22:12] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 6 others: Fix Coinbase file to support importing UTM fields - https://phabricator.wikimedia.org/T153791#3066274 (DStrine) [00:22:14] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising Sprint Qwerty Thwacking, and 3 others: CentralNotice banner sequence: implement feature for MVP - https://phabricator.wikimedia.org/T144453#3066276 (DStrine) [00:22:17] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, and 5 others: Create an import method for matching gifts and payroll deductions - https://phabricator.wikimedia.org/T115044#3066278 (... [00:22:19] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint Dirt Farming, and 8 others: Store and update list of currenly working iDEAL banks - https://phabricator.wikimedia.org/T128692#3066277 (DStrine) [00:22:21] Fundraising Sprint Costlier Alternative, Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, and 2 others: Move gateway-specific normalizations out of recurring.module - https://phabricator.wikimedia.org/T107372#3066279 (DStrine) [00:26:22] awight: you see any problem with restarting jenkins? i'll try the prep for shutdown thing [00:30:37] cwd: "Can't hurt" :) [00:30:38] restart sounds like a good first thing to attempt [00:30:55] I'm gonna look through the syslog for anything interesting around the event times [00:30:57] cool just waiting on a couple runs to finish [00:31:26] I think prepare for shutdown actually does that [00:31:33] it just prevents new runs from starting [00:32:19] yep that's all i meant [00:32:24] couple of the long ones are still going [00:33:02] ok i think it's good now [00:37:05] cwd: Don't be distracted by the empty queue message--that's the second-to-last thing all of our queue runners are going to emit. [00:37:13] Cos they stop when the queue is empty. [00:37:27] oh gotcha [00:37:51] I think this might be a PHP fatal: "FATAL: null" [00:38:00] lovely error message tho [00:38:33] I was thinking it was a java null pointer error, but java is actually crashing with a java.lang.ArrayIndexOutOfBoundsException [00:39:48] awight: might it be complaining about the php error? [00:40:19] aww nuts, there's no "FATAL" nothing in SmashPig [00:41:43] It's not the MediaWiki fatal error handler I configured, either. [00:42:06] that's from Jenkins [00:42:30] e.g., unrelated bug which emits the same line, https://issues.jenkins-ci.org/browse/JENKINS-35259 [00:42:41] Great, so it means absolutely nothing. [00:43:13] java.lang.ArrayIndexOutOfBoundsException [00:43:13] at java.io.FileInputStream.readBytes(Native Method) [00:43:23] well on a brighter note none of the nextBuildNumber files are empty [00:43:31] so the safe shutdown must take care of that [00:43:33] oh. Maybe it's an edge condition where the file goes away as we're reading it. [00:43:42] great to hear. [00:43:58] I'm sure there's no CLI way to include that in the service stop hook. [00:44:08] heh [00:44:12] at least there's a way [00:44:14] Maybe there's a Swing UI :p [00:44:31] makes me slightly less depressed about maintaining jenkins for awhile :) [00:46:12] we are having some friends over to play cards so i've gotta bolt but i'll watch email and check on this again later in case it happens again [00:46:23] hopefully it was just bad state in the java machine [00:46:53] later! [00:47:21] Nothing helpful in syslog. [00:47:27] Okay don't lose the house! [01:01:49] Fundraising-Backlog, FR-PayPal-ExpressCheckout: Refine PayPal EC donation itemization - https://phabricator.wikimedia.org/T137730#3066401 (awight) This is looking good for one-time donations: {F5986471} {F5986474} [01:04:22] Fundraising Sprint Internet Exploring, Fundraising Sprint Jabberwock Slaying, Fundraising Sprint Killing Time, Fundraising Sprint Licking Cookies, and 6 others: [Epic] Support Express Checkout recurring donations - https://phabricator.wikimedia.org/T134446#3066403 (awight) We're still getting the... [01:06:48] another global collect fail - I'll check on it [01:07:38] Fundraising-Backlog, FR-PayPal-ExpressCheckout: Refine PayPal EC donation itemization - https://phabricator.wikimedia.org/T137730#3066405 (awight) The description string is not being translated correctly: {F5986794} [01:07:47] Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3066406 (Eileenmcnaughton) @RLewis is this enough of a fix - or do you need it on advanced search too? [01:08:01] Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3066407 (Eileenmcnaughton) a:Eileenmcnaughton [01:09:38] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, Epic: [epic] PayPal upgrade - https://phabricator.wikimedia.org/T87621#3066414 (awight) @Ppena Sorry if you've already addressed this, but I'm wondering if the plan is still to exclusively target JP/JPY, or if we should open it up to all... [01:09:52] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#3066415 (Eileenmcnaughton) [01:10:56] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising Sprint Unbreaking Now, and 3 others: Longterm fix + regression test for T144489 - https://phabricator.wikimedia.org/T144557#3066417 (awight) [01:10:59] Fundraising Sprint Deferential Equations, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, Patch-For-Review: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3066416 (awight) Resolved>Open [01:11:17] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3063312 (awight) [01:11:25] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create table to track all of these on live. As we resolve them we will delete from there, allowing us to see which have been tac... - https://phabricator.wikimedia.org/T159396#3066419 [01:14:46] awight: I'm just running the query to see which ones got caught [01:14:58] we can increase date range to 5 days probably [01:16:30] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3066454 (awight) This just broke again... ``` +------... [01:17:04] eileen1: I need to read the safeguard code before I can say much... [01:17:36] basically it does an independent calclutation of the next date & compares the 2 [01:17:48] cool, is_found_globalcollect_invalid_next_sched_dates [01:17:51] currently it expects them to be within 3 days of each other [01:19:03] Seems like the date will continue to diverge as we retry stuff from the 28th [01:19:09] ejegg[m]: commented on my last fix he would have preferred to loosen to 5 days [01:19:16] +1 [01:19:34] I think there must be a max difference that would be reached [01:20:24] OK - just got the results - 2 contributions are 4 & 5 days respecitvely out [01:20:35] oh, I got 3 contributions [01:20:39] //phabricator.wikimedia.org/T159298#3066454 [01:20:44] https://phabricator.wikimedia.org/T159298#3066454 [01:21:07] the retry thing is messing with my head... [01:21:32] those knobs are configurable [01:21:47] in the worst case, we try once per day, three times before giving up. [01:22:11] Actually, it could be worse: for example if we have to shut off the recurring processor on the 28th [01:22:48] not considering that yet, I guess the worst case is a Feb 28th contribution fails and is retried on the 1st and 2nd [01:22:50] so what has happened is the next date has been calculated at 2 April but we have set it to 2017-03-28 [01:23:03] just what I was thinking [01:23:07] so there is a 5 day discrepancy. I feel OK about just opening up the date range to 5 days [01:23:16] Yeah that sounds good [01:23:24] what we are worried about is it running twice too close togehter [01:24:02] It's alarming though, that if the job is turned off for any one day in that feb 28-mar 2 window, then the next date will be April 3, etc. [01:24:30] Anyway, I guess these are edge cases cos they must have failed. [01:24:33] Doesn't it do catch up? [01:24:58] It does, but the max(receive_date) will be Mar 3 for example [01:25:00] They didn't fail - its really a discrepancy between the check-script calculation method & our code method [01:25:13] for reasons which may or may not be good we set a cycle_day [01:25:32] & the check script has not assumed one - it just assumes that we add on a month [01:25:44] Agreed that it's a little silly to honor the cycle day if we never gave the donor a choice to set it directly [01:25:56] that part makes sense to me [01:27:41] (PS1) Eileen: Loosen comparison in validation script for processing global collect. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340673 (https://phabricator.wikimedia.org/T159298) [01:28:03] yeah - I we ARE doing that - but I have my doubts - that commit will just loosen the check a bit though [01:30:55] eileen1: I found something interesting in the logs, the tricky subscriptions aren't failing but they're staying in status 600 even after we DO_PAYMENT [01:31:17] ok - hmm [01:31:41] so I was understanding this as being a false alarm based on the check being wrong - is that not your analysis? [01:32:17] I think it's mostly that, but I'm trying to understand why we aren't seeing this error for *all* the subscriptions with cycle_day=feb 28 [01:32:45] the status 600 thing might have been a red herring, now I need to re-read the workflow for these charges [01:33:43] DO_PAYMENT, GET_ORDERSTATUS, SET_PAYMENT. [01:34:03] And now I see nothing unusual about the three contributions that were setting off the alarm. [01:34:52] is it the fact cycle_day is set for some & not others? [01:36:45] Pretty sure it's set for everything [01:37:23] So, for one of these the cycle_day is 29, donation went through on 29 Jan & then 2 March [01:37:35] so, calculated next date is 2 April [01:37:37] select count(*) from civicrm_contribution_recur where cycle_day=28 and contribution_status_id in (1); [01:37:40] -> 1210 [01:38:01] but cycle-date based calc is 2017-03-29 [01:38:25] hence the 2 calc methods are out by 4 days - but no cause for alarm [01:38:41] totally [01:39:06] but aren't you concerned about why the other 1,210 contributions with cycle_day=28 aren't setting off the alarm? [01:39:12] we presumably started trying to process it on 2017-02-28 [01:39:44] but it only actually processed on 2017-03-02 [01:40:30] so, I'm not sure the cause for that lag, but I think the fact that lag sometimes happens was the reason for there already being a few days leeway in the query? just not enough to deal with February [01:41:48] okay now I'm seeing that order_id 8516384301 did fail on the 28th [01:42:28] (CR) Awight: [C: 2] "Great, looks safe!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340673 (https://phabricator.wikimedia.org/T159298) (owner: Eileen) [01:43:01] the 5-day leeway is a nice start. It should at least reduce the number of edge cases we have to care about. [01:43:49] yeah - it will get us past this, without removing protection against them not being incremented more than a day or 2 & re-charging [01:46:14] (Merged) jenkins-bot: Loosen comparison in validation script for processing global collect. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340673 (https://phabricator.wikimedia.org/T159298) (owner: Eileen) [01:48:20] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340674 [01:48:22] Thanks again for having the foresight to add that alarm! [01:48:31] gotta hold a baby... [01:48:36] yeah - we know it can do false alarms! [01:48:40] lol [01:48:47] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340674 (owner: Eileen) [01:48:54] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340674 (owner: Eileen) [01:52:22] !log civicrm changed... [01:52:22] from 58c8c06eee577edd8caa4a33b2c48fbd7651a005 to fb91fa8abe11dab53402300fc84dd37f79b3c690 [01:52:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:52:51] I think the next run is in 10 minutes so we'll see if it recovers [01:53:57] nice. yeah I didn't manipulate those three subscriptions via db. [02:00:20] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create table to track all of these on live. As we resolve them we will delete from there, allowing us to see which have been tac... - https://phabricator.wikimedia.org/T159396#3066527 [02:00:47] d'oh, picked a bad workspace [02:01:14] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create table to track all of these on live. As we resolve them we will delete from there, allowing us to see which have been tac... - https://phabricator.wikimedia.org/T159396#3066528 [02:09:38] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Create table to track all of these on live. As we resolve them we will delete from there, allowing us to see which have been tac... - https://phabricator.wikimedia.org/T159396#3066540 [02:12:51] eileen1: looks like that worked! [02:14:18] yay [02:14:51] btw I don't get why the geocode thing needs re-running but it did run through for me on staging with that change [02:16:01] (CR) Ejegg: "awight, PS5 ought to ensure shared HashBag. Thanks!" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/339330 (https://phabricator.wikimedia.org/T128692) (owner: Ejegg) [02:16:47] hmm, are all the columns the right type now? [02:16:59] should be decimal, I guess? [02:17:24] yeah - well on staging it failed due to column lenght - ie. rerunning what should have already run on live - but my reading of the code said it shouldn't have [02:17:46] I'm a little inclined to accept 'that worked so it's all good' [02:18:17] hehe, ok [02:18:25] dang it! this cafe is closing too [02:18:51] gone from closing bars to closing cafes.... [02:19:01] I tried to ask if they were open an hour more. guess my español still needs work [02:29:08] (PS1) Eileen: Create table of blank addresses. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340675 (https://phabricator.wikimedia.org/T159396) [02:29:22] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Create table to track blank addresses created of these on live. As we resolve them we will delete from the... - https://phabricator.wikimedia.org/T159396#3066548 [02:43:56] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Delete blank addresses which have been created but have no other actions against them - https://phabricator.wikimedia.org/T159402#3066570 (Eileenmcnaughton) [02:50:59] (CR) Ejegg: Create table of blank addresses. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340675 (https://phabricator.wikimedia.org/T159396) (owner: Eileen) [02:55:21] (CR) Eileen: Create table of blank addresses. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340675 (https://phabricator.wikimedia.org/T159396) (owner: Eileen) [02:57:37] (CR) Ejegg: [C: 2] "K, guess there's no need for the suspenders!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340675 (https://phabricator.wikimedia.org/T159396) (owner: Eileen) [03:01:24] (Merged) jenkins-bot: Create table of blank addresses. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340675 (https://phabricator.wikimedia.org/T159396) (owner: Eileen) [03:17:06] Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Can we make the nickname field searchable? - https://phabricator.wikimedia.org/T158784#3066635 (RLewis) @Eileenmcnaughton I think this is fine as it's not a search we will do every day but good to know we can search this f... [03:36:01] (CR) Ejegg: [C: 2] "Looks good, merged upstream with unit tests" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/338834 (https://phabricator.wikimedia.org/T153917) (owner: Eileen) [03:37:58] sigh - I'm trying to test my fix on staging but I thought it was hung & killed it, but it had deleted a bunch of the addresses & now I need to get some sort of state to retry [03:40:47] (PS9) Ejegg: Fix typos and PHPdoc hints [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/312069 [03:41:00] (PS3) Ejegg: Prune unused log parameters [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/318987 [03:41:05] (Merged) jenkins-bot: CRM-20061 Add tables as a parameter on the revert api [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/338834 (https://phabricator.wikimedia.org/T153917) (owner: Eileen) [03:41:10] (PS3) Ejegg: Rename some impressionDiet variables [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/319003 (https://phabricator.wikimedia.org/T121178) [03:41:21] (PS3) Ejegg: Undo 'require' debug mode hack [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/319259 [03:44:19] ah dang [03:44:48] would be nice if we could kick off an overnight db restore for staging [03:45:35] yeah [03:46:23] I think there used to be a back on staging somewhere but not sure quite where [03:47:17] maybe cwd can +s us a script [03:47:58] well, it's pretty late here. guess I'll do the iDEAL deploy tomorrow [03:48:02] gnight! [03:54:27] night [04:12:01] (PS1) Eileen: Remove unchanged insertions of blank addresses. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340686 (https://phabricator.wikimedia.org/T159402) [04:15:38] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Revert updates that set valid addresses to blank, and have no further actions against them - https://phabricator.wikimedia.org/T159408#3066680 (Eileenmcnaughton) [04:16:46] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#2894976 (Eileenmcnaughton) I've created... [04:49:25] Fundraising-Backlog, FR-PayPal-ExpressCheckout, FR-Paypal, Epic: [epic] PayPal upgrade - https://phabricator.wikimedia.org/T87621#3066698 (Ppena) @awight PP EC seems to be a better experience overall (UI, reporting, and support), so we can start with JP/JPY (or any convenient country) and open it... [08:06:56] fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#2852630 (Joe) @Jgreen any idea when it will happen? (all FR to jessie, I mean). [14:30:36] fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3067834 (Jgreen) We have six Precise boxes left to replace by the end of March, most are waiting on procurement tasks. Once these are done we'll have all metric-worthy services on Jessie. T... [14:48:27] something's hanging up the imports... [14:51:27] dedupe insert going on 100 minutes. killed it [14:51:58] but it's been persisting in 'Killed' status for more than a minute [14:52:45] ah, there is goes [16:09:42] Fundraising Sprint Deferential Equations, Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Ingenico recurring schedule glitch found in the wild - https://phabricator.wikimedia.org/T159298#3063312 (Ejegg) Open>Resolved Looks like that fixed it! [16:09:44] Fundraising Sprint Rocket Surgery 2016, Fundraising Sprint Stirring The Pot, Fundraising Sprint Testing on Production, Fundraising Sprint Unbreaking Now, and 3 others: Longterm fix + regression test for T144489 - https://phabricator.wikimedia.org/T144557#3068058 (Ejegg) [16:26:12] (PS1) Ejegg: Update SmashPig, add caching libs [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/340763 [16:26:52] (CR) Ejegg: [V: 2 C: 2] Update SmashPig, add caching libs [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/340763 (owner: Ejegg) [17:08:44] Hi XenoRyet , hope you're feeling better [17:11:21] Eh, only about 60% back, but that still qualifies as better I suppose. [17:11:28] (PS1) Ejegg: Missed some files in the last commit [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/340772 [17:11:37] Honestly not all here today, but figured I better at least try to get some work done. [17:19:21] (CR) Ejegg: [V: 2 C: 2] Missed some files in the last commit [extensions/DonationInterface/vendor] - https://gerrit.wikimedia.org/r/340772 (owner: Ejegg) [17:20:24] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/340773 [17:44:35] hey XenoRyet I glad the trend if more positive but do what you have to to feel better [17:44:44] wow lots of typos [17:44:49] I think you get it [17:45:07] Heh, oddly enough I'm bleary eyed enough that I didn't notice the typos and read it perfectly. [17:45:16] But yea, I'll take it easy if I need to. [17:45:42] awight|semi: commented on a task saying he still saw the old UI for recurring but a few weeks ago you thought that was resolved. What have you seen recently? [17:46:47] I haven't looked at that recently. I checked back at the same time he said it was resolved and haven't really looked since. I seem to remember it sort of resolved on it's own, so I wouldn't be totally surprised at a regression. [17:47:45] XenoRyet: and awight|semi it might be good to just to have a chat and transfer knowledge [17:48:36] Yea, I'll see if I can hook up with him sometime later. [17:48:36] The rep told me that the new workflow was being rolled out incrementally, so our location and stuff factors into what we see. [17:48:52] And couldn't give a schedule for when it would be fully rolled-out. [17:49:16] It was nice at least to have someone confirm that the API parameters weren't the culprit, cos there are an infinite number of variations we can make there. [17:49:37] So we're kind of in a 'just wait longer' situation there? [17:51:30] Nah I think we'd better get some PayPal explanations for what we're seeing. [17:53:29] Haha I'm trying the production workflow and get language key placeholder madness [17:53:49] It's all the new flow. k then. [17:53:55] woot [17:55:54] Does anyone else see all lang=qqx language keys after the first screen in paypal? [17:55:56] https://payments.wikimedia.org/index.php?title=Special:PaypalExpressGateway&appeal=JimmyQuote&ffname=paypal&language=en&utm_source=Waystogive&utm_medium=sitenotice&utm_campaign=C11_Waystogive&gateway=paypal&returnto=Thank_You%2Fen¤cy_code=JPY&country=JP&uselang=en&amount=150&recurring=true [17:57:54] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/340773 (owner: Ejegg) [17:58:53] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/340773 (owner: Ejegg) [18:00:21] fr-tech: <<<<< EVACUATION ROUTE <<<<< [18:00:21] -- discuss. [18:00:30] awight|semi: it goes straight to the TY page after clicking 'agree and continue' for me [18:00:51] awight|semi: Same here. No lang=qqx anywhere that I see. [18:00:52] ejegg: awight|semi cwd and XenoRyet I just got this from MBeat hey david, https://phabricator.wikimedia.org/T159455 is kind of funky, with the email address populating as donor name across different payment methods. no idea how serious this is but thought i’d pull on your sleeve, no need to reply if you’re swamped [18:01:28] I didn't want to wait until standup to check on how bad this might be [18:02:33] hmm, that does look odd. I'll see if there's anything obvious [18:03:05] it stood out to me because it seems to be affecting multiple PsPs [18:03:43] but yeah, that stuff comes in through very different channels [18:04:00] Wow, slander can be strangely apropos [18:04:07] right... [18:04:41] I guess fortune -o is closer to reality than we'd like to think :p [18:04:57] damn we should probably spend some time on this failmail [18:05:04] K, I'll take screenshots of the ridiculousness I see in the PP flow. [18:05:13] it is boom times for breakage [18:05:57] It's the Japanese locale and my 2FA settings. [18:13:28] aha [18:18:08] Starting fr-tech-talk minutes just for gits and shiggles [18:18:38] dang, this is really odd. logs for the pp recurring id mentioned in the ticket show it preserved the name up to the Contribution (normalized) line [18:19:36] awight|semi: cool. I'm in the hangout [18:19:53] maybe we can puzzle over these logs together [18:21:28] I'm not feeling up for getting on the mic, but I'll lurk the etherpad [18:22:45] crap, this is still happening, to a large percentage of contacts [18:22:57] I'mma stop the queue imports for a bit [18:23:19] cwd / Jeff_Green: heads up, there will be some redis bloat [18:23:19] ejegg: heya, I'm hanging also [18:27:34] kk [18:38:44] (CR) Awight: Benevity import: ensure that the individual is soft credited when there is no individual part of the gift. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340271 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [18:45:40] (PS1) Awight: Fixup, assign name fields to the correct dictionary [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340792 [18:55:57] (CR) Ejegg: [C: 2] Fixup, assign name fields to the correct dictionary [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340792 (owner: Awight) [18:57:43] dang, cleaning this up is going to mean log parsing [19:00:06] noooo [19:00:12] argh [19:00:31] Can we get the info from audit files instead? [19:00:44] (Merged) jenkins-bot: Fixup, assign name fields to the correct dictionary [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340792 (owner: Awight) [19:01:05] hmm [19:01:10] In theory, it would be a better use of time to work on the audit processor refining and filling in missing information, IMO [19:01:20] meanwhile, lemme deploy that fix [19:02:19] (PS1) Awight: Fixup, assign name fields to the correct dictionary [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340799 [19:02:35] I shouldn't talk about a better use of time :) [19:02:45] It would probably be better if we all went on vacation [19:03:11] (CR) Awight: [C: 2] Fixup, assign name fields to the correct dictionary [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340799 (owner: Awight) [19:03:18] (Merged) jenkins-bot: Fixup, assign name fields to the correct dictionary [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340799 (owner: Awight) [19:04:06] well comrade [19:04:34] !log update civicrm from fb91fa8abe11dab53402300fc84dd37f79b3c690 to d012767f4f80ea6ca368d9c03aa84473b05ee980 [19:04:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:04:44] :D [19:04:58] hehe [19:05:03] ejegg: fix is deployed. Anything to turn back on? [19:05:17] yep, both the donations and recurring consumers [19:05:25] will do [19:05:26] oops, forgot to log their disablement! [19:06:30] !log reenabling donation and recurring queue consumers [19:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:06:36] oof thanks for mentioning that, i need to remember to log jenkins stuff [19:06:52] ty [19:07:06] It's wicked helpful when trying to understand the logs a month from now. [19:07:19] yes [19:09:47] Confirmed that the names are showing up again. [19:10:29] woohoo [19:10:51] so, we could write a script that uses globalcollect_audit_get_log_data_by_order_id to fix the glitched ones [19:11:21] well, that and the equivalents for the other processors :( [19:14:43] 8600 of em to fix [19:17:02] dang, what happened? [19:19:42] cwd little mistake in a patch left the first and last names unset for most incoming donations over the last day [19:21:15] awight|semi: so, how about a drush command that uses the audit processor log reading functions to populate a db table with ct_id + donor info? [19:22:00] d'oh, paypal parser is not php [19:22:21] ah, but also isn't reading our logs [19:26:11] What about adding a flag to the audit parsers which causes them to not check for CRM.exists? [19:26:29] Then we can do all the merge magic in a uniform way, in wmf_civicrm [19:27:19] that sounds nicer [19:27:28] We can write the merge stuff in an incremental way [19:27:35] Could leave it off by default [19:27:59] But for today, write a thing which will just merge the names in, if contribution exists and contact name is empty. [19:28:30] strike "leave it off by default" above. This sounds always-useful. [19:28:54] Maybe once we've backfilled, we leave the audit processors checking for crm.exists however [19:29:02] mmm. [19:29:09] Maybe we leave that on, too. [19:29:21] It should exactly double our volume, which is never a problem outside of December [19:32:11] ok, so... wmf_civicrm is checking for duplicates in verify_message_and_stage [19:32:46] so we fetch more fields and only throw the exception when there's no change [19:38:16] I think something like that [19:43:03] (PS1) Ejegg: WIP only cry DUPLICATE if there's no change [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340810 (https://phabricator.wikimedia.org/T159455) [19:46:43] (CR) jerkins-bot: [V: -1] WIP only cry DUPLICATE if there's no change [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340810 (https://phabricator.wikimedia.org/T159455) (owner: Ejegg) [19:49:35] (CR) Awight: "I like it! IMO, we should actually throw the exception only if source_type is identical, whether or not the other contrib details are sam" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340810 (https://phabricator.wikimedia.org/T159455) (owner: Ejegg) [20:09:25] Sorry all, totally distracted with wikimedia-l [20:09:38] Heh, I was just falling down that hole myself. [20:10:22] Madness that way lies. [20:11:06] Indeed, but compelling and engaging madness. Therein lies the danger. [20:13:52] hehe, well said. [20:13:58] I'm doing my best to not reply. [20:14:15] That's not true. I've written a reply and I'm doing my best to sharpen the corners ;) [20:37:53] hey awight I see a regression happened off my change - is it sorted [20:46:56] Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Delete blank addresses which have been created but have no other actions against them - https://phabricator.wikimedia.org/T159402#3068941 (DStrine) [20:47:05] Fundraising Sprint E 2017, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Revert updates that set valid addresses to blank, and have no further actions against them - https://phabricator.wikimedia.org/T159408#3068942 (DStrine) [20:49:50] eileen1: Yes indeed, it was a one-word change :) [20:49:59] We're figuring out how to deal with the backfill however. [20:50:15] aargh, standup! [20:50:28] :-( [21:26:31] Oversharing, but--I need to do a round of poop scooping while the sun is up [22:14:25] gotta run, will restore civi db this eve [22:28:41] (PS1) Ejegg: WIP Test more fields on import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 [22:31:38] * ejegg needs to read up on civicrm_api3 related-field-getting [22:32:31] (CR) jerkins-bot: [V: -1] WIP Test more fields on import [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340885 (owner: Ejegg) [22:32:46] oh hey, it did fail on prod. as it was supposed to [22:32:50] but... not locally [22:33:10] * ejegg needs to clear out his civi cruft [22:35:37] oh hey, we can salvage the names from the db! [22:36:03] sort_name and display_name were still populated [22:36:13] so, just some string splitting in sql [22:36:26] nowhere near as bad as feared [22:37:13] but also: no longer have an excuse to do the progressive enhancement work :P [22:40:17] That was a nice (nested) parenthesis... [22:40:22] I was sitter-napped by the neighbor and press-ganged into playing Adult for an hour. [22:50:38] (PS1) Ejegg: Fix missing names [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340891 (https://phabricator.wikimedia.org/T159455) [22:50:50] awight: I think that should do it ^^^ [22:51:14] scooping in dangerous territory, huh? [22:53:50] hmm, still a few in there with just email addresses. let's see what's up with them [22:54:15] oho [22:54:56] Just saw your comment--awesome! [22:55:07] display_name? [22:55:21] Great shortcut :D [22:57:00] ah, is_deleted=1 on those [22:57:27] awight: sort_name actually coz delimited with a comma instead of space, but yeah, that seems to work! [22:58:15] ok, is_deleted=1 on /some/ of those [22:58:35] but the vast majority have fname & lname [22:58:46] err, one sec awight, going to change a bit [23:00:44] (PS2) Ejegg: Fix (most) missing names [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340891 (https://phabricator.wikimedia.org/T159455) [23:01:12] k, that one won't stuff fname and lname with junk if sort_name is useless ^^^ [23:03:04] leaves us with 44 rows that don't have good info, i'll root about in the logs for answers on them [23:04:47] ooh nice one wrt sort_name [23:06:00] ejegg: Why the is_deleted check? [23:08:43] awight: eh, no good reason [23:09:18] i guess not to confuse history [23:10:03] well phooey, the remaining baddata rows don't show much of a pattern [23:10:28] i guess they tend toward recurring paypal, but there are a bunch of other gateways [23:11:41] A variety of reasons the names were bad, you're saying? [23:11:55] Cos I would believe we aren't using all the available recurring keys effectively. [23:12:42] Mmm. XenoRyet|nope ejegg is the plan to use Transformers for the recurring normalizations? [23:12:43] oho, they were initially created with the useful sort name, but I think civi updated it to just the email [23:12:56] snap. useful subtask! [23:13:01] Anything I can do to be helpful? [23:13:36] awight: I had a horrible thing with static inheritance normalizing the paypal messages [23:13:57] please dynamite it [23:14:00] I think the audit auditor task is still worth writing--shall I? [23:14:14] ejegg: Sounds lovely :-) [23:14:16] awight: ah right, sorry I didn't get to that [23:14:48] My argument is, cos often we do encounter data corruption that can only be recovered through logs or audit. [23:16:39] new version of fix incoming [23:16:52] awight: yeah, definitely worthwhile work [23:17:02] at least the merge foundation [23:17:05] I had a thought while sitting, which seems less compelling now--we could save all incoming data associated with contributions in a raw form somewhere. [23:17:28] like not deleting pending db rows? [23:17:35] Then run over it in batches if we can refine or for backfill needs. [23:17:36] or only deleting on fail [23:17:38] yah [23:17:44] exactly, the latter. [23:18:01] We would have raw IPN, get_orderstatus, audit, etc. [23:18:51] Everyone with Civi access could help us troubleshoot API things [23:19:28] Nice to be able to discuss this sort of scheme now that we're taking data retention seriously :) [23:21:06] (CR) Awight: [C: 2] "That looks grammatical!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340891 (https://phabricator.wikimedia.org/T159455) (owner: Ejegg) [23:25:20] (Merged) jenkins-bot: Fix (most) missing names [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340891 (https://phabricator.wikimedia.org/T159455) (owner: Ejegg) [23:25:43] (PS1) Ejegg: Extend fix to contacts that have been touched [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340897 (https://phabricator.wikimedia.org/T159455) [23:26:06] awight ^^^ should fix the ones that civi clobbered, too [23:26:41] ejegg: Don't you need an ordering on the joined log table? [23:26:57] ehh, any row with a comma works :P [23:27:04] cool, right! [23:28:13] (CR) Awight: [C: 2] "These log tables are the eighth wonder of the world. Nice usage!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340897 (https://phabricator.wikimedia.org/T159455) (owner: Ejegg) [23:28:29] yay! [23:28:35] will deploy and run as soon as that merges [23:29:17] oh hmm, I think eileen was holding off on running a db update [23:29:49] but... I think we batched it nicely, and Jeff_Green said it was OK if we thought we dealt with replag [23:29:56] That sounds right. [23:29:58] so I'll just turn off dedupe while that one runs [23:30:14] err the presence of a conflicting db update sounds right. I have no idea if it's ready to run. [23:30:36] You could set schema_version manually and still let eileen run the earlier update separately. [23:30:57] or hey, use update_7465 [23:31:04] eileen and jeff emailed about it last evemorn [23:31:09] haha [23:31:13] first I've heard that [23:31:30] heh, it's not mornfternoon if it straddles the dateline the other way, right? [23:32:01] Now if only we could perfect technology so that individuals could actually be in simultaneous locations [23:32:17] (Merged) jenkins-bot: Extend fix to contacts that have been touched [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/340897 (https://phabricator.wikimedia.org/T159455) (owner: Ejegg) [23:34:27] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340899 [23:34:33] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340899 (owner: Ejegg) [23:36:33] oh nice, screen is available on crm host for running long updates [23:42:20] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/340899 (owner: Ejegg) [23:42:42] !log disabled dedupe jobs for civi update [23:42:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:43:44] !log updated civicrm from d012767f4f80ea6ca368d9c03aa84473b05ee980 to 2d1de872d5b3ef81a1f2918e73dd7088af9844ed [23:43:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:44:20] k, i'm re-running the geocoding [23:49:20] !log running batched geocoding update and donor name fixes [23:49:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:51:33] huh, looks like there's still a tetch of work left to tag ec donations as such. this is marked as paypal but should be paypal_ec: https://civicrm.wikimedia.org/damaged/13992 [23:53:34] grr, still getting AUTHORIZE_XXX order ids via Amazon IPN. I thought we were looking up the real IDs now... [23:55:12] I hadn't touched recurring, I think XenoRyet|sortof has a WIP for that [23:56:25] (PS1) Ejegg: Split error check into own function [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/340905 [23:58:32] (CR) Awight: [C: 2] "Nice one." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/340905 (owner: Ejegg)