[00:02:19] eileen: sorry, I can do that +2 now [00:02:32] cool - I just wasn't sure where you were at [00:02:34] (03CR) 10Ejegg: [C: 03+2] Fix another cache miss [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826402 (owner: 10Eileen) [00:02:40] queue has grown :-) [00:02:50] (yes i did get called away all of a sudden) [00:02:55] (03CR) 10Eileen: [V: 03+2] Fix another cache miss [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826402 (owner: 10Eileen) [00:03:58] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826406 [00:04:08] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/826406 (owner: 10Eileen) [00:04:36] whoa - I don't miss having to then do a submodule commit & then wait for that & then merge to deploy [00:04:52] :) :) [00:08:23] !log civicrm upgraded from ff9b377d to a31c7590 [00:08:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:08:54] !log config revision changed from ab95bc89 to 2d10cc5f [00:08:54] queues re-enabled [00:08:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:13:30] hmm - the jobs should be on now - did I mess up [00:13:40] i was just logging on to look [00:14:21] the yamls on civi are still with the #schedule [00:14:26] so something didn't connect [00:14:52] yeah. i don't see the fcu line for the new revision from the irc bot [00:15:07] redoing it now [00:15:38] ok thanks - was just gonna re-check [00:15:39] !log fundraising scheduled jobs re-enabled [00:15:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:15:53] my civi patch is out though? [00:16:10] ok, that time the queue schedules did get updated on civi1001 [00:16:14] let me look at the civi file [00:16:35] yeah it looks like it did [00:16:38] haley_: back on [00:21:11] odd. the queue jumped up. [00:21:43] UCT just rolled over midnight 20 min ago [00:22:01] so the first batch of scheduled charges would have been dumped on the queue [00:23:18] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 100 days 10 hours - memory use is 3.96M (peak 121.65M, 0.13% of max, fragmentation 2.74%), connected_slaves is 3, donations is 2755, jobs is 0, jobs-adyen is 0, jobs-paypal is 8, payments-antifraud is 11, payments-init is 10, pending is 0, recurring is 4, refund is 4, unsubscribe is 29 https://icinga.wikimedia.org/cgi-bi [00:23:18] extinfo.cgi?type=2&host=frqueue1003&service=check_redis [00:24:47] huh, processing rate is a bit down [00:25:30] ah. makes sense [00:25:59] caches cold? [00:26:32] eileen: the previous performance fix made it over to the new civicrm folder? [00:26:37] I forgot to check on that [00:27:24] yep - looks ok [00:29:11] ah yeah, the OptionValue fix is there on live [00:30:21] I'm assuming once caches are warm & averageing is not including an off-time it will improve [00:30:58] huh, we're on the 4th queue run and still not so good timings [00:31:10] should have warmed the queue by now! [00:31:17] err, warmed the cache [00:33:11] ok, there are some deadlocks in there [00:34:19] but those are all in the first queue run [00:34:36] in the first 20 sec, even [00:34:48] hmm [00:35:00] oh yay, latest run is back up to 7 per sec [00:35:54] yeah - not sure what to make of that [00:36:04] how does it average - do you know? [00:36:58] for the import times, it runs the clock for each stage for each donation, then does a mean for each stage across all donations at the end of the queue consumer run [00:37:16] if cleared out four k in fifteen mins from the looks [00:37:24] just looking at the donations/second thing - not sure if that counts donations discarded as duplicate queue messages [00:42:46] ok. looks like it is good to go. i'm going to step away and have dinner with the family. feel free to reach out on phone/sms/signal/whatsapp/etc if you need anything. [00:43:11] ejegg: yeah I think we need to leave it a while & not draw any conclusions yet [00:43:44] there is no logical change - other than that cache fix [01:00:31] 10Fundraising-Backlog: Braintree donation do not pass PayPal Transaction Fee as paypal did "Fee Amount" at civicrm - https://phabricator.wikimedia.org/T316180 (10AnnWF) [01:19:09] (03PS1) 10Ejegg: Silverpop has_recur: require donation within 60 days [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826410 (https://phabricator.wikimedia.org/T315102) [01:21:19] (03CR) 10CI reject: [V: 04-1] Silverpop has_recur: require donation within 60 days [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826410 (https://phabricator.wikimedia.org/T315102) (owner: 10Ejegg) [01:22:08] looks like a syntax check got more strict? [01:24:16] (03PS2) 10Ejegg: Silverpop has_recur: require donation within 60 days [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826410 (https://phabricator.wikimedia.org/T315102) [01:24:18] (03PS1) 10Ejegg: Fix 'return' syntax [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826411 [01:24:51] i somehow doubt that time_util is actually used in anything these days [01:52:33] (03CR) 10Ejegg: [C: 04-1] "After looking into the $force behavior in session_resetForNewAttempt, I realize that function is regenerating the order_id even when $forc" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) (owner: 10Damilare Adedoyin) [01:54:44] (03CR) 10Ejegg: Pass merchantAccountId with currency to support multi currencies (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/822192 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [01:58:14] (03CR) 10Ejegg: "If you don't want to throw an exception, how about adding a 'validateParams' function that could return an array of invalid fields? The to" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/822192 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [02:02:07] sounds like a good approach, will do [02:57:52] (03PS1) 10Eileen: Do not exclude CiviCRM files from git [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826414 [02:58:11] (03CR) 10Ejegg: "A few comments from a first look over the code" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/822191 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [02:58:27] Just did a gitignore fix - I think that is the last thing before closing that repo phab [03:00:35] eileen: oh weird, the 'vendor' pattern is matching drupal/sites/default/civicrm/vendor? [03:01:04] maybe better to use the full path in the ! line ? [03:01:21] ok, really need to be |away [03:02:47] (03PS2) 10Eileen: Do not exclude CiviCRM files from git [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826414 [03:02:49] opps that was wrong [03:02:55] sites/all/modules/civicrm [03:03:36] (03PS3) 10Eileen: Do not exclude CiviCRM files from git [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826414 [03:03:45] yep - I just tested & we do need the full path to vendor [03:37:37] (03CR) 10Eileen: "Hmm - now I look at it - we send up four pieces of info" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826410 (https://phabricator.wikimedia.org/T315102) (owner: 10Ejegg) [06:46:36] (03PS10) 10Wfan: Pass merchantAccountId with currency to support multi currencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/822192 (https://phabricator.wikimedia.org/T312808) [07:11:09] (03PS1) 10Eileen: Update docs, dev setting for performance tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826493 [07:17:12] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: ipset_error 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [07:22:12] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [14:03:56] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10krobinson) Hey @Ejegg - Just a friendly ping on @AMJohnson's question above as donors are waiting on responses. Thanks! [15:39:55] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10Patch-For-Review: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10KHaggard) Thanks both. I'm happy to search for more cids after T293945 is resolved if the is... [15:45:16] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10MBeat33) May I also add a Braintree console refund request for Transaction ID BRAINTREE DHJHBNNHY3RPB25FZJEWMXI3OGC / Inv... [15:49:14] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10AnnWF) Should we do full refund for those transactions or partially refund? Please confirm, I can issue the refund from Bra... [16:03:00] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10MBeat33) For the one I just added, a full refund would be great as it was a test donation. [16:19:45] 10Fundraising-Backlog: Braintree: review exactly which currently used Paypal IPNs are unsupported and investigate ways to use supported ones instead - https://phabricator.wikimedia.org/T315012 (10Dwisehaupt) [16:28:20] 10Fundraising-Backlog: Use nightly rather than weekly audit file for Adyen - https://phabricator.wikimedia.org/T314753 (10Ejegg) [16:33:08] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Way to send Civi TY email to a group? - https://phabricator.wikimedia.org/T314525 (10AnnWF) search kit: https://civicrm.wikimedia.org/civicrm/admin/search#/edit/597 [16:33:20] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Way to send Civi TY email to a group? - https://phabricator.wikimedia.org/T314525 (10Dwisehaupt) [16:35:09] 10Fundraising-Backlog, 10Epic: Upgrading payments-wiki from buster to bullseye - https://phabricator.wikimedia.org/T315585 (10Dwisehaupt) [16:35:41] 10Fundraising-Backlog, 10Epic: Upgrading payments-wiki from buster to bullseye - https://phabricator.wikimedia.org/T315585 (10Dwisehaupt) Setting sprint points to 0 as this is an epic and the point based work will show up in subtasks that will be created. [17:17:31] (03PS1) 10Jgleeson: Ignore Paypal mp_notification IPNs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/826629 [17:38:20] Hi Jgleeson:, do you have the link that indicates the mp_notification is related with Braintree? I also see mp_id from paypal_smashpig_job and thought that might also related with braintree, but not 100% sure. Thanks~ [17:40:12] I was trying to find one before wfan but google was showing me the local MP for the town of Braintree in Essex, UK [17:41:24] haha, oh right the link you shared [17:46:25] maybe we should ask him [17:50:10] Do you mean email Braintree? This PayPal ipn listener with Braintree stuff is very annoying 😅 [18:16:35] no, the local Braintree MP [18:16:44] sorry I'm joking ^_^ [19:07:25] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10Ejegg) Looks like someone has just refunded that one @MBeat33. I tried to refund it and got the error 'Refund was declined... [19:09:09] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10Ejegg) Ahh, perhaps it was refunded via the PayPal console? The refund does not show up anywhere in the Braintree console. [19:09:47] oof fr-tech, the braintree console doesn't reflect refunds that were performed at the PayPal console [19:12:51] oof indeed [19:14:46] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10MBeat33) Thank you Elliott and Ann. > I guess the lesson is that in the Braintree console it's tricky to see what's been r... [19:17:48] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10Ejegg) Hi @AMJohnson and @krobinson CID 56622799 - I just issued a refund now in the Braintree console CID 35973901 - Ref... [19:19:35] (03PS2) 10Wfan: Add extra braintree test link for recurring and CAD GBP, round HUF [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/822195 (https://phabricator.wikimedia.org/T312808) [19:23:07] ughhh are those long TXN ids useful for ANYthing in the console? [19:23:14] not seeing them anywhere [19:23:23] ok, so... where do we stash any extra IDs ? [19:23:59] wmf_contribution_extra? [19:25:02] (03CR) 10Ejegg: [C: 03+2] Do not exclude CiviCRM files from git [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826414 (owner: 10Eileen) [19:25:42] (03CR) 10Ejegg: [C: 03+2] "Thanks!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826493 (owner: 10Eileen) [19:30:47] (03CR) 10Ejegg: Add extra braintree test link for recurring and CAD GBP, round HUF (031 comment) [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/822195 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [19:32:36] we were using floor so round down not up [19:37:17] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10fr-donorservices: Donation Form Currency Issue PayPal USD/CNY - https://phabricator.wikimedia.org/T315592 (10Ejegg) OK, I think I have deleted all of the incorrectly-recorded CNY donations from Civi. I double checked that all of the... [19:39:13] 10Fundraising Sprint Machine Unlearning, 10Fundraising Sprint NaN is a Number, 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, and 3 others: Don't send failmail on Adyen unsupported card - https://phabricator.wikimedia.org/T313069 (10Ejegg) Oops, I moved this one backwards when I... [19:40:13] 10Fundraising Sprint Overused petting Zoo Memetics, 10Fundraising-Backlog, 10Patch-For-Review: Custom field "AF_has_active_recurring_donation" is showing false positives - https://phabricator.wikimedia.org/T315102 (10Ejegg) a:03Ejegg [19:40:16] (03Merged) 10jenkins-bot: Do not exclude CiviCRM files from git [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826414 (owner: 10Eileen) [19:40:18] (03Merged) 10jenkins-bot: Update docs, dev setting for performance tracking [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/826493 (owner: 10Eileen) [19:41:53] oh ok wfan, i didn't realize [19:50:36] (03PS15) 10Wfan: Check if currency supported by braintree merchant account [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/822191 (https://phabricator.wikimedia.org/T312808) [20:04:46] (03PS11) 10Wfan: Pass merchantAccountId with currency to support multi currencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/822192 (https://phabricator.wikimedia.org/T312808) [20:05:11] (03CR) 10CI reject: [V: 04-1] Pass merchantAccountId with currency to support multi currencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/822192 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [20:26:11] (03PS12) 10Wfan: Pass merchantAccountId with currency to support multi currencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/822192 (https://phabricator.wikimedia.org/T312808) [20:28:34] (03PS1) 10Wfan: Add more invalid error translation currency, order_id and payment_token [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/826653 (https://phabricator.wikimedia.org/T312808) [21:13:20] (03PS11) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [21:14:29] hi ejegg|afk, looks like we can't move the session_resetForNewAttempt out of the conditional because of the saltedToken required by Adyen for MC [21:15:09] (03CR) 10CI reject: [V: 04-1] Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) (owner: 10Damilare Adedoyin) [21:22:04] (03PS12) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [21:28:04] damilare: oh darn. can we back that key up too? [21:30:00] ok I'd check [22:22:56] (03PS13) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [22:29:10] (03PS14) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [22:36:36] (03CR) 10Wfan: [C: 03+2] Fix 'return' syntax [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826411 (owner: 10Ejegg) [22:37:54] (03Merged) 10jenkins-bot: Fix 'return' syntax [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/826411 (owner: 10Ejegg) [22:38:03] (03PS15) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [22:50:22] (03PS16) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [22:51:44] (03PS17) 10Damilare Adedoyin: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) [22:53:10] (03CR) 10Damilare Adedoyin: "Thanks for the review Ejegg, all commented lines have been revised and the patch is ready for review" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) (owner: 10Damilare Adedoyin) [23:41:15] (03PS18) 10Ejegg: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) (owner: 10Damilare Adedoyin) [23:41:39] (03CR) 10Ejegg: [C: 03+2] "Looks good, works across different gateways. Thanks!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) (owner: 10Damilare Adedoyin) [23:42:44] (03PS3) 10Ejegg: Add extra braintree test link for recurring and CAD GBP, round HUF [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/822195 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [23:42:50] (03CR) 10Ejegg: [V: 03+2 C: 03+2] Add extra braintree test link for recurring and CAD GBP, round HUF [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/822195 (https://phabricator.wikimedia.org/T312808) (owner: 10Wfan) [23:44:03] (03Merged) 10jenkins-bot: Backup Donor session to allow additional donations [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/824806 (https://phabricator.wikimedia.org/T314681) (owner: 10Damilare Adedoyin)