[02:20:15] Fundraising-Backlog: JP fraud - card testing??? - https://phabricator.wikimedia.org/T258550 (EMartin) [02:23:39] Fundraising-Backlog: JP fraud - card testing??? - https://phabricator.wikimedia.org/T258550 (Ejegg) Thanks for picking up on this @EMartin! Let's keep fraud tickets private. [05:15:35] (PS2) Eileen: Add new shell wmf-civicrm extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/614920 [05:15:37] (PS1) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [05:15:39] (PS1) Eileen: Remove no-longer called function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615326 (https://phabricator.wikimedia.org/T256316) [05:15:41] (PS1) Eileen: Comment fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615327 (https://phabricator.wikimedia.org/T256316) [05:15:43] (PS1) Eileen: Remove unused function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615328 (https://phabricator.wikimedia.org/T256316) [05:15:45] (PS1) Eileen: Remove unused function Preliminary cleanup Bug: T256316 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615329 (https://phabricator.wikimedia.org/T256316) [05:18:16] (CR) jerkins-bot: [V: -1] Remove no-longer called function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615326 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:18:46] (CR) jerkins-bot: [V: -1] Remove unused function Preliminary cleanup Bug: T256316 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615329 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:22:10] (PS2) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [05:23:01] (CR) jerkins-bot: [V: -1] Remove unused function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615328 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:23:12] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:23:36] (CR) jerkins-bot: [V: -1] Comment fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615327 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:25:19] (PS3) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [05:26:40] (PS4) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [05:27:47] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:28:33] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [05:47:38] (PS5) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [05:55:08] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [06:30:04] Fundraising-Backlog: JP fraud - card testing??? - https://phabricator.wikimedia.org/T258551 (Aklapper) @EMartin: Assuming this task is about WMF Fundraising., hence adding that project tag so other people can also find this task when searching via projects. Please set appropriate project tags when possible.... [06:43:22] (PS6) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [06:49:37] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [07:04:41] (PS7) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [07:12:18] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [07:18:47] (PS8) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [07:27:16] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [09:48:11] (PS9) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [09:54:56] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [12:56:12] Wikimedia-Fundraising-Banners: [Mobile Lg + Sm][Android] No spacing between currency symbol and amount in main copy and PTF text - https://phabricator.wikimedia.org/T258411 (Pcoombe) Open→Resolved a:Pcoombe Great! Applied these fixes to the pre-test banners: - [m lg cnt](https://meta.wikimedia.o... [12:56:14] Wikimedia-Fundraising-Banners: QA for enIN (English India) pre-test on July 21, 2020 - https://phabricator.wikimedia.org/T258279 (Pcoombe) [14:19:35] Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: 3 Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (mepps) I'm seeing another one from 10936530. [14:21:29] Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: 3 Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (mepps) This is from adyen but there's a similar one that came in with the message Recurring donation, but no subscription ID or recurring payment token found fr... [14:23:14] Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: 3 Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (mepps) I see four of these adyen ones. At least one has no contact associated. I wonder if this is worth further investigation? [14:49:27] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Investigation: Phase out cross-domain hide banner cookie stom? If so, how and when? - https://phabricator.wikimedia.org/T258368 (Pcoombe) FYI I've now amended the Thank You page to only load HideBanners from wikipedia.org [14:49:57] Wikimedia-Fundraising-Banners: 2020-07-21 enIN pre-test: mobile small batch - https://phabricator.wikimedia.org/T258283 (jbolorinos-ctr) PAN variant: {F31944532} [14:50:35] Wikimedia-Fundraising-Banners: 2020-07-21 enIN pre-test: mobile small batch - https://phabricator.wikimedia.org/T258283 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr Testing has been completed on these banners, and the PAN text functionality has been fully verified. These banners are now READY... [14:50:38] Wikimedia-Fundraising-Banners: QA for enIN (English India) pre-test on July 21, 2020 - https://phabricator.wikimedia.org/T258279 (jbolorinos-ctr) [14:51:07] Wikimedia-Fundraising-Banners: QA for enIN (English India) pre-test on July 21, 2020 - https://phabricator.wikimedia.org/T258279 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr Aside from the remaining open bugs, these banners are now READY TO TEST (pass with issues) [14:51:59] Wikimedia-Fundraising-Banners: [CANDIDATE] QA for 'RML options' banner - https://phabricator.wikimedia.org/T253054 (jbolorinos-ctr) Open→Resolved Aside from the remaining open issues, this variant is READY TO TEST (pass with issues) [14:59:20] tzag fr-tech [15:09:50] hi cstone! [15:10:12] so... looks like maybe I was just dumb about where I put that tokenize param? [15:10:34] gonna try that as soon as I update the Scrum-o-scrums thing [15:10:43] any cross-team news, fr-tech? [15:14:28] fundraising-tech-ops: reconfigure 'statler' to use fundraising analytics database (frdb1003) instead of frdev1001 - https://phabricator.wikimedia.org/T258600 (Jgreen) [15:16:19] ejegg: you were able to get it to tokenize then? [15:18:11] cstone haven't tried yet, just reading this email [15:25:49] Fundraising-Backlog, fundraising-tech-ops, Epic: move frdev1001:/srv/br/analytics out to it's own directory - https://phabricator.wikimedia.org/T258601 (Jgreen) [15:26:09] Fundraising-Backlog, fundraising-tech-ops: move frdev1001:/srv/br/analytics out to it's own directory - https://phabricator.wikimedia.org/T258601 (Jgreen) [15:27:01] fundraising-tech-ops: reconfigure 'statler' to use fundraising analytics database (frdb1003) instead of frdev1001 - https://phabricator.wikimedia.org/T258600 (Jgreen) [15:27:48] huh, authorization errors [15:27:55] like our key is bad? [15:28:04] checking if I'm using an expired one [15:28:17] code 400540 [15:32:48] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] [15:34:09] Fundraising Sprint Octopus hugs, Fundraising-Backlog: Fr-tech chores list - https://phabricator.wikimedia.org/T258527 (mepps) a:mepps I noticed a failmail and looked it up in the failmail zoo and found this: > Get recipient data from Silverpop failed with code 1 > This happens fairly often with er... [15:42:17] Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: 3 Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (mepps) Similarly, we got one with "Recurring donation, but no subscription ID or recurring payment token found" [15:44:08] Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: 3 Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (Cstone) @mepps That recurring donation with no subscription might be this one https://phabricator.wikimedia.org/T255317 [15:47:56] hey cstone ejegg|brb [15:48:05] hi jgleeson [15:49:16] how's it goin cstone ? [15:49:19] Fundraising Sprint Octopus hugs, Fundraising-Backlog: AMEX cards not working for india donors - https://phabricator.wikimedia.org/T258509 (EMartin) Hi all, I am in favor of just disabling AMEX for the India campaign. It is insignificant (less than 1%) for this market and low approval rates seem to be a... [15:51:16] its raining here which cancelled my morning paddleboarding but should clear up later, how about you jgleeson [15:55:08] it's been raining here too :/ but I think we're due some good weather on Friday hopefully. I used the elliptical trainer for the first time in a week earlier today and I'm feeling now moving around the house like a giant tortoise [15:56:11] I thought I'd try and be organised today and cooked dinner earlier in the day so I don't have to do the rush between standup and tech-talk [15:56:56] Fundraising Sprint Octopus hugs, Fundraising-Backlog: Solving for Indian donors that don't have a PAN but want to donate - https://phabricator.wikimedia.org/T258086 (jgleeson) a:jgleeson [15:57:13] actually cstone do you have your eye on that ticket? I can pick up something else if you [15:57:16] ^ [15:57:29] jgleeson: i havent looked at it yet go for it [15:57:30] I know you've been PANning much of late [15:57:34] ah cool [15:59:36] that reminds me i should just buy a gold pan online since rock shows wont be happening for a while :P [16:01:56] Wikimedia-Fundraising-Banners: [Enhancement] Include a reference to local laws for opt-in copy tests - https://phabricator.wikimedia.org/T258607 (jbolorinos-ctr) [16:02:48] RECOVERY - check_log_messages on frav1002 is OK: OK [16:12:13] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 20268 20000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 58 days 16 minutes - memory use is 24.57M (peak 26.43M, 0.36% of max, fragmentation 1.18%), connected_slaves is 2, jobs is 0, jobs-adyen is 7, jobs-paypal is 514, payments-antifraud is 1213, payments-init is 490, pending is 2, recurring is 1215, refund is 0, unsubscribe is 28 [16:17:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 22397 20000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 58 days 21 minutes - memory use is 27.59M (peak 28.68M, 0.38% of max, fragmentation 1.14%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 605, payments-antifraud is 1311, payments-init is 830, pending is 0, recurring is 1348, refund is 0, unsubscribe is 37 [16:22:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 23174 20000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 58 days 26 minutes - memory use is 27.70M (peak 28.68M, 0.39% of max, fragmentation 1.16%), connected_slaves is 2, jobs is 0, jobs-adyen is 10, jobs-paypal is 532, payments-antifraud is 1277, payments-init is 502, pending is 7, recurring is 1189, refund is 0, unsubscribe is 4 [16:27:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 25512 20000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 58 days 31 minutes - memory use is 30.24M (peak 31.28M, 0.42% of max, fragmentation 1.13%), connected_slaves is 2, jobs is 0, jobs-adyen is 5, jobs-paypal is 656, payments-antifraud is 1295, payments-init is 920, pending is 58, recurring is 1401, refund is 0, unsubscribe is 5 [16:27:42] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 1 [=1] [16:32:12] PROBLEM - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 26223 20000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 58 days 36 minutes - memory use is 31.35M (peak 32.85M, 0.44% of max, fragmentation 1.15%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 908, payments-antifraud is 1598, payments-init is 600, pending is 270, recurring is 1515, refund is 0, unsubscribe is [16:32:48] RECOVERY - check_log_messages on frav1002 is OK: OK [16:34:44] Fundraising-Backlog, FR-Q2-FY2020-21-cleanup-list: 3 Paypal recurring payments missing predecessor - https://phabricator.wikimedia.org/T248420 (mepps) @Cstone it's actually paypal. It sounds like we might want to do an investigation and link these tickets. [16:34:57] ACKNOWLEDGEMENT - check_redis on frqueue1001 is CRITICAL: CRITICAL: donations is 26223 20000 - REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 58 days 36 minutes - memory use is 31.35M (peak 32.85M, 0.44% of max, fragmentation 1.15%), connected_slaves is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 908, payments-antifraud is 1598, payments-init is 600, pending is 270, recurring is 1515, refund is 0, unsubsc [16:34:57] upt heavy donation flow. acking for a few hours - The acknowledgement expires at: 2020-07-23 18:33:51. [16:43:39] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Fix stuck recurring contributions, add alarm - https://phabricator.wikimedia.org/T258013 (MBeat33) Thank you for offering to batch cancel these, @Ejegg ! [17:15:59] (PS1) Ejegg: Disable Amex for India [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/615525 (https://phabricator.wikimedia.org/T258509) [17:16:09] fr-tech anyone wanna give ^^^ a glance? [17:21:18] (CR) Cstone: [C: +2] Disable Amex for India [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/615525 (https://phabricator.wikimedia.org/T258509) (owner: Ejegg) [17:23:10] fundraising-tech-ops: Bugfix for get_lock() function on fran1001 inserts - https://phabricator.wikimedia.org/T258454 (Dwisehaupt) Found the issue. Writing the PID to the file after it has locked seems to release the lock. Looking to see if I can use a write lock to allow the write. If not, we can just not in... [17:26:57] (Merged) jenkins-bot: Disable Amex for India [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/615525 (https://phabricator.wikimedia.org/T258509) (owner: Ejegg) [17:27:09] thanks cstone! [17:28:05] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/615530 [17:28:12] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/615530 (owner: Ejegg) [17:30:28] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/615530 (owner: Ejegg) [17:51:11] sorry fr-tech, think i have to reboot [17:51:47] ejegg: no worries, we've devolved into talking about non-work and may jump off ahead of the other meeting, I'm guessing [17:53:56] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Security-Team, Performance-Team (Radar): Uncaught SecurityError: Failed to construct 'Worker': Access to the script at 'blob:https://ca.wikipedia.org/92522a15-8318-403f-bb45-8e554fc893c0' is denied by... - https://phabricator.wikimedia.org/T258459 [17:53:58] Fundraising Sprint Octopus hugs, Fundraising-Backlog, Patch-For-Review: AMEX cards not working for india donors - https://phabricator.wikimedia.org/T258509 (Pcoombe) I've hidden the Amex logo on tomorrow's India banners: [example diff](https://meta.wikimedia.org/w/index.php?title=MediaWiki:Centralnot... [17:56:45] oh hey, there's a 1.35 branch in the mediawiki repo! [17:59:49] Wikimedia-Fundraising-Campaigns: Fundraise in India - https://phabricator.wikimedia.org/T131118 (Pcoombe) [18:01:39] fr-tech: hangouts is being weird for me, be in the meeting in a moment. [18:01:58] Just waiting for it to figure it's life out. [18:08:08] Fundraising-Backlog: Donation processing rate seems a little random. Might be worth investigating why. - https://phabricator.wikimedia.org/T258629 (jgleeson) [18:25:15] Wikimedia-Fundraising-Banners: [Candidate] NL Banners (Duplicate of Asana task) - https://phabricator.wikimedia.org/T255833 (jbolorinos-ctr) Closing this task now as all open issues have been addressed. [18:25:25] Wikimedia-Fundraising-Banners: [Candidate] NL Banners (Duplicate of Asana task) - https://phabricator.wikimedia.org/T255833 (jbolorinos-ctr) Open→Resolved a:jbolorinos-ctr [18:26:13] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 23 - https://phabricator.wikimedia.org/T258177 (jbolorinos-ctr) QA has now been completed on these banners and they are READY TO TEST! [18:26:32] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 23 - https://phabricator.wikimedia.org/T258177 (jbolorinos-ctr) Open→Resolved [18:57:50] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/615542 [18:57:53] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/615542 (owner: Ejegg) [18:59:09] ejegg: fancy pushing up your WIP from tech-talk if you get a minute? I figure it would be a good base for https://phabricator.wikimedia.org/T258086 [18:59:21] jgleeson: sure, one sec [18:59:42] thanks! [19:01:31] (PS1) Ejegg: WIP make PAN optional for India [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/615546 (https://phabricator.wikimedia.org/T258086) [19:01:34] there you go jgleeson ^^^ [19:03:45] (CR) jerkins-bot: [V: -1] WIP make PAN optional for India [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/615546 (https://phabricator.wikimedia.org/T258086) (owner: Ejegg) [19:04:44] (CR) jerkins-bot: [V: -1] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/615542 (owner: Ejegg) [19:06:32] (CR) Ejegg: [V: +2 C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/615542 (owner: Ejegg) [19:13:02] Fundraising Sprint Octopus hugs, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: add translations to generic email for failed recurring donations - https://phabricator.wikimedia.org/T255837 (DStrine) [19:13:39] Fundraising Sprint Octopus hugs, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: add translations to generic email for failed recurring donations - https://phabricator.wikimedia.org/T255837 (mepps) Highest priority: japanese, swedish, and french. [19:14:34] !log updated payments-wiki from bf91f8adff to 31a3de1130 [19:14:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:22:18] Fundraising Sprint Octopus hugs, Fundraising-Backlog, MW-1.36-notes (1.36.0-wmf.2; 2020-07-28): AMEX cards not working for india donors - https://phabricator.wikimedia.org/T258509 (Ejegg) @EMartin The Amex button is gone from the India / D-Local form. [19:40:14] Fundraising Sprint Octopus hugs, Fundraising-Backlog, MW-1.36-notes (1.36.0-wmf.2; 2020-07-28): AMEX cards not working for india donors - https://phabricator.wikimedia.org/T258509 (EMartin) Thank you! [19:42:14] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Fix stuck recurring contributions, add alarm - https://phabricator.wikimedia.org/T258013 (Ejegg) OK, they should all be marked as 'Cancelled' now, with cancel_date of 2020-07-22 and cancel_reason as 'Stuck for months in Process... [19:42:23] RECOVERY - check_recurring_contrib_processing on frdb1002 is OK: OK recurring_contrib_processing=3 [19:52:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Fix stuck recurring contributions, add alarm - https://phabricator.wikimedia.org/T258013 (MBeat33) Open→Resolved a:MBeat33 Thank you for the followup. Adri is following up with these donors, so I'll change this to R... [19:57:37] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 30 - https://phabricator.wikimedia.org/T258646 (spatton) [20:01:25] Wikimedia-Fundraising-Banners: 2020-07-30 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T258647 (spatton) [20:12:24] hmm, database lockups? [20:12:31] oh! [20:15:36] ejegg: I just checked the running jobs and see these: [20:15:38] delete_deleted_contacts [20:15:40] donations_queue_consume [20:15:42] ingenico_orphan_slayer [20:15:44] paypal_orphan_slayer [20:15:46] thank_you_mail_sen [20:15:48] not sure if deleted_contacts would cause any locks [20:16:07] best is to use drush civicrm-sql-cli [20:16:21] then show full processlist [20:16:31] ahhh the one with the scary drop table cmd underneith it [20:16:46] underneath* [20:16:52] ?? [20:16:58] oh, a different drush command [20:17:05] hah, yeah, don't run that other one [20:17:06] yeah sorry [20:17:10] :> [20:17:22] ok, the processlist doesn't actually show any stuck queries [20:17:34] so I guess it's not going to hang any more [20:18:09] yeah [20:18:36] the delete job has complete also [20:18:47] I've not seen that job [20:18:53] possibly entirely unrelated [20:19:40] but I think deletes lock the table on innodb? [20:20:15] * jgleeson releases how much he doesn't know about mysql [20:21:03] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: permission update needed for DS agent to Send Failure Notifications? - https://phabricator.wikimedia.org/T258651 (MBeat33) [20:22:12] nothing in the process-control logs [20:23:07] jgleeson it deletes from the contact and contact log tables [20:23:42] it's new [20:23:58] it picks batches of contacts flagged as deleted more than I think 2 years ago? [20:26:18] donations_queue_consume/donations_queue_consume-20200722-202101.log:2020-07-22 20:21:28,346 ERROR Contribution not saved due to database load [20:26:20] donations_queue_consume/donations_queue_consume-20200722-202101.log:2020-07-22 20:21:29,402 ERROR database load [20:26:34] not seen the database load error before [20:26:39] it's kind of new [20:26:47] jgleeson@civi1001:/var/log/process-control$ ls -tr */*.log | tail -n30 | xargs grep -i --color=always database [20:26:52] I'm running that just browsing through [20:27:01] trying to catch specifically the db lock errors [20:27:06] so that we can re-queue them [20:27:12] RECOVERY - check_redis on frqueue1001 is OK: OK: REDIS 5.0.3 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 58 days 4 hours - memory use is 6.53M (peak 59.11M, 0.15% of max, fragmentation 1.89%), connected_slaves is 2, donations is 3306, jobs is 0, jobs-adyen is 6, jobs-paypal is 0, payments-antifraud is 200, payments-init is 0, pending is 0, recurring is 1562, refund is 0, unsubscribe is 90 [20:27:14] though it still doesn't see to get all of them [20:30:26] Fundraising Sprint Octopus hugs, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: add translations to generic email for failed recurring donations - https://phabricator.wikimedia.org/T255837 (MBeat33) Thank you for scheduling this. The current translations will be very useful to be able to send in... [20:33:10] ejegg: maybe a slow query from civicrm? do we log slow queries? [20:33:20] I poked around frlog and civi1001 but couldn't see anything [20:34:08] Jeff_Green: dwisehaupt so you know if we have the mysql slow query log enabled and if so where we'd find it? [20:35:10] jgleeson: yes, it's logging on frdb1002, what timeframe do you need? [20:35:22] just today? [20:35:35] the last 2 hours even? [20:35:41] or today is fine [20:36:06] actually just the last hour would be fine also [20:36:28] we had some locking up around 54 minutes ago [20:36:43] kinda on the tip of the hour timebox [20:37:52] ok, sec, I'll put it on frpm1001 [20:37:58] thanks! [20:41:00] jgleeson: ok actually it's here frlog1001:/tmp/frdb1002-slow.log [20:41:06] thanks! [20:41:10] np! [20:46:46] hmmm [20:46:49] # Query_time: 24.201266 Lock_time: 0.000727 Rows_sent: 1 Rows_examined: 7099050 [20:46:51] # Query_time: 24.971042 Lock_time: 0.003536 Rows_sent: 14 Rows_examined: 7099066 [20:46:53] # Query_time: 139.213709 Lock_time: 0.000076 Rows_sent: 60000 Rows_examined: 40148813 [20:46:55] # Query_time: 31.166151 Lock_time: 0.248090 Rows_sent: 0 Rows_examined: 947014 [20:46:57] # Query_time: 25.007820 Lock_time: 0.000398 Rows_sent: 2 Rows_examined: 7099068 [20:46:59] # Query_time: 22.962602 Lock_time: 0.004267 Rows_sent: 1 Rows_examined: 7099071 [20:47:01] nothing there that I think would explain the locks [20:47:35] what was it that locked up? [20:47:57] the failmail - FAILMAIL - ERROR: civi1001 (no_provider) SmashPig::recurring [20:48:17] contains a bunch of Deadlock found when trying to get lock; try restarting transaction [20:50:16] is there any chance it wasn't talking to frdb1002 there? [20:51:24] I couldn't say [20:51:30] ejegg: I'm wondering if I can get you to review the first 2 in this chain - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/613010 - it's just moving stuff to the deduper & I suspect that it's not 'meaty enough' for anyone who hasn't seen that before to get their heads into in review but you at least know how that is structured [20:52:08] ok eileen, give me a few min and I'll dive right in [20:52:22] it shouldn't be too big I don't think [20:52:32] but not urgent [20:54:49] jgleeson: syslog may actually have better info here [20:55:52] jgleeson: grep mysqld on frlog1001:/var/log/remote/fundraisingdb-misc [20:58:17] ohh ok I'll check it out [20:58:24] Fundraising Sprint Octopus hugs, Fundraising-Backlog: Fr-tech chores list - https://phabricator.wikimedia.org/T258527 (mepps) I looked briefly at error logs and saw a weird paypal client side error, then got distracted by the start of standup. I'll try to follow up on it. [21:01:11] ok lots of output [21:01:36] grep out frdb1002 also I guess, I did see some deadlocky stuff in there [21:03:36] ya just sifting through it now trying to make sense of it [21:04:04] but I think this is the *log* [21:04:36] yep, it's the regular mariadb error log as opposed to the slow log [21:05:14] or I guess they call it the 'general' log [21:08:39] I just wish I could see something that would explain the failure [21:08:51] I mean it looks like it's here but my eyes can't see it [21:08:53] lol [21:08:57] ha [21:09:42] do you have the time for the error? [21:11:47] Fundraising Sprint Octopus hugs, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Readonly DB connection available to turn on - https://phabricator.wikimedia.org/T237350 (Eileenmcnaughton) I just re-enabled the 2 rpow extensions on staging - they are disabled in this screenshot {F31944941} The d... [21:11:47] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: desktop large batch - https://phabricator.wikimedia.org/T258647 (spatton) [21:12:26] it looks like it started around 1h ago and it looks like the email titled 'Fail Mail : DATABASE_CONTENTION: civi1001 (wmf_common)' from 1h ago is the beginning og it [21:12:28] of* [21:12:47] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: desktop small batch - https://phabricator.wikimedia.org/T258654 (spatton) [21:13:00] Jeff_Green: I've gotta jump off now but thanks for all the help funding the logs. I'll follow-up on it tomorrow also! bye for now [21:13:07] finding*** [21:13:11] ok, have a good night! [21:15:12] Wikimedia-Fundraising-Banners: QA for en4C pre-test on July 30 - https://phabricator.wikimedia.org/T258646 (spatton) [21:15:20] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: mobile large batch - https://phabricator.wikimedia.org/T258655 (spatton) [21:15:28] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: mobile small batch - https://phabricator.wikimedia.org/T258656 (spatton) [21:15:30] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Hover text is offset significantly from form - https://phabricator.wikimedia.org/T258287 (jbolorinos-ctr) @DStrine do you think it might be possible to simply disable that text on the payment form? Is it the title text? [21:19:28] ac [21:19:57] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: desktop large batch - https://phabricator.wikimedia.org/T258647 (jbolorinos-ctr) Screenshot Test Results - Desktop: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z455c427cb97af2569d7 [21:21:16] Wikimedia-Fundraising-Banners: QA for en4C pre-test on July 30 - https://phabricator.wikimedia.org/T258646 (spatton) [21:21:18] Fundraising-Backlog, Wikimedia-Fundraising-Banners: Hover text is offset significantly from form - https://phabricator.wikimedia.org/T258287 (DStrine) the hover text is very helpful for some people. We would get DS tickets if we disabled it. Can you prove that it is happening for any other computer? no o... [21:30:40] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: desktop small batch - https://phabricator.wikimedia.org/T258654 (jbolorinos-ctr) Screenshot Test Results - Desktop: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z9b351474f643deda6df [21:38:15] (PS10) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [21:43:58] Wikimedia-Fundraising-Banners: 2020-07-30 en4C pre-test: mobile large batch - https://phabricator.wikimedia.org/T258655 (jbolorinos-ctr) Screenshot Test Result - Mobile: - Control: https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/zdae08e422dcea67d3cb [21:45:32] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [21:52:06] (PS11) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [21:59:22] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [22:01:09] (PS12) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [22:08:28] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [22:13:59] (PS13) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [22:22:05] (CR) jerkins-bot: [V: -1] Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) (owner: Eileen) [22:23:34] Fundraising Sprint Octopus hugs, Fundraising-Backlog, MW-1.36-notes (1.36.0-wmf.2; 2020-07-28): AMEX cards not working for india donors - https://phabricator.wikimedia.org/T258509 (EMartin) Hi - I heard back from Dlocal today. They had this to say "Regarding Amex, I double checked the logs and I'm n... [22:25:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spate of database contention - https://phabricator.wikimedia.org/T258661 (Eileenmcnaughton) [22:25:42] I created this re failmails - https://phabricator.wikimedia.org/T258661 [22:27:04] (PS14) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [22:27:58] (PS15) Eileen: Move MergeTest to new wmf-civicrm [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615325 (https://phabricator.wikimedia.org/T256316) [22:40:56] (PS2) Eileen: Remove no-longer called function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615326 (https://phabricator.wikimedia.org/T256316) [23:03:37] I'm finding a few no-longer called functions like this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/615326/2/sites/all/modules/wmf_civicrm/wmf_civicrm.module [23:04:46] (PS2) Eileen: Remove unused function [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615328 (https://phabricator.wikimedia.org/T256316) [23:05:55] (PS2) Eileen: Comment fixes [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615327 (https://phabricator.wikimedia.org/T256316) [23:06:44] (PS2) Eileen: Remove unused function Preliminary cleanup Bug: T256316 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/615329 (https://phabricator.wikimedia.org/T256316) [23:12:51] I have some fairly quick review items in the queue if anyone has a short amount of time to fill