[00:00:59] fundraising-tech-ops: 2020-2021 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T257244 (Jclark-ctr) [00:01:50] hrm, do we have some extensions listed twice? [00:02:41] oh no! [00:02:57] it's going ahead and enabling the extensions that it already enabled [00:03:31] guessing the results of that first check were cached [00:04:20] eileen I'm just going to delete the extraneous instances of wmf_civicrm_reconcile_extensions for now [00:05:11] ejegg: good plan :-) I think maybe jgleeson|away's approach of exposing that as a drush command or api rather than adding to the upgrade script might have merit [00:11:41] (PS1) Ejegg: Stop double-enabling civi extensions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645208 [00:14:38] ok, i gotta go make some dinner [00:16:47] (CR) Eileen: [C: +2] "If jenkins is happy I am (this would fail hard if they are not still enabled)" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645208 (owner: Ejegg) [00:17:05] I'm not sure if ^^ will help but it won't hurt [00:17:54] I just started poking around in api3\Extension & Manager.php trying to find where it would do those pings [00:18:01] and nothing jumped out at me [00:21:11] ejegg|food: is it definitely during the extension enabling it's slow? [00:21:31] ejegg|food: also - I have another relevant patch now I think about it.... [00:22:31] https://lab.civicrm.org/dev/core/-/issues/2031 [00:22:39] - pretty sure I created a phab related [00:35:51] (Merged) jenkins-bot: Stop double-enabling civi extensions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645208 (owner: Ejegg) [01:31:33] fundraising-tech-ops: Difficulty accessing mysql through server - https://phabricator.wikimedia.org/T269407 (Reedy) [01:47:53] (PS1) Eileen: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645218 [01:48:13] (CR) Eileen: [C: +2] Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645218 (owner: Eileen) [01:48:32] fundraising-tech-ops: Difficulty accessing mysql through server - https://phabricator.wikimedia.org/T269407 (Dwisehaupt) @rmurthy Your username for MySQL should be rmurthy. So you could use this for the last mysql command: `mysql -u rmurthy -p -h frdb1003` However, since you have the .my.cnf file in place,... [01:56:35] (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645218 (owner: Eileen) [01:59:29] eileen: if you look at https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/3931/console you see the pauses in between module enablements [01:59:41] that was with logging before each module install call [02:01:33] yeah so 5 mins a pop for extensions huh? [02:03:28] looks more like 15 sec-ish each, but like 3.5 min total [02:03:38] and then that was tripled because of the extra reconcile calls [02:05:10] yeah so I suspect there is some cache clearing that does those 15 secs - although it could be file scanning - see https://lab.civicrm.org/dev/core/-/issues/2031 [02:06:47] (Merged) jenkins-bot: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645218 (owner: Eileen) [02:13:10] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/645219 [02:14:40] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/645219 (owner: Eileen) [02:16:08] !log civicrm revision changed from 913ccdfd2b to 5fa107d32a, config revision is ffe0a99133 [02:16:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:16:16] ejegg: that ^^ is the emoji patch [02:28:12] nice [07:38:34] (PS1) AndyRussG: Send logger host name to Payments container via envar [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/645296 [14:16:33] hi fr-tech [14:19:21] hey ejegg [14:21:21] hi jgleeson! [14:46:43] looks like we're pretty closely tracking last year's day 4 [15:10:37] Fundraising-Backlog, FR-Ingenico, fr-donorservices: Ingenico connection: PR_END_OF_FILE_ERROR - https://phabricator.wikimedia.org/T269367 (MBeat33) Open→Resolved a:MBeat33 Resolving this as any remaining concerns will be covered in {T269378}. Thank you for the testing and the tip about th... [15:26:43] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (EMartin) Status as of 12.4.2020 7am PT: From Ingenico: “Connection issues for some merchants to Miami. We are e... [15:41:03] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Ingenico, fr-donorservices: iframe error / error ref #s before entering card #s - https://phabricator.wikimedia.org/T269378 (DStrine) Open→Resolved [15:42:59] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog, MW-1.36-notes (1.36.0-wmf.20; 2020-12-01): Make payments phpunit tests pass under Docker - https://phabricator.wikimedia.org/T266088 (DStrine) [15:43:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Fix our buildkit install to handle fredge DB - https://phabricator.wikimedia.org/T267970 (DStrine) [15:43:39] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Use Guzzle to make API request to CiviProxy to retrieve opt-in / opt-out fields. - https://phabricator.wikimedia.org/T268497 (DStrine) [15:43:48] Fundraising-Backlog: Create civiproxy on docker - https://phabricator.wikimedia.org/T268683 (DStrine) [15:43:58] Fundraising-Backlog: Docker dev setup: Set up email export - https://phabricator.wikimedia.org/T268688 (DStrine) [15:44:04] Fundraising-Backlog: Docker dev setup: Set up queue consumers - https://phabricator.wikimedia.org/T268686 (DStrine) [15:44:13] Fundraising Sprint Upbeat Upholstery, Fundraising Sprint Vagranty McVagrantface, Fundraising Sprint We all meet again, Fundraising-Backlog: Set up text that goes next to payments forms on Docker - https://phabricator.wikimedia.org/T266092 (DStrine) [15:46:07] Fundraising Sprint We all meet again, Fundraising-Backlog, Patch-For-Review: Payment Method will not update on contribution record - https://phabricator.wikimedia.org/T266617 (DStrine) [15:47:38] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 1 - https://phabricator.wikimedia.org/T269248 (jbolorinos-ctr) [15:47:40] Wikimedia-Fundraising-Banners: Missing Amazon Pay option for US - https://phabricator.wikimedia.org/T269379 (jbolorinos-ctr) Open→Resolved Fix verified {F33935448} Closing now as Resolved [15:51:34] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, FR-Adyen: Process Adyen IPN messages in parallel (improve TY delivery times) - https://phabricator.wikimedia.org/T233251 (DStrine) p:High→Unbreak! [15:51:56] Fundraising Sprint Octopus hugs, Fundraising Sprint Pseudopretzels, Fundraising Sprint Raw data never hurt anyone, Fundraising Sprint Upbeat Upholstery, and 6 others: Fr-tech chores list - https://phabricator.wikimedia.org/T258527 (DStrine) [16:09:29] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-Banners: Error when trying to complete a recurring paypal transaction - https://phabricator.wikimedia.org/T269303 (spatton) Posting here (late) that we also took recurring paypal out of all the live banners, yesterday aroun... [16:10:43] Wikimedia-Fundraising-Banners: [CANDIDATE] Desktop small fixed element with RML - https://phabricator.wikimedia.org/T269454 (spatton) [16:23:03] hey fr-tech jgleeson ejegg [16:24:27] Hey there [16:24:58] howdy AndyRussG XenoRyet ! [16:29:34] hi AndyRussG [16:32:10] XenoRyet jgleeson ejegg :) [16:35:20] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (EWilson) [16:39:34] (PS1) Ejegg: WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [16:40:11] fr-tech there's a first stab at multiplexing the jobs-adyen queue ^^ [16:40:24] no test yet [16:41:28] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (DStrine) @MBeat33 can you stop this recurring charge? @EWilson these are super rare and there is no obvious way this could have happened.... [16:44:34] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (MBeat33) Sure thing @DStrine, I canceled the two recurrings for this donor. [16:44:35] looks promising ejegg [16:44:42] (CR) jerkins-bot: [V: -1] WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) (owner: Ejegg) [16:53:44] (PS2) Ejegg: WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [16:54:15] (CR) jerkins-bot: [V: -1] WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) (owner: Ejegg) [17:00:59] (PS3) Ejegg: WIP Adyen job queue fan-out [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645397 (https://phabricator.wikimedia.org/T233251) [17:04:06] (PS2) Ejegg: Restore log level filtering [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/645177 (https://phabricator.wikimedia.org/T269371) [17:09:05] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (EWilson) Hi, all. Please pardon me. I should have been more precise. We don't want to lose this donor by cancelling his contributions. Rat... [17:20:09] Wikimedia-Fundraising: Bad font stack for "average donation" message - https://phabricator.wikimedia.org/T269456 (Pythoncoder) [17:29:07] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Run away queries ->fail mail - https://phabricator.wikimedia.org/T268793 (Dwisehaupt) Just ran across another one of these queries and I killed it off. It had caused the processing rate to drop to 0 while it held a... [17:34:01] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (EYener) @DStrine Here is most of what you'd want to know about these recurring endow donors: select * from analytics.all_donations_cube whe... [17:46:18] Fundraising-Backlog, fundraising-tech-ops: Package PrivateBin for FR-Tech hosts - https://phabricator.wikimedia.org/T255063 (Jgreen) [17:46:27] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (MBeat33) @EWilson I was too hasty! Sorry about that, I'm checking to see if we can restart the AF recurring and if not, I will ask this don... [17:46:34] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: Investigate impact of buster upgrade on queue processing - https://phabricator.wikimedia.org/T264819 (Jgreen) Declined→Resolved [17:47:27] Fundraising-Backlog, fundraising-tech-ops: Package PrivateBin for FR-Tech hosts - https://phabricator.wikimedia.org/T255063 (Jgreen) [17:48:03] fundraising-tech-ops, Patch-For-Review: rack/setup/install civi2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T242270 (Jgreen) [17:48:07] Fundraising-Backlog, fundraising-tech-ops: test civicrm and peripheral software on Debian Buster - https://phabricator.wikimedia.org/T253159 (Jgreen) duplicate→Resolved [17:48:30] Fundraising-Backlog, fundraising-tech-ops: Package PrivateBin for FR-Tech hosts - https://phabricator.wikimedia.org/T255063 (Jgreen) [18:09:53] Fundraising-Backlog, Patch-For-Review: Docker dev setup: Minor improvements and doc following initial feedback - https://phabricator.wikimedia.org/T268954 (DStrine) [18:09:58] Fundraising-Backlog, Patch-For-Review: Get civi working on docker - https://phabricator.wikimedia.org/T262975 (DStrine) [18:10:07] Fundraising-Backlog, Patch-For-Review: Docker dev setup: Set up process control - https://phabricator.wikimedia.org/T268685 (DStrine) [18:13:32] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Run away queries ->fail mail - https://phabricator.wikimedia.org/T268793 (DStrine) a:Eileenmcnaughton [18:19:52] Fundraising-Backlog, MediaWiki-extensions-CentralAuth: Code review and release management for CN - https://phabricator.wikimedia.org/T268646 (DStrine) [18:21:24] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 1 - https://phabricator.wikimedia.org/T269248 (jbolorinos-ctr) All issues found have been reported and addressed. Closing task as Resolved, QA sign-off is complete [18:21:34] Wikimedia-Fundraising-Banners: [Candidate] Popular amounts style 1 - https://phabricator.wikimedia.org/T269248 (jbolorinos-ctr) Open→Resolved [18:21:59] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Add CiviProxy to crm repo, write configuration or code for filtering API calls - https://phabricator.wikimedia.org/T268495 (DStrine) [18:22:48] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: unsearchable email domain? - https://phabricator.wikimedia.org/T267760 (DStrine) [18:46:53] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Code review and release management for CN - https://phabricator.wikimedia.org/T268646 (Legoktm) [18:47:33] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (DStrine) Ah sorry. He have canceled these in the past so that was my knee-jerk reaction here. There isn't a way to re-enable these. @EYener... [18:48:03] Wikimedia-Fundraising-Banners: Bad font stack for "average donation" message - https://phabricator.wikimedia.org/T269456 (Pcoombe) [19:13:04] (CR) Jgleeson: Dockerize process-control (8 comments) [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/643767 (https://phabricator.wikimedia.org/T268685) (owner: Jgleeson) [20:13:51] ejegg: I just saw another query issue - I think I'm just gonna throw an exception instead of running that query & ask them to log a phab explaining what they need to find so fr-tech can help with it [20:16:11] (PS1) Eileen: Avoid slow search [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/645427 (https://phabricator.wikimedia.org/T268793) [20:18:00] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Turn off import of RML during peak load - https://phabricator.wikimedia.org/T269185 (Eileenmcnaughton) This is 'done' (off) but lets not close this until it's on again [20:21:34] Fundraising Sprint Xtreme Lolcats, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Run away queries ->fail mail - https://phabricator.wikimedia.org/T268793 (Eileenmcnaughton) I just put up a patch to hard-block these queries with a message to log a phab explaining what the s... [20:24:37] (CR) Eileen: [C: +1] "This looks fine in terms of the template change - but when I check the prod db the field does not exist there yet" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644951 (owner: Cstone) [20:30:28] (CR) Ejegg: [C: +2] "Thanks! It ain't the prettiest solution, but better than leaving the self-destruct button out in the open." [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/645427 (https://phabricator.wikimedia.org/T268793) (owner: Eileen) [20:31:35] (CR) Eileen: "yep - apart from being quick & dirty I thought just removing the option might feel like gaslighting & also it might be good to get the pha" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/645427 (https://phabricator.wikimedia.org/T268793) (owner: Eileen) [20:46:14] Wikimedia-Fundraising-Banners: Bad font stack for "average donation" message - https://phabricator.wikimedia.org/T269456 (Pcoombe) Open→Resolved a:Pcoombe Thanks for the great spot and report! I've fixed this to use the same font stack as the rest of the banner. [20:46:37] (Merged) jenkins-bot: Avoid slow search [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/645427 (https://phabricator.wikimedia.org/T268793) (owner: Eileen) [21:38:43] (PS1) Eileen: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645438 [21:39:20] (CR) Eileen: [C: +2] Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645438 (owner: Eileen) [21:54:54] (Merged) jenkins-bot: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/645438 (owner: Eileen) [21:55:53] Fundraising-Backlog, fundraising-tech-ops, FR-Tech-Analytics: Efficiency of querying views - https://phabricator.wikimedia.org/T259731 (Jgreen) >>! In T259731#6378244, @EYener wrote: > This query takes 42.7 seconds on last run to execute in Superset: > > ` > SELECT donation_type AS donation_type, >... [22:42:58] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/645449 [22:43:20] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/645449 (owner: Eileen) [22:52:51] fundraising-tech-ops: Difficulty accessing mysql through server - https://phabricator.wikimedia.org/T269407 (RMurthy) This seems to have resolved itself -- thanks Dallas! We can close out. [22:53:02] fundraising-tech-ops: Difficulty accessing mysql through server - https://phabricator.wikimedia.org/T269407 (RMurthy) Open→Resolved [23:29:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Recurring-Donations, fr-donorservices: Civi: Recur fail message sent to donor w/non-failed donation - https://phabricator.wikimedia.org/T269372 (MBeat33) cid=37782621 got the email even though 4003088116 was still at status Failing [23:41:34] eileen: I just hopped on to review cstone's patch and saw you'd already reviewed it. I saw your comment about a missing field, did you mean gift source? [23:42:04] this patch https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/644951 [23:43:19] oh wow i totally missed that part at the top that must have snuck in jgleeson [23:44:20] oh I see [23:45:18] I was confused as I can see gift source was added a while back here https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/545121 [23:45:55] (PS2) Cstone: Remove the for your record section when it is a stock. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644951 [23:46:35] i think that was some late night desperate debug that snuck in [23:48:25] got it [23:48:29] ok that looks good to me [23:49:31] (CR) Jgleeson: [C: +2] "Looks good to me! nice work" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/644951 (owner: Cstone) [23:50:21] thanks jgleeson ! [23:52:08] Fundraising-Backlog, Fr-endowment, Recurring-Donations: Review and close loophole allowing recurring Endowment gifts - https://phabricator.wikimedia.org/T269455 (MBeat33) No worries. I reached out to the donor to invite them to make a new donation. [23:54:13] np. I just doubled checked on production and I can see the 'stock_value' field so I'm guessing eileen was looking at one of the other fields [23:56:12] have a good weekend fr-tech, bye for now o/