[00:01:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: add translations to generic email for failed recurring donations - https://phabricator.wikimedia.org/T255837 (DStrine) Also to differentiate: T244797 is for the monthly convert email (the one you get when you start a recurring transaction from the month... [00:08:21] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: add translations to generic email for failed recurring donations - https://phabricator.wikimedia.org/T255837 (CDenes_WMF) @DStrine Thank you for looking out David-- we definitely do not want to kick off translations until English copy is finalized! But... [00:55:47] thanks for the CR cstone! [00:56:11] hmm, I guess I could rebase that back onto master and rebase the sync patch back on top of it [00:56:46] (PS13) Ejegg: Add matching gift employer export [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607633 (https://phabricator.wikimedia.org/T250328) [01:03:54] (CR) jerkins-bot: [V: -1] Add matching gift employer export [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607633 (https://phabricator.wikimedia.org/T250328) (owner: Ejegg) [01:10:41] grr [01:11:16] haha, timezone fail [01:11:26] 01:03:49 - this month of June\n [01:11:27] 01:03:49 + this month of July\n [01:11:58] CRM_FailureEmailTest::testRender and testSend [01:13:01] wait, June is hardcoded in that test [01:14:16] hah, and it was written on June 2nd [01:14:22] nice time bomb [01:15:58] (PS1) Ejegg: Fix month in SmashPig failure mail test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608747 [01:18:00] (CR) jerkins-bot: [V: -1] Fix month in SmashPig failure mail test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608747 (owner: Ejegg) [01:18:32] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608747 (owner: Ejegg) [01:29:00] (PS2) Ejegg: Fix month in SmashPig failure mail test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608747 [02:06:10] back now... [02:07:35] eileen: want to take a quick peek at https://gerrit.wikimedia.org/r/608747 ? [02:13:00] (CR) Eileen: [C: +2] "Looks like that was my opps" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608747 (owner: Ejegg) [02:13:16] thanks ejegg - looks like I used copy & paste too liberally [02:18:04] :) [02:19:37] (PS14) Ejegg: Add matching gift employer export [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607633 (https://phabricator.wikimedia.org/T250328) [02:20:02] ok, I think ^^^ should merge now [02:20:16] (Merged) jenkins-bot: Fix month in SmashPig failure mail test [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608747 (owner: Ejegg) [02:21:07] ejegg: are you sorted on the MG install/update thing? [02:21:28] eileen so in order to get the job progress table created for new installs, I thought I would just add the executeSqlFile call to the install() function [02:21:32] (CR) jerkins-bot: [V: -1] Add matching gift employer export [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607633 (https://phabricator.wikimedia.org/T250328) (owner: Ejegg) [02:21:47] sounds right [02:21:49] but when I tried that, I got a DB Error:unknown error [02:21:57] hmm - on install? [02:22:10] yeah [02:22:33] lemme see, got a separate patch out [02:22:51] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/607915/1 [02:23:32] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607633 (https://phabricator.wikimedia.org/T250328) (owner: Ejegg) [02:24:37] I guess I need to dig into that more [02:24:59] Eileen do you know if the upgrade_XXXX functions are called on install? [02:25:19] hmm, again, I should debug through this [02:25:44] So I would assume not - because they wouldn't be in drupal & that is what this is based on [02:26:43] ok, got it [02:26:59] but it does install with the schema version at the highest level [02:27:41] Then I need to handle the uninstall too [02:28:14] current test fail seems gerrit related [02:28:24] yeah, that bit is annoying [02:28:34] and seems to be really frequent these days [02:30:23] :-( [02:31:07] ok, looks like it's going to pass this time [02:31:18] ejegg: so that patch looks OK to me - I guess yeah it makes sense to drop the table on uninstall but that's a niceity [02:32:31] yeah, I've added the drop of that job progress table on uninstall to the latest version of the main sync patch [02:33:00] OK - so that one I just looked at is stale now? [02:33:12] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/605638/19/sites/default/civicrm/extensions/matching-gifts/CRM/MatchingGifts/Upgrader.php [02:33:23] I should rebase it and see what it does now [02:33:59] yeah so the install is not in there - just update & delete [02:34:02] (PS2) Ejegg: Ensure matching gifts job table created on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607915 [02:34:29] ok, maybe it's the 'if not exists' bit that caused the problem? [02:34:50] if not exists is good - I am just confused what is in which patch [02:35:18] I'll squash it into the main sync patch if it works [02:35:30] Or I could just merge your update patch now? [02:35:42] I just wanted to get the main sync patch reviewed and having a -1 jenkins score seems to dissuade people! [02:35:49] :-) [02:35:58] the job table is created in the main patch [02:36:04] The thing is public function install() { is still commented out in the main one [02:36:52] (If I understand correctly which is which ) [02:37:19] yeah, I wrote it initially for ppl that had the extension already installed [02:37:46] which is the case on live I think? [02:38:21] yep [02:38:24] so this is the main patch? [02:38:25] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/605638 [02:38:40] yep! [02:39:03] Then I add a test here: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/608486 [02:39:25] but that fails because the table isn't created [02:39:47] (CR) jerkins-bot: [V: -1] Ensure matching gifts job table created on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607915 (owner: Ejegg) [02:41:16] ok, so adding the create script to the install fn seems to screw up ALLL the tests: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/2825/console [02:41:58] is there a typo I'm not seeing? [02:43:33] hmmm, does install() need to return something? [02:43:46] no, doesn't look like it [02:44:23] hmm the upgrade just worked on staging - but note it also created a log table I think [02:44:34] oh, looks like it automatically runs anything like this: glob($this->extensionDir . '/sql/*_install.sql') on install too [02:44:55] ah weird, a log table for the job progress? [02:45:01] I'm surprised [02:45:09] since it's not any kind of entity [02:45:17] and we only ever access it with raw sql [02:45:33] that's not the custom value log table? [02:46:30] we can disable that - we do for some other tables [02:47:19] see wmf_civicrm_civicrm_alterLogTables [02:47:53] the custom value log table seems fine to me [02:49:03] shoot, on frdev I don't see the custom field value table :( [02:49:30] so that part of the install seems to have broken ? [02:49:53] hmm on frdev the extension was already installed [02:50:05] & maybe without the fields. [02:50:09] oh dang [02:50:15] I might need to uninstall & re-install there [02:50:20] oh fun [02:50:27] so that'll take some manual cleanup [02:50:36] good thing we have sql-cli [02:50:40] :-) [02:50:54] let's try running the uninstall [02:50:58] I'm just trying the drush command on staginn [02:51:06] ok, thanks [02:51:12] I'll just watch in the db [02:51:32] so am I right that the state of the commit is that you have responded to all the feed back & are just waiting on final review / tests to pas [02:52:18] hmm staging didn't get an api response :-( [02:52:20] cURL error 7: Failed to connect to gpc.matchinggifts.com port 443 [02:55:13] so on staging I uninstalled & re-installed & got the right stuff (custom table created, no dummy spat) [02:56:19] oh, interesting! [02:56:46] ok, just the job progress table missing [02:58:33] so what on earth is giving me all those db error:unknown error? [03:01:36] no nothing missing on staging [03:02:16] civicrm_matching_gift_job_progress isn't there this time around [03:02:28] just civicrm_value_matching_gift [03:02:32] and log_civicrm_value_matching_gift [03:03:03] oh, let me check what schema_version it got installed with [03:03:39] ok, 1001 [03:03:54] ah - because I installed the version with the install commented out [03:04:09] so when you install it, it doesn't run the upgrade_XXXX functions, but it does examine them to determine the schema version [03:04:22] yep [03:04:49] the patch you had earlier called that create table on install & upgrade line [03:06:06] yep, and that patch is making almost ALL the tests fail with DB Error:Unknown error - https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/2825/console [03:06:11] no idea why [03:11:22] ok, uninstalling and reinstalling locally to see if I can figure out at least which line that's coming from [03:13:01] ejegg: it's logging [03:13:24] ohhh? [03:13:27] it's when the logging is turned on [03:13:46] so one option is to add the alterLogTables hook to the extension & exclude from the extension [03:14:03] so how the heck does logging know what kind of table to make when i'm creating the log table in an executeSqlFile call? [03:14:29] does it parse all the create table statements? [03:14:49] ok, will try that alterLogTables hook [03:15:40] so I just unset the one that I don't want logged [03:17:39] (PS3) Ejegg: Ensure matching gifts job table created on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607915 [03:17:45] let's see if that does it [03:17:47] thanks! [03:18:15] eileen oh, just saw your Q about the state of the patch [03:18:32] Yeah, I think I did respond to all the feedback [03:19:21] and split the tests off in the hopes of getting the main patch merged without automated tests passing yet [03:20:16] OK - I am just testing a bit more locally but I feel like it's working for me & has had a good level of engagement (& obviously my original issues are fixed) [03:20:16] crossing my fingers https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/2826/console [03:23:22] looking good so far! [03:23:31] OK, I can squash that into the main patch then [03:23:37] and even the tests... [03:24:08] (CR) Eileen: [C: +2] "I was able to test this locally & it worked. The upgrade script worked for me on staging and the install script & test are being ironed ou" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605638 (https://phabricator.wikimedia.org/T249924) (owner: Ejegg) [03:25:00] ejegg: It says there are 4 unresolved comments - new gerrit feature I guess [03:25:06] oh? [03:25:07] (CR) jerkins-bot: [V: -1] New MatchingGiftPolicies api action 'sync' using job [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605638 (https://phabricator.wikimedia.org/T249924) (owner: Ejegg) [03:25:11] :P [03:27:16] (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605638 (https://phabricator.wikimedia.org/T249924) (owner: Ejegg) [03:28:13] (CR) Eileen: [C: +2] "Jenkins is finally happy" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607915 (owner: Ejegg) [03:28:25] (PS20) Ejegg: New MatchingGiftPolicies api action 'sync' using job [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605638 (https://phabricator.wikimedia.org/T249924) [03:28:32] (CR) jerkins-bot: [V: -1] Ensure matching gifts job table created on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607915 (owner: Ejegg) [03:28:32] oops, I just squashed them all [03:29:20] I'mma abondon those standalone little ones [03:30:06] (Abandoned) Ejegg: Ensure matching gifts job table created on install [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/607915 (owner: Ejegg) [03:30:25] (CR) Eileen: [C: +2] New MatchingGiftPolicies api action 'sync' using job [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/605638 (https://phabricator.wikimedia.org/T249924) (owner: Ejegg) [03:30:47] (Abandoned) Ejegg: Test for matching gift policy synchronizer [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608486 (https://phabricator.wikimedia.org/T249924) (owner: Ejegg) [03:31:05] ejegg: yeah I realise we cross over there. [03:31:30] so is the test squashed in too now? [03:32:52] yep [03:35:42] it passed! [03:37:09] ejegg: just looking at the 'Incoming reviews' section - a few look like they could be abandoned / set to WIP (which moves them out of that section I think) [03:38:00] (I just set 2 of mine to WIP) [03:41:45] oh? [03:41:53] which in particular? [03:49:14] Fundraising-Backlog, FR-AutoTY-Email: Civi: possible email_greeting TY email feature request - https://phabricator.wikimedia.org/T253326 (MBeat33) fwiw [[ https://donorbox.org/nonprofit-blog/create-a-501c3-tax-compliant-donation-receipt/ | this sample form ]] looks like option 1a, with the donor's first... [03:51:02] ejegg: well in incoming reviews the oldest of yours that I see is 2019 - not sure you are still hoping it will be reviewed [03:51:40] (Abandoned) Eileen: clarified currency in Russian TY footer part 2 [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/533249 (owner: TSkaff) [04:08:48] ok, I'll review those tomorrow [04:09:37] woohoo, it merged! thanks eileen [04:09:42] yay [04:09:46] heading to bed [04:09:55] night [04:24:01] Fundraising-Backlog, FR-AutoTY-Email: translate monthly conversion TY email - https://phabricator.wikimedia.org/T244797 (MBeat33) Open→Resolved Closing this as duplicate, thanks for flagging @CDenes_WMF [04:24:03] Fundraising-Backlog, Epic: EPIC: Make monthly convert default and add options for on/off switch - https://phabricator.wikimedia.org/T235689 (MBeat33) [05:41:04] Hahahah just had the funiest vagrant issue ever [05:41:33] I thought I was on my local vagrant site but I was actually on payments on production [05:41:44] So I was like, huh why don't my code changes do anything!?! [05:42:13] eileen: ^ [05:42:37] oh dear! [05:42:49] that's not how it's supposed to go! [05:43:09] Normally it's the opposite & it's much much safer.... [05:44:53] hehe fortunately I didn't make any actual payments [06:25:58] Fundraising-Backlog: Consider and address possible matching gifter data security issues - https://phabricator.wikimedia.org/T256837 (AndyRussG) [06:35:42] (PS1) AndyRussG: Add comment about escaping for employer autocomplete [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/608757 (https://phabricator.wikimedia.org/T256837) [06:43:13] Fundraising-Backlog: In the DonationInterface form, handle case in which a user clicks on an employer from the dropdown, then types the name of an employer not on the list - https://phabricator.wikimedia.org/T256838 (AndyRussG) [12:32:46] (PS19) ItSpiderman: Cloning Campaign [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/574667 (owner: AlexPinchuk) [12:39:59] (CR) ItSpiderman: "This change is ready for review." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/607461 (owner: ItSpiderman) [12:40:08] (CR) ItSpiderman: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/607461 (owner: ItSpiderman) [12:42:04] (CR) ItSpiderman: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/604279 (owner: AlexPinchuk) [12:46:22] (PS26) ItSpiderman: Campaign filtering [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/604279 (owner: AlexPinchuk) [12:47:01] (PS3) ItSpiderman: Control displaying campaigns based on user preferences [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/607461 [12:47:50] (CR) AlexPinchuk: "This change is ready for review." (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/604279 (owner: AlexPinchuk) [12:58:03] (PS27) ItSpiderman: Campaign filtering [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/604279 (owner: AlexPinchuk) [12:58:10] (PS4) ItSpiderman: Control displaying campaigns based on user preferences [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/607461 [12:59:08] (CR) jerkins-bot: [V: -1] Cloning Campaign [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/574667 (owner: AlexPinchuk) [13:00:50] (CR) ItSpiderman: Campaign filtering (2 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/604279 (owner: AlexPinchuk) [13:06:34] (CR) jerkins-bot: [V: -1] Campaign filtering [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/604279 (owner: AlexPinchuk) [13:06:52] (CR) jerkins-bot: [V: -1] Control displaying campaigns based on user preferences [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/607461 (owner: ItSpiderman) [14:57:51] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KHaggard) Just updating h... [15:17:43] fr-tech, anything to add to the scrum of scrums notes? [15:17:59] ejegg: thanks, nothing here! :) [15:19:55] (CR) Mepps: "Comment is fine but introduced typo." (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/608757 (https://phabricator.wikimedia.org/T256837) (owner: AndyRussG) [15:20:52] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Small number of contacts being inappropriately put on the suppression list - https://phabricator.wikimedia.org/T256522 (mepps) @Eileenmcnaughton c... [15:21:51] (PS2) AndyRussG: Add comment about escaping for employer autocomplete [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/608757 (https://phabricator.wikimedia.org/T256837) [15:22:17] (Abandoned) Mepps: This got tests passing locally [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/607329 (owner: Mepps) [15:22:33] (CR) AndyRussG: "Woooops!!! Thanks for catching that!! :)" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/608757 (https://phabricator.wikimedia.org/T256837) (owner: AndyRussG) [15:29:03] (PS2) Jgleeson: WIP: Command to diff matching gifts employer data file [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) [15:29:17] (CR) jerkins-bot: [V: -1] WIP: Command to diff matching gifts employer data file [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) (owner: Jgleeson) [15:32:25] ah good find about jquery autocomplete automagically escaping stuff AndyRussG [15:45:07] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KHaggard) MG email has be... [16:14:39] jgleeson: oh thanks! just based on a review of the doc we did in tech-talk, then a quick dive into the actual autocomplete code... [16:21:40] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Tony Le - https://phabricator.wikimedia.org/T256808 (Dwisehaupt) SSL certificate created and sent via email. Password sent via SMS. [16:21:59] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Tony Le - https://phabricator.wikimedia.org/T256808 (Dwisehaupt) [16:22:28] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Julianne Joe - https://phabricator.wikimedia.org/T256809 (Dwisehaupt) SSL certificate created and sent via email. Password sent via SMS. [16:23:00] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Julianne Joe - https://phabricator.wikimedia.org/T256809 (Dwisehaupt) [16:40:52] Fundraising-Backlog, Epic: Epic: matching gifts HEP integration (was employer field V2) - https://phabricator.wikimedia.org/T228765 (Jgreen) [16:40:59] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KHaggard) Hi @Eileenmcnau... [16:55:00] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Julianne Joe - https://phabricator.wikimedia.org/T256809 (Dwisehaupt) Superset account created and password sent via SMS. The password can be changed after logging in by using the 'Reset my password' button at the bottom of https://analytics... [16:55:22] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Julianne Joe - https://phabricator.wikimedia.org/T256809 (Dwisehaupt) [16:55:40] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Tony Le - https://phabricator.wikimedia.org/T256808 (Dwisehaupt) Superset account created and password sent via SMS. The password can be changed after logging in by using the 'Reset my password' button at the bottom of https://analytics.frde... [16:56:01] Fundraising-Backlog, fundraising-tech-ops: Create civi cert for Tony Le - https://phabricator.wikimedia.org/T256808 (Dwisehaupt) [17:34:22] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/608915 [17:34:44] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/608915 (owner: Ejegg) [17:49:20] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog: Japanese MC 2nd Upsell Email - https://phabricator.wikimedia.org/T255227 (CDenes_WMF) @Cstone Ichi proofed the email and found 1 mistake: looks like there is a sentence repeated twice in the email. Please see in [[ https://docs.google.co... [17:52:10] Fundraising-Backlog, Epic: Epic: matching gifts HEP integration (was employer field V2) - https://phabricator.wikimedia.org/T228765 (Dwisehaupt) [17:56:12] dstrine erikab : are you around? we're chatting through the flowchart on tech-talk and have a few questions [17:56:29] https://phabricator.wikimedia.org/T254783 [18:02:35] jgleeson: hey I'm around now [18:05:06] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KHaggard) Hi all - the ma... [18:07:38] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog: Flowchart all incoming data from non fundraising sources - https://phabricator.wikimedia.org/T254783 (DStrine) [18:25:55] sorry jgleeson, was tied up, did your questions get answered? [18:32:37] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Coleman Watts - https://phabricator.wikimedia.org/T252693 (Dwisehaupt) Open→Stalled [18:32:42] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Tim Otten - https://phabricator.wikimedia.org/T252694 (Dwisehaupt) Open→Stalled [18:33:41] erikab: no problem! dstrine hopped on and helped [18:37:34] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 8 - https://phabricator.wikimedia.org/T256901 (spatton) [18:40:32] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 8 - https://phabricator.wikimedia.org/T256901 (spatton) [18:42:00] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 8 - https://phabricator.wikimedia.org/T256901 (spatton) [18:48:09] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T256902 (spatton) [18:50:36] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T256902 (spatton) [18:54:11] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: desktop small batch - https://phabricator.wikimedia.org/T256903 (spatton) [18:55:25] erikab: is there a specific type of diagram you're looking for? dataflow like this? https://miro.medium.com/max/1400/1*Fmwy_DkAkmZLCDoZIuHQfw.png [18:58:21] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: mobile large batch - https://phabricator.wikimedia.org/T256905 (spatton) [18:58:47] Fundraising-Backlog, fundraising-tech-ops: Setup shell access and civi access for civi staging for Coleman Watts - https://phabricator.wikimedia.org/T252693 (Lgruwell-WMF) approved [18:59:27] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: mobile small batch - https://phabricator.wikimedia.org/T256910 (spatton) [19:13:47] fundraising-tech-ops, Gerrit: move main fundraising puppet repo to gerrit - https://phabricator.wikimedia.org/T222840 (Jgreen) a:Jgreen→None [19:14:36] Fundraising-Backlog, fundraising-tech-ops, Epic: [epic] PCI gap analysis and Improvements - https://phabricator.wikimedia.org/T77912 (Jgreen) a:Jgreen→None [19:14:53] fundraising-tech-ops: fundraising tech operations GDPR tracking task - https://phabricator.wikimedia.org/T226749 (Jgreen) a:Jgreen→None [19:15:34] fundraising-tech-ops, Patch-For-Review: rack/setup/install civi2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T242270 (Jgreen) a:Jgreen→None [19:19:39] ejegg I was thinking more like a UML activity diagram (or set of them), more like this: https://www.visual-paradigm.com/guide/uml-unified-modeling-language/what-is-activity-diagram/ [19:19:47] there are lots of ways to approach it though [19:20:59] dwisehaupt: I did a run-job & then killed it on the command line & was wrong to kill (but I think it's dead) & now there is a lock file I can't get rid of [19:21:26] Wikimedia-Fundraising-CiviCRM: Engage Weekly Import not importing due to error message with Direct Mail Appeal - White Mail - https://phabricator.wikimedia.org/T256912 (RLewis) [19:21:54] dwisehaupt: need to run run-job silverpop_emails_upload_files [19:22:10] fr-tech urgent email from ccogdill [19:22:24] looks like we need to repush the export? [19:22:37] jgleeson we're discussing it in civi fortnightly [19:22:53] oh? [19:22:58] ok I don't see that meeting [19:23:00] eileen: ok. [19:23:02] sorry! [19:23:26] huh jgleeson weird you're not on it.. [19:23:40] no worries! [19:23:49] no you are on the invite [19:24:20] dwisehaupt: just need this file gone /tmp/jenkins-upload.py.lock [19:24:21] I don't think so but it's fine aslong as you saw the email [19:24:36] jgleeson it was because i'd tried to add you. done now [19:26:31] Fundraising-Backlog, fundraising-tech-ops: Monitor and investigate possible event dropping by Kafkatee - https://phabricator.wikimedia.org/T239564 (Jgreen) Stalled→Resolved p:Triage→Medium a:Jgreen We've improved kafkatee monitoring significantly since this task was created. We are no... [19:26:43] eileen: cool. was just going to verify that was the file. [19:27:16] that process is still running. [19:27:20] 10202 [19:27:30] ok it is? [19:27:36] yes. [19:27:46] well, at least it's still in the process list. [19:28:16] fundraising-tech-ops, observability, Epic: [Epic] overhaul fundraising cluster monitoring - https://phabricator.wikimedia.org/T91508 (Jgreen) [19:28:18] Fundraising-Backlog, fundraising-tech-ops, FR-Smashpig: Write Grafana monitors for SmashPig database things - https://phabricator.wikimedia.org/T162946 (Jgreen) [19:28:34] yeah. verified it's using cpu and doing work. [19:28:47] dwisehaupt: OK I guess that's good then [19:28:50] tail silverpop_emails_upload_files-20200701-192256.log [19:28:58] isn't showing much [19:29:20] Fundraising-Backlog, fundraising-tech-ops, FR-Smashpig: collect SmashPig database/queue metrics to fundraising prometheus/grafana - https://phabricator.wikimedia.org/T162946 (Jgreen) [19:29:37] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T256902 (jbolorinos-ctr) p:Triage→High Screenshot Test Results: - Control - https://app.crossbrowsertesting.com/public/i15b74a2dc92badb/screenshots/z7811fa291b4d9a9a378 - Blue Gradiant Bac... [19:29:48] Wikimedia-Fundraising-Banners: 2020-07-08 en6C pre-test: desktop large batch - https://phabricator.wikimedia.org/T256902 (jbolorinos-ctr) a:jbolorinos-ctr [19:30:44] looking at the strace output. looks like it may be spinning waiting on a resource. [19:30:52] would you like for me to kill it off proper? [19:31:56] dwisehaupt: yeah maybe [19:32:01] I'll re-kick [19:32:54] if anyone in frtech can review the ones that will put up the data they start with https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/607652 - which is also the biggest one [19:33:15] looking eileen [19:33:57] thanks [19:35:02] eileen: ok. process killed and lock file removed. [19:35:48] ok - I just retriggered run-job silverpop_emails_upload_files [19:36:29] the log looks much cleaner this time [19:36:59] eileen: there's a lot of stuff removed here https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/607652/3/silverpop_export/rebuild_schema.sql [19:37:31] are they moved into other files? [19:37:38] as in a reshuffle [19:37:59] oh wait [19:38:09] jgleeson: I think there is movement within the file that is hard to read [19:38:18] it looks like indent changes? [19:38:39] yeah the diff looks horrible in new gerrit [19:39:18] it's all just case changes i think? [19:39:27] oh, and indent [19:39:40] gerrit lets you ignore the blankspace but not the case [19:40:19] heh, yeah, i just searched out that setting. :) [19:41:36] jgleeson: yeah I think I may have altered my ide formatting & it is also caps [19:42:01] I'm just opening up pycharm to see if the diff view is better [19:42:34] which it is [19:42:39] I find phpstorm diffs pretty good [19:43:04] yeah the jetbrains stuff works well mostl [19:43:06] y [19:44:04] ok that looks more clear, it's just a bunch of reformat changes [19:44:14] clearer* [19:45:01] varbinary! [19:45:37] I can't say I've ever seen that used elsewhere [19:46:56] ok code looks good I'll just comment out the SQL that doesn't work on vagrant and run it to test [19:47:01] fundraising-tech-ops, observability, Epic: Create Grafana dashboard for Online Fundraising to monitor live campaigns - https://phabricator.wikimedia.org/T256916 (Jgreen) [19:50:54] jgleeson: I think varbinary comes from our live servers -ie create a table there & use show create table [19:50:59] but, not in a good way [19:52:01] eileen: I'm getting some failures running due to some of the tables not being fully qualified [19:52:34] e.g. pymysql.err.ProgrammingError: (1146, "Table 'silverpop.civicrm_contribution' doesn't exist") [19:52:35] jgleeson: argh - that sounds like a real issue - you mean no precedinng civicrm. [19:52:42] yeah [19:53:00] ok let me fix that - [19:53:36] happening on civicrm_contribution, civicrm_address, civicrm_country [19:53:54] (PS6) Eileen: Silverpop refactor - reduce queries to build country & language [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/607652 [19:54:32] (PS7) Eileen: Silverpop refactor - reduce queries to build country & language [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/607652 [19:54:34] jgleeson: does that fix [19:54:46] I have mine all mapped to the same db locally so didn't hit it [19:55:15] I think so, it's now failing in the usual place due to the missing prospect db which I'll comment out and tyr now [19:55:17] try* [19:55:33] prospect db tbl* [19:58:19] ok that looks good [19:58:19] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog: Japanese MC 2nd Upsell Email - https://phabricator.wikimedia.org/T255227 (Cstone) @DStrine @TSkaff Would the recurring notification footer still need the tax info? It might be confusing since they haven't donated that amount yet. [20:01:00] (CR) Jgleeson: [C: +2] "Updated queries working for me. Export working and tests passing. Nice!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/607652 (owner: Eileen) [20:01:16] thanks [20:01:23] do the others need rebasing now? [20:01:38] nope they look ok [20:01:58] (Merged) jenkins-bot: Silverpop refactor - reduce queries to build country & language [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/607652 (owner: Eileen) [20:02:08] I'm still getting to grips with the new gerrit UI [20:02:30] (PS3) Eileen: Prefix all total colums in our staging tables with foundation_ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608492 (https://phabricator.wikimedia.org/T252245) [20:02:40] actually it did! [20:02:53] I just rebased this one https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/608492 [20:04:17] there is a whole lotta field renaming in our internal tables to make it clear which fields are just foundation fields [20:04:56] (PS4) Eileen: Prefix additional internal fields with foundation_ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608493 (https://phabricator.wikimedia.org/T252245) [20:05:06] that one looks good also [20:06:07] the new gerrit UI is cleaner but it doesn't address the lack of readability of the comment stream or the inability to add screenshots - I'm hoping the move-to-gitlab goes through [20:07:08] agreed on all points eileen [20:07:20] I'm struggling with the new gerrit UI atm [20:07:36] I'm finding it bearable on dark mode [20:08:23] the CAPS overload irks me [20:09:00] I feel like the relation chain takes up too much space [20:09:16] anyway [20:10:07] (CR) Jgleeson: [C: +2] "Working well for me. Tests passing. Can see changes in exported csv. Nice!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608492 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [20:11:33] dang the tools dir actually wasn't deployed - I could swear I did [20:11:35] (Merged) jenkins-bot: Prefix all total colums in our staging tables with foundation_ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608492 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [20:11:52] !log tools revision changed from aab96444df to 6f38c14fe3 [20:11:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:12:08] eileen: the only thing I noticed is the topic/branch name is called "testfix" [20:12:17] (PS4) Eileen: Further field renaming to clarify foundation vs all funds [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608498 (https://phabricator.wikimedia.org/T252245) [20:12:31] jgleeson: I don't think that means anything [20:12:37] oh wait you're updating the test [20:12:43] will just be because I had to rebase or something [20:12:47] yeah I guess. [20:13:38] actually likely because I had to go back & fix the test as it it was testing the old field name... [20:15:34] (PS2) Eileen: Remove duplicate line [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608503 (https://phabricator.wikimedia.org/T252245) [20:16:02] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: TS update import back into Civi - https://phabricator.wikimedia.org/T255805 (mepps) I'm looking at this, and trying to update the saved field mappings to ignore address (in staging). For some reason I k... [20:16:05] (PS4) Eileen: Add endowment_latest and endowment_highest fields [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608501 (https://phabricator.wikimedia.org/T252245) [20:16:15] (PS2) Eileen: Refactor of staging queries - use stats table, don't update staging table. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608506 (https://phabricator.wikimedia.org/T253152) [20:16:23] (PS2) Eileen: Move autocommit back to the top [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608507 [20:16:31] (PS2) Eileen: Add endowment_highest_usd_amount field to final output [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608512 (https://phabricator.wikimedia.org/T252245) [20:16:40] (PS4) Eileen: Populate field for all_funds_highest_donation_date annd all_funds_highest_usd_amount and all_funds_latest_native_amount [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608513 (https://phabricator.wikimedia.org/T252245) [20:16:45] fundraising-tech-ops, observability, Epic: Create Grafana dashboard for Online Fundraising to monitor live campaigns - https://phabricator.wikimedia.org/T256916 (Jgreen) [20:16:48] (PS2) Eileen: Simplify handling of foundation_has_recurred_donation [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608514 [20:16:56] (PS2) Eileen: Silverpop refactor - further reduce updating of silverpop table [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608515 (https://phabricator.wikimedia.org/T253152) [20:17:04] (PS2) Eileen: Update timings [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608549 [20:18:08] (CR) Jgleeson: [C: +2] "Updates working for me and fields present in exported csv. Tests also passing. Nice!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608493 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [20:18:52] (Merged) jenkins-bot: Prefix additional internal fields with foundation_ [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608493 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [20:25:34] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/608952 [20:25:53] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/608952 (owner: Eileen) [20:26:30] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/608952 (owner: Eileen) [20:27:15] !log tools revision changed from 6f38c14fe3 to 806e2b4412 - [20:27:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:28:00] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog: Japanese MC 2nd Upsell Email - https://phabricator.wikimedia.org/T255227 (TSkaff) That makes sense to me @Cstone, and good catch. I amended [[ https://docs.google.com/presentation/d/1mau4tFwYqxLMaNCtf-uH9FfjG020G7gGECBp61DRvV4/edit#slide... [20:29:35] I just kicked off the export files job again - will do twice today if we get stuff merged but I want to check it [20:32:27] ejegg: I'm still seeing some weirdness with the matching gifts sync. Just now I run it and got back no results (count=0) and ran it again as I was expecting results and got back 7 results with the same api call [20:32:51] run/ran... one of them [20:32:55] I think run [20:32:57] but it sounds wrong [20:33:55] ejegg: https://phabricator.wikimedia.org/F31913105 [20:34:29] it might be a local thing but the worry is if it's not and it happens live we might miss a month due to whatever the issue is [20:34:52] sorry not a month, miss a batch depending on whatever frequency we decide to run the thing on [20:41:30] ejegg: I deleted that first screenshot URL to remove the details about the company included. Here is a cropped version showing the count difference when running the same api call twice https://phabricator.wikimedia.org/F31913111 [20:41:54] three times * (the one in the middle hung so I had to cancel it) [20:42:09] I feel like the API might be slow for me [20:42:39] but no others folks have reported issues so maybe it's at my end [20:43:47] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: TS update import back into Civi - https://phabricator.wikimedia.org/T255805 (mepps) I uploaded the import file to our civi server in tmp. [20:44:09] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: TS update import back into Civi - https://phabricator.wikimedia.org/T255805 (mepps) and dev [20:44:21] hmm, odd [20:44:30] sorry jgleeson, just looking now [20:44:58] np just wanted to show you what I was running into. it might just be me [20:45:17] I think I was getting consistent result in my own testing [20:45:28] but we should definitely keep an eye on it when we put it in production [20:53:56] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: TS update import back into Civi - https://phabricator.wikimedia.org/T255805 (mepps) Basically if we're trying to ignore address fields, we'd want to update the saved field mapping to do not import on th... [21:17:25] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Patch-For-Review: Create job to update/generate employer dataset for usage within paymentswiki - https://phabricator.wikimedia.org/T251201 (Dwisehaupt) The directory for storing these downloads has been added to the civicrm role mani... [21:30:33] ah man. being on the wiki vpn seems to constantly cut out my music streaming [21:32:46] jgleeson: I'm guessing you have done a push on that review & are looking at something else - is anyone else OK to pick up on the next one ? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/608498/4 [21:34:05] sorry eileen I moved back to the MG stuff I'm working on [21:34:26] I'll check them out first thing in the event no one else catches them [21:34:51] jgleeson: that's fine - I'm just trying to rustle up some more enthusiasm elsewhere :-) (I know it's a bit of a slog) [21:35:42] to add my endorsement they're really straightforward patches to review fr-tech :) [21:40:06] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KHaggard) Updating here a... [21:40:39] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Add and delete fields from the _all_Wikimedia database (civi export to ESP) - https://phabricator.wikimedia.org/T252245 (KHaggard) The field chang... [21:59:34] (CR) Ejegg: [C: +2] Further field renaming to clarify foundation vs all funds [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608498 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [22:00:24] (CR) Ejegg: [C: +2] Remove duplicate line [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608503 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [22:00:26] (Merged) jenkins-bot: Further field renaming to clarify foundation vs all funds [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608498 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [22:01:12] (Merged) jenkins-bot: Remove duplicate line [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608503 (https://phabricator.wikimedia.org/T252245) (owner: Eileen) [22:09:12] fundraising-tech-ops: Add logging and lockfiles to fr analytics scripts where needed - https://phabricator.wikimedia.org/T256924 (Dwisehaupt) [22:11:18] fundraising-tech-ops: Determine reason for daily increasing proc count on fran1001 - https://phabricator.wikimedia.org/T256420 (Dwisehaupt) [22:19:56] (PS3) Jgleeson: WIP: Command to diff matching gifts employer data file [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) [22:20:09] (CR) jerkins-bot: [V: -1] WIP: Command to diff matching gifts employer data file [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) (owner: Jgleeson) [22:21:22] (PS4) Jgleeson: WIP: Command to diff matching gifts employer data file [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) [22:46:47] ejegg: the next commit adds most of the new fields so would be great to get that through too - I know these are a drag :-( [23:03:11] yep yep, looking at it now eileen [23:03:20] thanks! sorry to push you on it [23:04:41] at the end of that chain the new fields are populated but still working on reducing the update size / time [23:07:56] the new highest / latest tables don't have currency symbol - that's ok though? [23:35:22] !log updated fundraising CiviCRM from 391d0fdf75 to a48caf0f37 [23:35:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:38:44] ok, updated the table [23:38:57] or created it rather [23:46:07] ejegg: I think so - they didn't ask for it [23:55:42] dang, getting some primary key validations when testing locally [23:55:54] probably have some contacts with two primary addresses [23:58:07] yeah I logged a separate ticket for adding checks / fixes for that on live [23:59:31] ok, yep, that was all.