[02:24:12] Fundraising-Backlog, FR-Ingenico: Convert the rest of old ingenico recurring to new - https://phabricator.wikimedia.org/T257172 (DStrine) [04:23:12] eileen: heyyyy... quick question, if you're there... I'm reviewing this patch https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/608089 which seems to add a menu item... But I can't find where it's added... any thoughts? Thanks in advance! [04:25:04] AndyRussG: it's a drupal menu item (probably Jack didn't know how to do a civi one) - if you clear caches you should be able to put that url are the .org - ie .......org/admin/config/wmf_civicrm/matching_gifts [04:26:03] I'm not sure exactly on the menu - our drupal menu is pretty messy - but you wouldn't see if from a civi screen [04:27:38] eileen: ahhh ok got it! mmmm to clear caches I go to civicrm/menu/rebuild&reset=1 ? [04:27:54] AndyRussG: that is on the civi side [04:28:00] ahh hmmm [04:28:17] drush cc --all would clear drupal [04:28:25] ahhhh okok thx [04:29:25] eileen: also btw I imagine Jack's intent might not have been to make it an easy-to-find menu option... actually I was kinda surprised to see it configurable via a menu. I would have expected something stored in a config file [04:30:28] historically we created drupal modules, more recently civi extensions - but he might have found examples in our drupal modules more readily than the civi side [04:30:43] yeah I mean that certainly could be too [04:33:22] eileen: this is a setting that has to coordinate with some other server-side stuff that won't be configured via the UI... Is there a way to create a setting and have it set via a config file instead, without the menu? It's currently accessed in the other file in that patch, like so: $currentEmployerDataFile = variable_get( [04:33:23] 'matching_gifts_employer_data_file_path', [04:33:25] '/srv/matching_gifts/employers.csv' [04:33:27] ); [04:33:51] as in, I'm wondering if the menu is needed or appropriate then [04:34:02] AndyRussG: yeah good question [04:34:34] I would probably say not - in drupal you have variables and in civi settings and both can be set without a UI [04:35:07] eileen: and do we have other variables or settings that are not ui-settable? [04:35:22] sorry for all this bother btw [04:35:27] Yes - silverpop credentials are set in civicrm.settings.php [04:35:32] (no bother) [04:35:40] :) [04:35:45] I think we have some drupal variables set in settings.php too [04:36:06] eileen: and where is that managed? can I find it in localsettings on frpm? [04:36:12] yes [04:36:43] In general it's better to do things on the civi-side as the drupal stuff will need to be changed when we upgrade drupal [04:36:55] but that's a general rule - not sure about this case [04:37:00] eileen: ah okok gonna take a peek there [04:37:14] I mean I guess they're questions that should be brought up in the CR [04:37:24] otherwise the patch looks very nice and spiffy so far [04:38:13] CiviCRM settings docs https://docs.civicrm.org/dev/en/latest/framework/setting/ [04:44:06] thx! [07:08:18] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Package[php7.0],Package[php7.0-cgi],Package[php7.0-cli],Package[php7.0-curl] [07:09:02] (CR) AndyRussG: "Hey! Really nice work, super clean and spiffy!!! :)" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) (owner: Jgleeson) [07:13:22] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 11 minutes ago with 6 failures. Failed resources (up to 3 shown): Package[php7.0],Package[php7.0-cgi],Package[php7.0-cli],Package[php7.0-curl] [07:18:22] RECOVERY - check_puppetrun on payments1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:06:28] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Add and delete fields from the _all_Wikimedia database (civi export to ESP) - https://phabricator.wikimedia.org/T252245 (KHaggard) Confirming that... [12:08:41] Fundraising-Backlog, MobileFrontend, WMF-Design, Design, and 2 others: Mobile web donate link - https://phabricator.wikimedia.org/T219793 (ovasileva) Open→Resolved Now live on production. Thanks all! [13:49:38] Fundraising-Backlog, MobileFrontend, WMF-Design, Design, and 2 others: Mobile web donate link - https://phabricator.wikimedia.org/T219793 (spatton) This is a milestone and I appreciate everyone who worked on it. [14:43:58] (CR) Ejegg: [C: +1] "Good catch. Little bit of cleanup might be possible." (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [14:55:11] Fundraising-Backlog: Adyen recurring double recurring records in civi - https://phabricator.wikimedia.org/T257067 (Ejegg) Might have to do with an Adyen complication - they give the transaction one ID when it gets authorized, then another when it gets captured. We TRY to just record it with the captured TXN... [14:58:06] Fundraising-Backlog: Adyen recurring double recurring records in civi - https://phabricator.wikimedia.org/T257067 (Ejegg) Preliminary notes on one txn recorded in Civi under two gateway_txn_ids Merchant reference 80483246.2 4755927065750980 - recorded as received 5/21, but shows up in Adyen as received 6/2... [14:58:19] cstone ^^^ some initial notes [15:06:56] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, fundraising Sprint Grep works IRL, MediaWiki-extensions-CentralNotice: Implement redirect for hide banner cookie issue - https://phabricator.wikimedia.org/T251780 (mepps) a:mepps I'm looking at how to do the redirect in Centr... [15:08:08] ah yeah that double transaction ID thing, was there a separate ticket for that ejegg? [15:32:54] cstone: oh i don't think so [15:32:59] that's just how Adyen works [15:33:06] and we try to take it into account [15:33:21] I should add a comment to that bit of the SmashPig code [15:33:52] anyway, I can't see where we'd be saving the old one and still going on to capture the transaction [15:43:11] cstone: ok, so the ID that was sent as source_type direct was definitely the ID of the authorization [15:44:03] unfortunately we're not logging the API calls and responses during the SmashPig recurring processor calls [16:12:35] ejegg: cstone good morning. I see you ate talking about the double adyen thing. Advancement thinks this will block the NL campaign tomorrow. is there any chance it's fixable today? I want to make this UBN. Even if it's not totally fixable, if we could stop them from getting 2 TY letters that would be workable to continue with the campaign [16:12:41] wow typos [16:12:49] * dstrine looks for glasses... [16:14:46] i'm still trying to understand it dstrine [16:14:46] also fr-tech it looks like the email export failed again. Is there anything you can try before Eileen gets online? Would people be able to come to the advancement standup to discuss? [16:14:58] i'll be at the standup [16:15:10] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog: Adyen recurring double recurring records in civi - https://phabricator.wikimedia.org/T257067 (DStrine) p:Triage→Unbreak! [16:16:14] thanks ejegg I know jgleeson had done some stuff on friday too [16:18:47] looks like there's another hung query blocking it [16:18:55] dstrine: yeah I was looking at the unsubscribe stuff and confirmed (I think) the numbers are slightly off from the silverpop_export side. Eileen responded on that point in the latest reply on the thread 'Fail Mail (civi1001) run-job: Silverpop emails - Build export files timed out after 480 minutes' [16:18:58] I think the same one that eileen emailed about over the weekend [16:19:17] jgleeson this time it's just not running the export at all [16:19:31] it's failing on the very first part where it drops the tables from the last run [16:19:43] because there's still a query from the last run executing [16:21:46] according to the timings in the comments it should take less than 3 min [16:22:20] but it's been copying to tmp table on disk for like 12 hrs+ [16:23:09] oh yeah I saw your email over the weekend [16:23:57] so we can kill that query and re-run [16:24:17] but it might just do the same thing again [16:24:35] and even if it worked it would be 8 hrs from now [16:26:10] jgleeson: ejegg so from what I remember through last week, we had to export a ton of new data because we added a bunch of fields. then the new export will be simpler. Are we still on the first step of that process? Is that why we are running into trouble? [16:26:49] dstrine yeah, the currently-deployed code is still regenerating the whole list every night [16:26:58] and now doing it with a bunch of new fields [16:27:07] ugh ok [16:27:19] it looks like the 'highest endownment donation' calculation is the one that's been hanging up for 12 hrs + [16:27:35] ok [16:28:08] but I can't for the life of me see why - it's very similar to other queries we were already doing [16:28:19] ejegg: jgleeson the email export and the adyen bug are top priorities for today. If there is anything anyone can do before eileen gets online, that would be great [16:28:22] ejegg: ok [16:28:26] that's frustrating [16:28:30] Wikimedia-Fundraising-Banners: QA for en6C pre-test on July 8 - https://phabricator.wikimedia.org/T256901 (Pcoombe) [16:31:47] XenoRyet: be a minite or 2! [16:31:51] sorry [16:32:00] Ha, I was just realizing I was running a bit late as well [16:32:02] no worries [16:33:05] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog: Check on old Ingenico to new Ingenico test recurrings - https://phabricator.wikimedia.org/T256181 (EMartin) Great Christine. Looking forward to converting the rest ! [16:36:24] (CR) Jgleeson: "Thanks for all the review!" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) (owner: Jgleeson) [16:49:20] ejegg where in that query is it tripping up? usd or native currency and was it with one specific donor? [16:49:28] if you can see that [16:49:34] i can also look it occurs to me [16:49:37] i was just looking at the code [16:50:16] (PS1) Mepps: Allow redirect on Special:HideBanners on limited redirectUrl [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) [16:50:29] yayy got git review working ^^ :) [16:50:39] AndyRussG for when you return [16:53:40] sorry ejegg dstrine I was slightly tuned out just digging into drupal on the back of some good review from AndyRussG [16:53:47] ejegg: do you need any extra eyes anywhere? [16:53:59] (CR) jerkins-bot: [V: -1] Allow redirect on Special:HideBanners on limited redirectUrl [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) (owner: Mepps) [16:54:10] fr-tech I'm not up to speed with the adyen bug, is anyone on that at the moment? [16:54:18] im looking at it jgleeson [16:54:28] its not a new bug, just that we now have fail mail from CT switchover [16:54:37] jgleeson just trying to figure out why the silverpop_endowment_highest calculation wouldn't be working [16:54:52] cstone oh no, we've been double counting those forever? [16:55:04] well, since we started doing adyen recurring? [16:55:05] is there a task for the adyen bug cstone ejegg? also maybe we should make one for the silverpop export [16:55:10] its not every recurring ejegg [16:55:18] mepps: https://phabricator.wikimedia.org/T257067 [16:55:20] just reading it myself [16:55:44] i think situations where the person double clicks or internet lag that two get sent initially to adyen [16:57:12] cstone: if there is a bandaid that would stop the second TY that would be enough to make advancement happy for the next few days [16:57:20] and unblock the campaign [16:57:23] cstone oh weird, I thought this was happening with the first one that we charge a month after they sign up on site? [16:57:29] but if not, I get it. sounds tricky [16:57:56] cstone so for that one I was looking at, the donor signed up on 5/21 [16:58:34] and then on 6/21 we recorded two donations, one with the ID of the auth and one with the ID of the capture [16:58:48] but... we ALSO got the date wrong for one of those donations [16:58:58] and it has a recieve_date of 5/21 [16:59:58] i THINK we got that date because the message from the listener didn't include any date [17:00:23] and then maybe some logic in the queue consumer tried to reconstruct the date from the c_t data [17:00:42] but i may be imagining things - haven't confirmed that date reconstruction logic still exists [17:00:47] hmm thats the older one then? I was looking on friday at the newer ones let me check one of those [17:01:29] ejegg should we be concerned that silverpop emails export only completely failed on the 4th? [17:02:44] ejegg i'm also seeing a bunch of unknown table errors in the silverpop build files log [17:02:47] i'll put this on a phab [17:12:36] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (mepps) I'm looking at the... [17:24:54] Fundraising-Backlog, fundraising-tech-ops, Epic: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [17:25:45] (PS2) Mepps: Allow redirect on Special:HideBanners on limited redirectUrl [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) [17:26:50] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, fundraising Sprint Grep works IRL, MediaWiki-extensions-CentralNotice, Patch-For-Review: Implement redirect for hide banner cookie issue - https://phabricator.wikimedia.org/T251780 (mepps) The next step will be adding the red... [17:29:27] (CR) jerkins-bot: [V: -1] Allow redirect on Special:HideBanners on limited redirectUrl [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) (owner: Mepps) [17:30:15] (CR) Mepps: "recheck" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) (owner: Mepps) [17:30:59] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, fundraising Sprint Grep works IRL, MediaWiki-extensions-CentralNotice, Patch-For-Review: Implement redirect for hide banner cookie issue - https://phabricator.wikimedia.org/T251780 (mepps) So I'm looking at how we currently d... [17:32:14] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, fundraising Sprint Grep works IRL, MediaWiki-extensions-CentralNotice, Patch-For-Review: Implement redirect for hide banner cookie issue - https://phabricator.wikimedia.org/T251780 (mepps) Also I assumed we'd only want to whi... [17:54:44] Fundraising-Backlog, fundraising-tech-ops, Epic: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) p:Triage→Medium [17:58:22] Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: TS update import back into Civi - https://phabricator.wikimedia.org/T255805 (LeanneS) @Eileenmcnaughton Did another review and the contacts look good! [18:09:51] mepps: hey I'm back at the keyboard [18:10:04] hi AndyRussG! see my patch and comments on the task [18:10:12] for some reason the build is unstable.. [18:12:55] mepps: okok I'll check it out! [18:13:05] I have a meeting in 20 minutes then am free again at the top of the hour [18:13:23] and also in the later afternoon after the backlog stuff [18:13:32] (if you want to work on it together) [18:16:42] AndyRussG i'll have to sign off after backlog if it goes til the scheduled end time [18:17:01] take a look at what i've put up and maybe we can chat after your meeting [18:46:36] AndyRussG any thoughts so far? [18:50:28] mepps: yes just a few suggestions [18:50:41] do you want to talk about it on a call, or here? [18:52:57] let's try here and hop on a call if needed--did you also see my comments on the task? [18:53:13] mepps: ok! yes I saw them [18:53:20] So, +1 to making it an array from the start [18:53:46] I think we may start off with more than one domain because mobile [18:53:53] okay AndyRussG [18:54:43] Second suggestion, even though it'd make it even longer, I'd like to see a more specific global variable name [18:55:02] Like, it'd be nice to be able to see the variable name and understand immediately what it's for even after 2 years of never having seen it [18:55:21] and since it's not used in a lot of code, I think length is less of an issue [18:55:39] so maybe like CentralNoticeAllowHideBannerRedirectURLs ? [18:56:13] ok AndyRussG [18:57:06] Third suggestion is around the logic in SpecialHideBanners [18:57:48] How about first a conditional just to see if the URL parameter is present [18:58:23] So if it's present but it won't redirect properly because not in the list we might warn or something [18:58:41] AndyRussG i don't think we'd want a warning [18:58:46] based on security... [18:58:51] but maybe i'm wrong [18:58:55] mepps: not to the user, but to us [18:59:01] okay AndyRussG [18:59:16] Like so that we notice if we've misconfigured it [19:00:31] fundraising-tech-ops: frdb1001 hardware refresh - https://phabricator.wikimedia.org/T257242 (Jgreen) [19:00:44] Fourth suggestion is that there must be some standard way in the Mediawiki codebase for parsing URLs [19:01:24] We need to use that to make sure the allow list is matching the full host name, and only that, not just any part of the URL [19:01:53] So actually come to think of it, the variable name could be instead CentralNoticeAllowHideBannerRedirectHosts [19:02:58] mepps: hmmm actually now I'm not sure [19:03:39] because we also might want to limit what pages we go to at payments? something to think about I guess [19:04:41] i guess it depends, we can probably assume we'll always go to the thank you page [19:05:23] which is actually on donatewiki [19:05:29] ah right [19:05:38] but does the thank you page have different names in different languages? and we don't want to limit what URL params could be [19:05:52] I think for security the most important thing is to limit the hosts it can redirect to [19:05:53] agreed AndyRussG [19:06:01] maybe let's start with ensuring that? [19:06:41] yeah that's actually easy with preg_match and ^ it looks like [19:06:59] the larger thing is iterating over the array with that preg_match [19:08:02] mepps: I kinda wouldn't want to depend on our own code for URL parsing. There can be plenty of complications, with protocols, URL without protocols, or what if the value includes a newline and the ^ matches the beginning of the second line of the string? [19:08:27] yah, better the url parsing fns than preg_match [19:09:01] mepps: yeah there are definitely maintained, secure facilities for url parsing somewhere in the Mediawiki codebase... I was just looking to try to find them, but didn't see them on first glance. But they're definetly somewhere [19:09:19] finally, there is surely also some Mediawiki standard way to redirect [19:09:51] So we could find and use that, and then move the redirect block to before the call to disable output and reset output buffers [19:09:56] does that make sense? [19:10:02] also apologies if this is a lot!!! ;) [19:10:35] ok AndyRussG [19:10:47] :) mepps thanks so much for digging into this! [19:11:03] unfortunately AndyRussG i feel like external redirects might be trickier [19:11:42] mepps: I'll try to find where other MW code does redirects, to see if there is stuff we can use [19:12:00] The direct setting the header approach might also be best too [19:12:37] i liked that it mimicked the header setting later AndyRussG [19:13:50] fundraising-tech-ops: 2010-2021 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T257244 (Jgreen) [19:14:17] fundraising-tech-ops: 2010-2021 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T257244 (Jgreen) [19:14:17] mepps: right.. but that is a pretty unusual way to do things in Mediawiki. I think it's just like that because no one else did that... There may even be better facilities for that nowadays, too [19:14:24] also just the php fn parse_url [19:14:29] https://www.php.net/manual/en/function.parse-url.php [19:14:39] ejegg: yeh also an option! [19:15:17] fundraising-tech-ops: 2010-2021 fundraising hardware refresh and capex - https://phabricator.wikimedia.org/T257244 (Jgreen) [19:15:19] fundraising-tech-ops: frdb1001 hardware refresh - https://phabricator.wikimedia.org/T257242 (Jgreen) [19:15:39] fundraising-tech-ops: frdb1001 hardware refresh - https://phabricator.wikimedia.org/T257242 (Jgreen) [19:16:53] so if we do it that way, it means we can only match on host [19:17:09] which is okay although as AndyRussG pointed out, we migth want to specify a path [19:17:20] but i'll work on a patch with parse_url [19:17:27] fundraising-tech-ops: procure and deploy new frmail server role - https://phabricator.wikimedia.org/T257245 (Jgreen) [19:23:21] mepps: can we first see how it's done elsewhere in the Mediawiki codebase? [19:23:25] I think that's the first option [19:24:36] Best is always to keep things in line with how they're done elsewhere in Mediawiki. There may be details we're not aware of, specific to WMF and Mediawiki, beyond what the standard PHP function offers [19:25:03] And in general doing it that way keeps things maintainable [19:27:12] mepps: here's the standard Mediawiki way to redirect. Allows arbitrary URL. https://doc.wikimedia.org/mediawiki-core/master/php/classOutputPage.html#a31687d4b17e31f3dae3cf043f2aa9a3c [19:36:03] coming fr-tech! [19:41:37] mepps: here's the Mediawiki function for parsing URLs: https://doc.wikimedia.org/mediawiki-core/master/php/GlobalFunctions_8php.html#a178b2b51ef87926e5daa08f66fbae9b0 [19:57:53] Fundraising-Backlog, fundraising-tech-ops, Epic: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [19:59:16] Fundraising-Backlog, fundraising-tech-ops, Epic: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [20:01:43] katers: there should have been one file that completely uploaded over the weekend [20:02:09] (CR) Ejegg: [C: +2] Force received_date index on highest amount query. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609555 (owner: Eileen) [20:03:09] (Merged) jenkins-bot: Force received_date index on highest amount query. [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609555 (owner: Eileen) [20:03:21] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Q4 FY2019/20 investigate export and upload issues with the silverpop export - https://phabricator.wikimedia.org/T253152 (KHaggard) To address the... [20:04:16] katers: ejegg DatabaseUpdate-20200704235524.csv [20:04:23] one from sunday ^ [20:04:39] let me check my inbox [20:05:43] Ah yes that came through but everything was 0 [20:06:10] it didn't "fail" but no rows populated [20:07:04] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/609843 [20:07:10] sorry I missed that before, updating the phab about it now [20:08:50] !log tools revision is e974147f27 [20:08:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:10:00] Fundraising Sprint Lazy Loading Life, Fundraising Sprint MySQL is YourSQL and WeSQL, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Add and delete fields from the _all_Wikimedia database (civi export to ESP) - https://phabricator.wikimedia.org/T252245 (KHaggard) Update: this ru... [20:12:34] katers: I just kicked off the job again - but the full job takes 3-4 hours [20:12:38] Fundraising-Backlog, fundraising-tech-ops, Epic: map out a civicrm.log_% table data expiration/delete strategy - https://phabricator.wikimedia.org/T257232 (Jgreen) [20:12:45] so it should be done again before the main run [20:13:51] katers: this is going to run through your evening. mepps and jgleeson will be your point of contact tomorrow morning your time. they are up to speed on all this [20:15:48] ok, unfortunately I've reached the end of my work day but I will keep an eye on my inbox in the morning and will reach out to mepps and jgleeson, per dstrine's comment just now if there any issues. thanks! [20:16:19] katers: yeah I understand that's why I wanted to call out who can be point people in your morning [20:16:49] Thanks so much dstrine! [20:29:48] (CR) Eileen: Use summary table to determine opted in (1 comment) [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [20:42:52] Fundraising-Backlog: Update silverpop ftp upload script to push up new matching gifts csv - https://phabricator.wikimedia.org/T257000 (DStrine) [20:43:06] Fundraising-Backlog: Update silverpop ftp upload script to push up new matching gifts csv - https://phabricator.wikimedia.org/T257000 (DStrine) [20:43:45] Fundraising-Backlog: Update silverpop ftp upload script to push up new matching gifts csv - https://phabricator.wikimedia.org/T257000 (XenoRyet) [20:49:00] Fundraising-Backlog: In the DonationInterface form, handle case in which a user clicks on an employer from the dropdown, then types the name of an employer not on the list - https://phabricator.wikimedia.org/T256838 (DStrine) [20:50:56] (PS3) Mepps: Allow redirect on Special:HideBanners on limited redirectUrl [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) [20:54:25] Fundraising-Backlog: In the DonationInterface form, handle case in which a user clicks on an employer from the dropdown, then types the name of an employer not on the list - https://phabricator.wikimedia.org/T256838 (XenoRyet) [20:54:28] (CR) jerkins-bot: [V: -1] Allow redirect on Special:HideBanners on limited redirectUrl [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/609810 (https://phabricator.wikimedia.org/T251780) (owner: Mepps) [20:59:43] Fundraising-Backlog: Process to transfer MG employers data file between civi1001 and frpm1001 - https://phabricator.wikimedia.org/T256825 (DStrine) [21:05:35] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Amy Parker - https://phabricator.wikimedia.org/T257255 (DStrine) [21:06:37] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Enloe Wilson - https://phabricator.wikimedia.org/T257256 (DStrine) [21:09:03] Fundraising-Backlog, FR-AutoTY-Email: Civi: possible email_greeting TY email feature request - https://phabricator.wikimedia.org/T253326 (DStrine) [21:09:23] Fundraising-Backlog, FR-AutoTY-Email: Civi: possible email_greeting TY email feature request - https://phabricator.wikimedia.org/T253326 (DStrine) We are going with 1a. I have updated the task description [21:10:53] mepps: nice!, thanks! ^ just to note, move the code on lines 50 and 51 to below the added code block :) [21:19:52] ejegg: when calling the matching gift sync from the wrapper script should I be setting the the batch size to 0 ? [21:21:55] jgleeson hmm, maybe 1000 ? [21:22:26] or... I guess 0 would actually work to always process all of them [21:24:00] ejegg: can I just talk about https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/609503 with you? [21:24:52] sure eileen - text or vid? [21:24:54] I think maybe the most true to the past (which is goal one) is to delete from suppression list if the master_id is opted_inn [21:25:01] text is fine I think [21:25:28] okok [21:25:57] yep, that sounds like it should be the same as before [21:26:47] (PS9) Eileen: Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 [21:26:57] I think that ^^ might be the right logic.... [21:26:58] (CR) jerkins-bot: [V: -1] Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [21:28:19] (PS10) Eileen: Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 [21:28:30] (CR) jerkins-bot: [V: -1] Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [21:29:18] (although it hasn't passed tests yet - but do you agree with the theory?) [21:29:40] eileen: okok, so delete from suppression where the staging row has opted_in=1 [21:29:45] AND opted_out=0 [21:30:26] ejegg: so I think the previous logic is where the master_email_id is not-not-opted-in [21:30:35] yep [21:30:55] either null or 1 [21:31:15] which this patch still uses as a condition for adding things to the export [21:33:55] (PS2) Eileen: Use inner join to only upload if rows exist [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609561 [21:33:57] (PS2) Eileen: Readme fix [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609562 [21:33:59] (PS2) Eileen: Explicity declare charset for all CiviCRM tables in the mock schema [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609563 [21:34:01] (PS11) Eileen: Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 [21:34:18] ejegg: yeah the thing is the old logic deleted all but the master_email_id from the suppression list [21:34:45] sorry from silverpop_export_staging [21:34:53] (CR) jerkins-bot: [V: -1] Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [21:35:26] so then it only removed from the excluded table if the remaining row was not-not-opted-in [21:35:44] - which is a bit odd if they didn't see the form.... [21:35:49] but I can punt that [21:40:44] (PS12) Eileen: Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 [21:41:56] yeah, I think that's OK - don't suppress as the default if they didn't see any form [21:42:32] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Amy Parker - https://phabricator.wikimedia.org/T257255 (Dwisehaupt) [21:42:51] right the trickiness is what about if they did last time (as defined by 'master_email_id' [21:43:28] however I think if we merge https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/609503 then the suppression list should look like it did [21:43:49] also this one - I wanted to discuss [21:43:50] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/609561 [21:44:19] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Enloe Wilson - https://phabricator.wikimedia.org/T257256 (Dwisehaupt) [21:45:00] basically I saw some non-donors get exported which didn't happen previously - are there any risks that we put donors (e.g MG investigations) into silverpop that shouldn't have gone up? [21:46:17] looking [21:47:43] eileen: oh, so you're worrying we might need to clear out some people we've recently exported [21:47:46] ? [21:47:55] well - yeah I guess so [21:48:16] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Enloe Wilson - https://phabricator.wikimedia.org/T257256 (Dwisehaupt) Approval received from Lisa: ` Date: Mon, 6 Jul 2020 13:25:21 -0700 From: Lisa Seitz Gruwell To: Caitlin Virtue Cc: fr-tech Subject: Re: Approving Civi access for Amy... [21:48:19] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Amy Parker - https://phabricator.wikimedia.org/T257255 (Dwisehaupt) Approval received from Lisa: ` Date: Mon, 6 Jul 2020 13:25:21 -0700 From: Lisa Seitz Gruwell To: Caitlin Virtue Cc: fr-tech Subject: Re: Approving Civi access for Amy a... [21:48:29] I mean in theory anyone who is not a donor but is in our DB would have 'do_not_solicit' or somethig [21:48:40] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Amy Parker - https://phabricator.wikimedia.org/T257255 (Dwisehaupt) [21:48:45] maybe I need to just check who our $0 contacts are [21:48:59] Fundraising-Backlog, fundraising-tech-ops: Setup civi access for Enloe Wilson - https://phabricator.wikimedia.org/T257256 (Dwisehaupt) [21:49:26] oh, so the concern is more that we put ppl on our suppression list that wouldn't have been exported before? [21:49:37] no I'm just thinking [21:49:59] let's say Bill Gates was added to our DB as a person of interest but he has never givenn [21:50:15] I think he *might* have got pushed to silverpop [21:50:25] & could he be accidentally emailed - not surre [21:52:26] hmm, so did we avoid that earlier with the where conditions on that initial select? [21:53:17] right, and now we're not joining on the contribs in that initial select [21:55:28] so this should prevent it in future - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/tools/+/609561 [22:01:15] the contact I hit was 12 - who is a former staff member I think [22:03:53] but this person - 22451 seems to have no reason to be in our db [22:05:58] going to get some breakfast but I think we need to deploy the suppression list change today [22:06:04] & prob that inner joinn [22:06:08] (PS8) Jgleeson: Command to check matching gifts employer data file [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) [22:06:12] but not sure if any cleanup is needed [22:06:47] select email, first_name, last_name, c.id FROm civicrm_contact c LEFT JOIN wmf_donor d ON c.id = d.entity_id INNER JOIN civicrm_email e ON e.contact_id = c.id WHERE c.is_deleted = 0 AND ( lifetime_usd_total = 0 OR lifetime_usd_total IS NULL) LIMIT 100; [22:50:16] fundraising-tech-ops: Determine reason for daily increasing proc count on fran1001 - https://phabricator.wikimedia.org/T256420 (Dwisehaupt) We have nailed this down to the scripts running concurrently. Going to close this out in favor of T256924 where the work to fix up these scripts is being done. [22:50:33] fundraising-tech-ops: Determine reason for daily increasing proc count on fran1001 - https://phabricator.wikimedia.org/T256420 (Dwisehaupt) [22:50:55] fundraising-tech-ops: Determine reason for daily increasing proc count on fran1001 - https://phabricator.wikimedia.org/T256420 (Dwisehaupt) Open→Resolved [22:52:37] dang the old query is runinng [22:54:20] I must not have merged / deployed right [22:54:51] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/609843 (owner: Eileen) [22:55:26] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/609843 (owner: Eileen) [22:58:59] Fundraising-Backlog, FR-AutoTY-Email: Civi: possible email_greeting TY email feature request - https://phabricator.wikimedia.org/T253326 (MBeat33) Thank you @DStrine [22:58:59] !log tools revision changed from e974147f27 to 73557b8038 [22:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:59:22] looks like I hit a pcr bug - "pre-caffiene related" [23:05:36] heh. my brain went immediately to: https://en.wikipedia.org/wiki/Polymerase_chain_reaction [23:05:42] lol [23:05:58] enough coffee might take me there too [23:06:18] been a long time since i've had to think about PCR or anything related. [23:14:32] ok. kids have officially shifted to insanity mode. see you all in the morning. [23:17:52] :-) [23:18:21] I have just pushed a commit to enable the targetsmart import - if anyone can check on it I will deploy - https://phabricator.wikimedia.org/T255805 [23:19:20] ejegg: if you are about can you check ^^ & I will turn it o [23:27:23] yep, looking eileen [23:27:38] (CR) Ejegg: [C: +2] Use inner join to only upload if rows exist [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609561 (owner: Eileen) [23:27:58] (CR) Eileen: "So at a big picture level part of moving to D8 is to move our code to being CMS agnostic - in practice this means leveraging civi more and" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/608089 (https://phabricator.wikimedia.org/T251201) (owner: Jgleeson) [23:28:27] (Merged) jenkins-bot: Use inner join to only upload if rows exist [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609561 (owner: Eileen) [23:29:48] eileen that yaml change looks fine [23:29:53] was there anything else to look at? [23:30:06] not on that side but the suppression one is still not merged [23:30:12] I *think* it's right now [23:30:52] (PS3) Eileen: Readme fix [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609562 [23:30:54] (PS3) Eileen: Explicity declare charset for all CiviCRM tables in the mock schema [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609563 [23:30:56] (PS13) Eileen: Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 [23:30:58] (PS9) Eileen: Silverpop refactor - remove last UPDATE from the silverpop script [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/608991 (https://phabricator.wikimedia.org/T253152) [23:31:00] (PS9) Eileen: Silverpop - build main table incrementally [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609238 (https://phabricator.wikimedia.org/T253152) [23:32:03] (CR) Ejegg: [C: +2] Readme fix [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609562 (owner: Eileen) [23:32:21] !log process-control config revision is 3fe6753e56 [23:32:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:32:55] (Merged) jenkins-bot: Readme fix [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609562 (owner: Eileen) [23:33:15] (CR) Ejegg: [C: +2] Explicity declare charset for all CiviCRM tables in the mock schema [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609563 (owner: Eileen) [23:34:03] (Merged) jenkins-bot: Explicity declare charset for all CiviCRM tables in the mock schema [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609563 (owner: Eileen) [23:35:00] oh jeez, just saw that flip from _out to _in [23:35:08] sorry I missed that in the earlier review [23:35:24] but yes, it does lookcorrect now [23:35:35] cool [23:36:00] I'm just trying to log into silverpop to see if that join bug created any contacts it shouldn't have [23:39:24] (PS1) Cstone: Add day_of_month to thank you email test UI. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/609864 [23:41:18] (CR) jerkins-bot: [V: -1] Add day_of_month to thank you email test UI. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/609864 (owner: Cstone) [23:44:51] (CR) Cstone: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/609864 (owner: Cstone) [23:45:26] (CR) Ejegg: [C: +2] Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [23:46:16] (Merged) jenkins-bot: Use summary table to determine opted in [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/609503 (owner: Eileen) [23:46:38] cool so that just leaves ones that are about speeding it up (plus more of that latter type to be written) [23:49:28] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/609865 [23:50:52] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/609865 (owner: Eileen) [23:51:26] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/609865 (owner: Eileen) [23:53:15] fr-tech could someone look at this real quick? its just adding a parameter to the test email in civi https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/609864/ [23:53:30] i didn't realize it was missing and causing confusion for the ja translation [23:54:37] (CR) Eileen: [C: +2] Add day_of_month to thank you email test UI. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/609864 (owner: Cstone) [23:55:09] thanks eileen ! [23:55:29] yeah ideally would have a test but to get it out quick seems fine [23:57:35] there is a test for when its actually used if that helps? hah [23:57:38] or maybe not [23:58:44] well I didn't block it so up to you if you want to follow up but prob getting late there [23:59:47] I need to modify the template for the en one, I can put it in that patch