[00:01:14] (PS1) Ejegg: Fix flake8 errors [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/506031 [00:01:35] (CR) jerkins-bot: [V: -1] Add extension to cleanup sort name for orgs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) (owner: Eileen) [00:10:47] (PS1) Eileen: Drupal security update [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506033 [00:10:49] (PS1) Eileen: Re-apply WMF patches [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506034 [00:12:06] (CR) jerkins-bot: [V: -1] Drupal security update [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506033 (owner: Eileen) [00:14:17] (CR) Ejegg: [C: +2] "Passes with the following re-apply. Guess we should grab those module updates, too?" [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506033 (owner: Eileen) [00:14:22] (CR) Ejegg: [C: +2] Re-apply WMF patches [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506034 (owner: Eileen) [00:14:31] thanks eileen ! [00:14:40] Want me to prep the ctools update? [00:14:52] ejegg: no worries - did I skip ctools? [00:15:07] oh - diff repo - will commit [00:15:10] it's just in a different repo [00:15:15] heh, yah [00:15:30] (CR) jerkins-bot: [V: -1] Drupal security update [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506033 (owner: Eileen) [00:15:32] (CR) jerkins-bot: [V: -1] Re-apply WMF patches [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506034 (owner: Eileen) [00:16:10] (PS1) Eileen: Drupal security update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506035 [00:16:25] (CR) Ejegg: [V: +2 C: +2] Drupal security update [wikimedia/fundraising/crm/drupal] - https://gerrit.wikimedia.org/r/506033 (owner: Eileen) [00:17:24] ejegg: this one passes now that civi one is merged - https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501128/ - [00:17:42] but tomorrow is good [00:18:25] k, will look tomorrow [00:20:19] (CR) jerkins-bot: [V: -1] Drupal security update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506035 (owner: Eileen) [00:21:01] oh weird [00:21:29] how did that make a dedupe test fail? [00:21:59] lots of Undefined index: sort_name [00:22:26] hmm, and the benevity test [00:25:58] is that all from the core update? [00:27:35] (PS2) Ejegg: Drupal security update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506035 (owner: Eileen) [00:27:42] ejegg: nope it’s in the extension - just found it & hopefully fixed [00:27:59] ahh, i was just trying a rebase [00:28:29] (PS8) Eileen: Add extension to cleanup sort name for orgs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) [00:28:33] so before this civi update it failed for a core reason (lack of support in core) so I hadn’t picked up the enotice [00:29:50] ejegg: if we merge that second drupal security one I’ll merge to deployment [00:30:36] (CR) Ejegg: [C: +2] Drupal security update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506035 (owner: Eileen) [00:30:42] thanks [00:31:46] (PS3) Eileen: Create new recurring records with a status of Pending [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501131 (https://phabricator.wikimedia.org/T218754) [00:33:46] (PS1) Eileen: Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506036 [00:34:50] (CR) Eileen: [C: +2] Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506036 (owner: Eileen) [00:38:03] go zuul go [00:38:39] (CR) Ejegg: [C: +2] Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T217565) (owner: AndyRussG) [00:38:49] (PS21) Ejegg: Create CampaignChange hook [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/493459 (https://phabricator.wikimedia.org/T217565) (owner: AndyRussG) [00:39:12] ejegg: woo thanks!! [00:43:36] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506037 [00:45:18] ejegg: there are a couple of commits in the merge commit unrelated - I think they are straight forward but can you eyeball [00:49:42] eileen: there were a couple of mine that should be fine to push out [00:50:05] (CR) Ejegg: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506037 (owner: Eileen) [00:50:13] eileen: yep, all looks good to me! [00:50:20] ejegg: yay [00:50:44] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/506037 (owner: Eileen) [00:53:06] ejegg: we still just use a process-control commit to stop the queues? [00:53:39] eileen: yep, and a couple minutes wait for existing processes to end [00:54:19] it feels a bit fragile in that we keep re-using an old commit but new jobs could have been added? [00:58:35] eileen: oh, there's a shell script [00:58:42] ejegg: ah [00:58:45] ./disable civi1001 [00:58:54] & I run that on frpm? [00:59:09] yeah, the shell script is right in the settings repo [00:59:40] just checking on the name [01:00:07] yep, in process-control, ./disable civi1001 [01:00:10] ejegg: ok - pushed that as a commit [01:00:21] cool, peeking at it [01:00:52] yep, that should do it [01:01:07] ejegg: ok - will deploy it [01:02:34] !log process-control config revision is 13b9eefe7b [01:02:34] (jobs disabled) [01:02:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:03:11] ejegg: run-job -r is empty so they are stopped? [01:04:47] I’m ready to push it out [01:06:40] hitting go [01:06:41] !log civicrm revision changed from 31982324b8 to 468f85e524, config revision is 13b9eefe7b [01:06:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:10:36] upgrade script is running - actually I suspect it’s just re-checking log tables now [01:15:05] ah [01:18:23] it seems to have worked although the upgrade is still running in drush [01:18:45] no it has finished [01:19:02] so in terms of slow starting… [01:19:09] what do you normally try [01:26:51] !log jobs restarted process-control config revision is ef6d4761e5 [01:26:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:33:22] eileen: sorry! lessee what has them [01:33:34] ejegg: I’ve started all the jobs now... [01:33:43] cool [01:33:45] I ran one on civi first [01:33:48] but all seems ok [01:38:15] Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (Eileenmcnaughton) @LeanneS the patch for the primary emails just went out [01:40:50] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Activities tab- multi select option for exclude filter - https://phabricator.wikimedia.org/T203057 (Eileenmcnaughton) @NNichols just depl... [01:51:09] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Activities tab- multi select option for exclude filter - https://phabricator.wikimedia.org/T203057 (NNichols) I already used it! Amazing... [02:18:27] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Activities tab- multi select option for exclude filter - https://phabricator.wikimedia.org/T203057 (Eileenmcnaughton) lol - you were quic... [02:18:30] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Activities tab- multi select option for exclude filter - https://phabricator.wikimedia.org/T203057 (Eileenmcnaughton) Open→Resolved [04:43:12] PROBLEM - check_gcsip on payments2002 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [04:45:14] PROBLEM - check_gcsip on payments2003 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [04:45:14] PROBLEM - check_gcsip on payments2001 is CRITICAL: CRITICAL - Socket timeout after 121 seconds [04:46:10] RECOVERY - check_gcsip on payments2002 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.216 second response time [04:48:14] RECOVERY - check_gcsip on payments2003 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.211 second response time [04:48:14] RECOVERY - check_gcsip on payments2001 is OK: HTTP OK: HTTP/1.1 200 OK - 343 bytes in 0.236 second response time [05:09:45] (PS1) Abijeet Patro: OutputPage::parseInline has been deprecated since 1.32 [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506044 (https://phabricator.wikimedia.org/T213978) [05:56:34] (PS1) Eileen: Update extended report [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506049 (https://phabricator.wikimedia.org/T218616) [09:21:30] Fundraising-Backlog, MediaWiki-Vagrant, Patch-For-Review: Add mediawiki-vagrant php7.2 xdebug support - https://phabricator.wikimedia.org/T220406 (Mainframe98) The change above results in the message `Cannot load Xdebug - it was already loaded` whenever I run a maintenance script in vagrant. Xdebug w... [10:04:51] Fundraising-Backlog, MediaWiki-Vagrant, Patch-For-Review: Add mediawiki-vagrant php7.2 xdebug support - https://phabricator.wikimedia.org/T220406 (jgleeson) Oh shoot, sorry about that! I don't see xdebug available via apt for php 7.2 on my own mw-vagrant install (prior to this patch), only xdebug 2... [10:28:14] Fundraising-Backlog, MediaWiki-Vagrant, Patch-For-Review: Add mediawiki-vagrant php7.2 xdebug support - https://phabricator.wikimedia.org/T220406 (jgleeson) revert patch is here: https://gerrit.wikimedia.org/r/#/c/mediawiki/vagrant/+/506121/ [13:14:11] Fundraising-Backlog: New mobile iFrame errors - https://phabricator.wikimedia.org/T221385 (krobinson) Two more examples, both SE: #578881 //Trying to donate via Visa from my IPhone, but when I try to fill in the card number, the "form" disappears. // #578816 //I'm trying to put in the credit card number bu... [14:38:54] Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (LeanneS) @Eileenmcnaughton Hooray! I'll test today and let you know my results. [14:45:38] howdy folks [14:57:56] howdy ejegg [14:58:27] thanks for the +2 on the revert :\ [14:58:43] I need to dig into what's going on regards that [14:58:43] hi jgleeson ! [14:59:04] I guess just none of us noticed the earlier patch [14:59:37] what's confusing me is I don't see the expected results of the earlier patch [14:59:58] and my apt sources on vagrant don't see the 7.2 package [15:00:03] so not sure what I'm missing [15:00:48] it would be in the apt.mediawiki.org repo [15:02:08] oops, that's apt.wikimedia.org [15:02:34] I can see it here https://apt.wikimedia.org/wikimedia/pool/thirdparty/php72/x/xdebug/ [15:02:47] but `apt show php-xdebug` isn't showing it [15:03:27] apt updating here to try [15:03:33] it still shows the 2.5.0 version associated with php 7.0 (that's also installed) [15:03:59] I'm wondering if there is some type of precedent thingy [15:04:17] hmm, right, I see the same [15:04:45] trying dist-upgrade to see what that does [15:04:52] since it was mentioned in the ticket [15:05:15] that updated the usual packages for me but still no luck with seeing xdebug 7.2 [15:05:21] for 7.2* [15:07:32] are you talking with -releng? [15:08:07] nope not currently [15:08:21] been trying to wrap up the exchange rates stuff, currently fighting with google permissions [15:14:22] jgleeson: do you need me to open up editing on the sheet I put up initially? [15:15:03] no the permissions issue is with the client-side script this time. The master rates sheet and the auto-update stuff is working fine I think [15:15:35] the current issue is due to google blocking SpreadsheetApp.openById() when called from custom function [15:15:37] s* [15:15:57] https://developers.google.com/apps-script/guides/sheets/functions#advanced [15:16:07] just trying to find a way around it [15:16:28] the logic all works, in development, but then when testing from a spreadsheet I'm getting hit with perms issues [15:17:35] Fundraising-Backlog: New mobile iFrame errors - https://phabricator.wikimedia.org/T221385 (Ejegg) OK, if this is happening while the refactor code from last week is still gone, I'm starting to wonder if it's just the known suboptimality of trying to shoe-horn the hosted checkout form into the iframe. For th... [15:17:47] as I'm only reading from the master sheet, I don't want to require auth for everyone who clones/uses the spreadsheet as that would be an annoyance [15:18:35] the final code is here: https://script.google.com/a/wikimedia.org/d/13AUtm0ziVLQKO7jvL6I5PnJbROR1HMW8ltWiUJ6LM2gVFTQ9xG9be1vj/edit [15:19:30] jgleeson: ah, there's no cell-formula syntax for cross-sheet vlookup? [15:19:41] I thought I'd be clever and retain the existing custom function name so that it "seamlessly" worked. Google had other ideas :) [15:20:46] does the importRange function ask for the same perms? [15:20:55] ejegg, there is, you can use importrange() but I initially went down that road and ended up with an unwieldy inline function due to handling all the error states [15:21:05] ah, fun [15:21:12] so thought this would be a cleaner approach [15:21:19] the only problem being it doesn't work... [15:21:44] I'm hopeful I can swap out the problematic calls with their read-only equivalents [15:22:41] in the table here https://developers.google.com/apps-script/guides/sheets/functions#advanced it mentions that you can use most Spreadsheet get*() methods, which I'm looking at now [15:23:56] a mix of import range with vlookup might be the alternative [15:24:04] hmm, also Cannot open other spreadsheets (SpreadsheetApp.openById() or SpreadsheetApp.openByUrl()). [15:24:18] yeah, openById() is the culprit [15:25:08] oh hey, scrum of scrums time. [15:25:20] I'll mention the xdebug stuff [15:25:25] any other news, fr-tech? [15:26:18] none from me ejegg [15:26:30] fr-tech, gonna pickup Oscar from nursery. be back for standup! [15:26:34] should the ssl stuff you and casey were discussing be mentioned ejegg? [15:28:37] sure [15:29:41] I'll investigate that more shortly [15:30:06] should have info in the logs by now [15:35:17] fundraising-tech-ops, Operations, decommission, ops-codfw, Patch-For-Review: decom rigel.frack.codfw.wmnet - https://phabricator.wikimedia.org/T202535 (Papaul) papaul@fasw-c-codfw> show interfaces descriptions | match "ge-[0-1]/0/14" ge-0/0/14 down down DISABLED ge-1/0/14 down... [15:39:05] Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (LeanneS) So I just tested a few and now it seems like it's creating duplicate records for all of th... [15:45:23] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, and 3 others: Update opt-in queue consumer to be able to create new donor records - https://phabricator.wikimedia.org/T217710 (mepps) @cstone where is y... [15:46:14] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CRM contribution tracking writing should have a switch to use the new infrastructure - https://phabricator.wikimedia.org/T215463 (mepps)... [15:46:55] fr-tech I've got to present at the spark project showcase, and that overlaps with the daily standup today [15:47:17] Cool ejegg! [15:59:22] Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (LeanneS) @Eileenmcnaughton Sorry false alarm - the dedupe process was just taking longer than expec... [16:03:54] (PS3) Cstone: WIP: Create new contact with just an email from opt-in. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/505686 (https://phabricator.wikimedia.org/T217710) [16:05:25] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, and 3 others: Update opt-in queue consumer to be able to create new donor records - https://phabricator.wikimedia.org/T217710 (Cstone) @mepps it's here... [16:08:25] (CR) jerkins-bot: [V: -1] WIP: Create new contact with just an email from opt-in. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/505686 (https://phabricator.wikimedia.org/T217710) (owner: Cstone) [16:24:48] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, and 3 others: Update opt-in queue consumer to be able to create new donor records - https://phabricator.wikimedia.org/T217710 (mepps) Weird @cstone look... [16:29:53] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint H 2019, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CRM contribution tracking writing should have a switch to use the new infrastructure - https://phabricator.wikimedia.org/T215463 (XenoRyet... [18:12:52] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Patch-For-Review: Create variants for fancy new opt in pages on payments wiki - https://phabricator.wikimedia.org/T218240 (spatton) Thanks for those details... [19:05:41] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog: Payments wiki form variant with employer field - https://phabricator.wikimedia.org/T219558 (spatton) Thanks all, just sharing some additional notes gleaned from related emails/IRC: 1. @jgleeson confirmed t... [19:10:05] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (LeanneS) Okay so I was able to import the whole list in segments. @Eileenmcnaughton How... [19:58:21] Fundraising-Backlog: iDEAL opt-ins not recording - https://phabricator.wikimedia.org/T221803 (CCogdill_WMF) p:Triage→Unbreak! [20:00:28] oh shoot ^^^ [20:00:56] Maybe we need to make the audit processor look for opt_in ? [20:01:04] fr-tech who wants to write a patch? [20:02:07] looks like it might just be one line! [20:02:18] To fix an 'unbreak now', that's pretty good [20:04:24] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: once-off import of planned giving list - https://phabricator.wikimedia.org/T221714 (LeanneS) Confirmed that it looks like lots still need to be matched. The number hasn't... [20:06:55] (PS1) Krinkle: ext.centralNotice.startUp: Convert to packageFiles module [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506232 (https://phabricator.wikimedia.org/T193826) [20:07:36] What's the offending line? [20:09:27] XenoRyet: I think you might just have to add opt_in to the list of filter_fields [20:09:48] in BaseAuditProcessor line 1230-ish [20:15:07] sorry all I should have checked here first :P [20:15:14] thanks ejegg and XenoRyet [20:15:18] hey all [20:15:35] fr-tech do you guys get why the audit fix is necessary for iDEAL opt-in? [20:15:39] (PS1) XenoRyet: Fix for iDEAL opt-ins not recording [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506234 (https://phabricator.wikimedia.org/T221803) [20:16:08] anything I can help with here? [20:16:17] Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Patch-For-Review: iDEAL opt-ins not recording - https://phabricator.wikimedia.org/T221803 (DStrine) [20:16:23] ccogdill I think we've got it [20:16:28] ah okay [20:16:35] thanks!! [20:16:36] well, a fix for the issue going forward, anyway [20:16:48] mmm okay. is it going to be possible to recover the data? [20:17:03] yep, the data should be in the payments-wiki logs [20:17:06] wheeeew [20:17:10] all good :) [20:17:12] in an easily-parsable format [20:17:21] we just need to write a script to gather it [20:18:36] (CR) Krinkle: "You can rebase on I6bb168c26fab717 to try this again for CentralNoticeCookiesToDelete." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [20:18:45] fr-tech does everyone understand why we're looking at the audit part here? [20:19:17] sorry team but I've gotta take over matilda duties. Will be back later to wrap up emails [20:19:20] ejegg i believe that's how we get ideal transactions? [20:19:25] I don't understand 100% of the context surrounding it, but I get why that field needed adding. [20:19:36] sorry to bail during a fix-now situation! [20:19:39] back later [20:19:46] no worries jgleeson [20:19:49] mepps yeah, iDEAL is asynchronous-ish [20:20:12] so we don't get a final messages to the donations queue while the donor is still on payments-wiki [20:20:25] gotcha ejegg [20:20:50] fr-tech so does anyone have an idea for the best way to get the existing donors' opt-in data? [20:21:30] ejegg i'd start with the payments-wiki logs based on what you said to ccogdill ;) [20:21:42] ok, looking for what? [20:24:38] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog: Oanda API Bug - https://phabricator.wikimedia.org/T220859 (DStrine) I just tried entering 2 expenses in the test document and both show this error. I left the data in the spreadsheet for someone to review.... [20:25:11] (Restored) D3r1ck01: CentralNoticeHooks: Bundle configuration vars into JS that needs it [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [20:26:08] hmm [20:27:21] okay ejegg i'm currently looking at payments-initial and seeing some txns marked rtbt_ideal...that seems like a good start? [20:27:57] does payments-init have the opt-in data? [20:28:06] doesn't look like it XenoRyet [20:28:08] poking around more [20:28:27] paymnents-globalcollect maybe? [20:28:57] ooh that looks more promising [20:29:13] it does have the opt_in data XenoRyet [20:30:00] yep, that's the file [20:30:11] (payments-globalcollect) [20:30:22] so what lines in particular? and what do we extract? [20:31:24] ok, so probably something like country nl, payment method rtbt and/or campaign C1819_email1 [20:31:40] so the lines with the messages and where opt_in=1 and i think we'd want the gateway_txn id? [20:32:09] or email though with duplicates that oculd be trickier [20:32:48] ejegg: also, should we deploy that fix, or do we have a while before the process runs again? [20:32:55] also XenoRyet, ejegg should we meet? [20:33:09] We could, unless this takes precidence. [20:33:35] it probably does [20:33:56] I'd be fine doing mid-sprint checkup tomorrow [20:34:18] mepps I was thinking we could sweep through all of the lines with opt_in=1 [20:34:32] and just pull out the email address [20:34:43] then feed them into the opt-in queue [20:34:59] Yea, that probably is simpler actually. [20:35:12] i have this: grep 'ideal' payments-globalcollect | grep "\"opt_in\":\"1\"" [20:35:33] that makes sense ejegg [20:35:57] mepps yep, we could even expand it to more than just ideal [20:36:18] since we would have missed opt-in from anything that came in via audit [20:36:36] yea, and I don't think it hurts to cast a wide net looking for opt-ins. [20:37:28] also, we'd want to grep in more than just today's logs, of course [20:38:06] (PS1) Krinkle: Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) [20:38:39] ooh so this just gets the email grep "\"opt_in\":\"1\"" payments-globalcollect | cut -d "," -f 10 | cut -d ":" -f 2...and of could we can swap out bzgrep for the archives [20:38:59] (PS2) Krinkle: Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) [20:39:40] looks good mepps :) [20:39:40] looks right [20:43:36] ejegg i'm generating the list now as a csv (can be changed of course) but i don't know what to do with it after that to go in the opt-in queue [20:43:55] also i have to sign off in fifteen to get the kids [20:44:22] mepps ok, want to drop the csv in /tmp ? [20:44:30] (CR) jerkins-bot: [V: -1] Move kvStoreMaintenance to 'ext.centralNotice.startUp' [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/506317 (https://phabricator.wikimedia.org/T221805) (owner: Krinkle) [20:44:39] XenoRyet: do you know a good way to get messages into a queue? [20:44:56] Not off the top of my head, no. [20:45:09] A maintenance script in SmashPig, maybe? [20:49:54] You mean like just make a new one in SmashPig/Maintenance? [20:52:27] XenoRyet: no, there's an existing one that should serve [20:53:00] PopulateQueueFromDump? [20:53:02] That one? [20:56:17] (PS5) D3r1ck01: CentralNoticeHooks: Bundle configuration vars into JS that needs it [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) [20:56:56] XenoRyet: yep, exactly! [20:57:25] So can you work out what format the file should be for the opt-in queue? [20:57:42] let me see [21:01:17] (CR) jerkins-bot: [V: -1] CentralNoticeHooks: Bundle configuration vars into JS that needs it [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [21:02:36] (PS6) D3r1ck01: CentralNoticeHooks: Bundle configuration vars into JS that needs it [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) [21:03:39] (CR) Ejegg: [C: +2] Fix for iDEAL opt-ins not recording [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506234 (https://phabricator.wikimedia.org/T221803) (owner: XenoRyet) [21:03:53] looks like that consumer just needs the email [21:08:00] (Merged) jenkins-bot: Fix for iDEAL opt-ins not recording [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/506234 (https://phabricator.wikimedia.org/T221803) (owner: XenoRyet) [21:12:14] (CR) D3r1ck01: "This change is ready for review." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/502772 (https://phabricator.wikimedia.org/T219342) (owner: D3r1ck01) [21:28:29] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog: Oanda API Bug - https://phabricator.wikimedia.org/T220859 (jgleeson) Yeah sorry @DStrine I forgot to copy the new formula down across all rows so that permissions error is due to the custom function I init... [21:28:59] (PS22) Ejegg: c_t ids from the sequence generator [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/496351 (https://phabricator.wikimedia.org/T215462) (owner: XenoRyet) [21:29:32] (CR) Ejegg: [C: +2] "OK, I think this should do the trick! We'll want to do plenty of local testing before we enable this in prod." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/496351 (https://phabricator.wikimedia.org/T215462) (owner: XenoRyet) [21:36:21] (Merged) jenkins-bot: c_t ids from the sequence generator [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/496351 (https://phabricator.wikimedia.org/T215462) (owner: XenoRyet) [21:37:18] (CR) Ejegg: [C: +2] Fix flake8 errors [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/506031 (owner: Ejegg) [21:37:42] (PS2) Ejegg: Added script for generating exchange_rates master doc [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/505803 (owner: Jgleeson) [21:38:00] (Merged) jenkins-bot: Fix flake8 errors [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/506031 (owner: Ejegg) [21:38:19] (CR) jerkins-bot: [V: -1] Added script for generating exchange_rates master doc [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/505803 (owner: Jgleeson) [21:38:55] :\ [21:41:11] I gotta get some food. Back in a bit. [21:43:35] jgleeson: whoops, meant to rebase it on top of that flake8 fix. one more rebase should do it [21:43:48] (PS3) Ejegg: Added script for generating exchange_rates master doc [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/505803 (owner: Jgleeson) [21:45:24] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Patch-For-Review: Create variants for fancy new opt in pages on payments wiki - https://phabricator.wikimedia.org/T218240 (Ejegg) Sure, that should be fine.... [21:50:08] thanks for fixing that ejegg [21:51:54] awww it looks like google is intent on dragging this out a little bit longer [21:52:27] the expense form works, until you clone it [21:53:55] aww man [21:54:16] then the clone shows errors when trying to look up the rate, which doesn't exist on the first. I expected maybe more permission issue, but apparently, it's complaining about the vlookup range, which hasn't changed between the original and the clone... [21:56:55] Fundraising Sprint Greps of Wrath, Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Regular CiviCRM upgrade - https://phabricator.wikimedia.org/T218754 (Eileenmcnaughton) Open→Resolved [21:57:05] ahhh, it is permissions [21:57:29] but that means everytime someone clones the expense form, it with need authenticating!!! [21:57:32] oh man [21:57:45] this is the worst user experience ever dstrine [21:58:03] maybe not as bad as myspace [21:58:06] but it's bad [21:59:24] jgleeson: wait, so that's still the case even if you use that importRange function? [21:59:31] yeah [21:59:39] jeez [21:59:43] although it doesn't tell you [21:59:43] dstrine: I am going to send an email out to people explaining the impact of changing handling for ‘the’ - I feel like it would be good to have an email list for a civicrm user group - ie. I’m thinking people like Danny & Kaitlin that I might not normally include might care about things that are about how to use civi - does it make sense to create a list like that do you think (& if so how do we create WMF lists) [22:00:00] I had to modify the formula just to use importRange in isolation [22:00:09] and then it prompted me to auth [22:00:17] so by default, the issue won't be obvious [22:00:20] :( [22:02:19] ejegg, if you clone https://docs.google.com/spreadsheets/d/1E-9Emnx9wAMUTXl0_LcBwPXmLGwCeiXCP-OU8lETw70/edit#gid=1483315145 do you see errors too? [22:10:55] there's a #REF error in the rates column, yeah [22:11:12] VLOOKUP evaluates to an out-of-bounds range [22:12:11] jgleeson: ^^ [22:12:26] yeah that's what I got [22:12:45] if you extract out just the importRange() call into a single cell, it prompts you to auth [22:12:54] thanks for checking [22:16:01] XenoRyet: so what's next for the ideal opt-in fix? [22:16:46] jgleeson: oh boy, yeah, that's a horrible user experience [22:17:02] Well, looks like the consumer only needs an email, which we've got, but it also looks like that script wants it to be json encoded. I'm a bit hung up on a good way to get the list of emails into that format. [22:17:15] the prompt to allow connecting the sheets only appears when you hover over the #REF! error [22:17:18] :( [22:17:54] XenoRyet: well, since it's just one field, I'd say a batch edit would be enough [22:17:58] sed -e [22:18:09] 's/^/{"email":"/' [22:18:27] -e 's/$/"}' [22:19:46] I figured something of the sort, but I'm not particularly well versed in sed [22:35:46] I mostly just use it as a quick way to do search-and-replace [22:35:58] that's the s/ [22:36:21] and here we're just using the beginning ^ and end $ markers [22:36:31] to say add stuff at the front and back of each line [22:36:41] XenoRyet: did that one work? [22:36:54] XenoRyet, ejegg, pretty good short intro to sed I refer to often http://www.theunixschool.com/p/awk-sed.html [22:36:59] LMK if you want me to review something before running [22:37:40] The list already has quotes so looks like I actually want something like sed -e 's/^/{"email":/' -e 's/$/}/' [22:37:49] but yea, looks like that's gonna work. [22:38:07] great! [22:38:34] So let's start by sending just a few to the queue [22:38:40] and run the consumer [22:38:47] then check if they really got opted in [22:39:09] before we send the rest [22:40:36] So I guess get a file with just a few peeled of the top [22:40:44] Or is there a better way to do just a few? [22:40:44] head -n 10 [22:42:14] jgleeson: wow that's annoying [22:42:49] ok, so how do I invoke that smashpig script? [22:43:14] eileen: a civi email group might be nice for minor updates and conversations about features but if we need to talk about an upgrade or something I'd want to notify as many people as possible [22:43:27] I can look into making the civi email list [22:43:59] dstrine: yeah - I currently use fr-all for upgrades & kinda cherry pick people for features [22:45:18] I think someone like Caitlin C might not care to follow features since she doesn’t use the UI but obviously cares about upgrades (or she might choose to subscribe & ignore ) [22:45:19] XenoRyet: it's just a php script [22:45:23] command-line [22:45:53] should be possible to run it as your own user from any frack server that has SmashPig installed [22:46:10] if you really wanted to, you could write up a process-control yaml [22:46:28] (or edit the one-off) [22:46:54] eh, command line seems fine [22:46:58] but I don't think there's enough valuable logging to warrant that [22:50:14] It would be good to roll this out if anyone gets a chance to peak at it https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501128/ [22:50:14] So what handy server has smashpig then. Does frdev? [22:51:09] fr-tech I saw the patches on the ideal UBN task but was anyone able to recover the opt in data? [22:51:22] from the previous donations [22:51:33] dstrine: I'm sort of midway in that process now. It is recoverable [22:52:11] Basically I have the list of opt-ins, just finishing feeding it to the consumer so it gets where it's supposed to go. [22:52:49] XenoRyet: ok thanks I sometimes loose track of this stuff in backscroll [22:52:56] Yea, no worries [23:11:03] Fundraising Sprint Hansel and grep -l, Fundraising-Backlog, Patch-For-Review: iDEAL opt-ins not recording - https://phabricator.wikimedia.org/T221803 (XenoRyet) a:XenoRyet [23:27:39] XenoRyet: so guess which job processes the opt-ins once you've got them in the queue [23:31:18] (CR) Ejegg: [C: +2] "looks like this works!" [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/505803 (owner: Jgleeson) [23:31:42] actually I'm not quite there yet. Spent some time remembering how to copy files around the cluster, then I wanted to look to make sure the people on the small list weren't already opted-in and realized I never reimported my cert, and now I'm not quite sure how to look up contacts by email in civi [23:31:55] (Merged) jenkins-bot: Added script for generating exchange_rates master doc [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/505803 (owner: Jgleeson) [23:31:59] I've also gotta go pick up the kids in about 15, so I'm not sure I've got enough time left for all of it. [23:32:24] Also that fix should probably be deployed, I haven't done that yet either. [23:32:39] Unless someone else did while I wasn't looking. [23:36:39] XenoRyet: the quicksearch should be the easiest way to get contacts be email [23:36:57] Yea, found that now, just wasn't where I expected it to be. [23:38:09] but yea, looks like I pulled the short list from the archive stuff, and the first 10 are already all opted in. Most of 'em are obviously tests anyway. [23:38:25] ok, then tail -n 10 [23:38:30] I'll pull a new short list from today's data. That should get someone who isn't. [23:40:09] Or actually, yea I only have about five minutes until I have to go get the kids anyway. I'm on dual pickup duty today. [23:40:41] Can the rest of this wait until morning? Or I guess should I maybe try to hand it off to mepps since she comes in earlier? [23:41:02] (CR) Ejegg: [C: -1] "Actually, let's set this upstream and just pass it through here. Paypal distinguishes between user cancellations and backing card expirat" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501133 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [23:43:39] XenoRyet: or i can do it [23:43:58] Ok cool. [23:44:09] I promise I'll learn the rest of the steps in the morning ;-) [23:45:14] What was the script to consume the queue? I see one for the pending queue, but not for opt-ins or a general one. [23:46:04] Well anyway, I'll figure that out tomorrow, gotta run. [23:46:31] XenoRyet|afk: it's the unsubscribe queue consumer, confusingly enough [23:49:08] ejegg: can you check the commit in process control - commit msg explains [23:55:28] eileen: looking [23:56:35] eileen: with the new rg_id, it starts over from cid 1, right? [23:57:00] ejegg: well it starts from the variable - which was already set at around 26.5m [23:57:09] we can change that var [23:57:49] ok, cool. just double-checking rgid and option name [23:59:00] eileen: looks like the startVariableName includes the ruleGroupID [23:59:18] ejegg: yep - that’s why it’s starting back at 26.5 m [23:59:27] because it had run in the past & saved that var [23:59:35] ahh, b/c that rule group had already run, got it! [23:59:43] yep [23:59:47] OK, let's go ahead with it [23:59:50] cool [23:59:50] thanks!