[00:00:06] hmm didn’t hear from ccogdill after email yesterday - just tried pinging on hangouts [02:01:34] (PS4) Eileen: Add showme for logging [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444482 (https://phabricator.wikimedia.org/T196644) [02:01:35] (PS3) Eileen: Add forgetme for logging [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444484 (https://phabricator.wikimedia.org/T196644) [02:01:37] (PS4) Eileen: Add code to delete select contact fields & custom fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444487 (https://phabricator.wikimedia.org/T196644) [02:01:41] (PS3) Eileen: Forget activities [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445331 (https://phabricator.wikimedia.org/T196644) [02:01:42] (PS3) Eileen: Forget activity logs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445335 (https://phabricator.wikimedia.org/T196644) [02:01:44] (PS4) Eileen: Forget relationships [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445342 (https://phabricator.wikimedia.org/T196644) [02:01:46] (PS2) Eileen: Forget mailing provider data [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445344 (https://phabricator.wikimedia.org/T196644) [02:14:27] (CR) jerkins-bot: [V: -1] Forget mailing provider data [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445344 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [03:44:34] (PS1) Eileen: Add apis to retrieve information about the ultimate destination contact and source contacts for merged contacts. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/446519 (https://phabricator.wikimedia.org/T199748) [07:55:48] (PS5) Eileen: Add code to delete select contact fields & custom fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444487 (https://phabricator.wikimedia.org/T196644) [07:55:50] (PS4) Eileen: Forget activities [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445331 (https://phabricator.wikimedia.org/T196644) [07:55:52] (PS4) Eileen: Forget activity logs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445335 (https://phabricator.wikimedia.org/T196644) [07:55:54] (PS5) Eileen: Forget relationships [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445342 (https://phabricator.wikimedia.org/T196644) [07:55:56] (PS3) Eileen: Forget mailing provider data [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445344 (https://phabricator.wikimedia.org/T196644) [07:55:58] (PS1) Eileen: Forget previous lives [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446536 (https://phabricator.wikimedia.org/T199748) [08:00:47] (CR) Florianschmidtwelzow: [C: 2] Replace Linker::link() with LinkRenderer [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/336010 (https://phabricator.wikimedia.org/T149346) (owner: Matěj Suchánek) [08:01:51] (CR) jerkins-bot: [V: -1] Add code to delete select contact fields & custom fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444487 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [08:02:36] (CR) jerkins-bot: [V: -1] Forget activity logs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445335 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [08:02:41] (CR) jerkins-bot: [V: -1] Forget relationships [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445342 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [08:03:38] (CR) jerkins-bot: [V: -1] Forget activities [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445331 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [08:07:15] (CR) jerkins-bot: [V: -1] Forget mailing provider data [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445344 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [08:08:25] (CR) jerkins-bot: [V: -1] Forget previous lives [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446536 (https://phabricator.wikimedia.org/T199748) (owner: Eileen) [08:16:04] (PS6) Eileen: Add code to delete select contact fields & custom fields [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444487 (https://phabricator.wikimedia.org/T196644) [08:16:22] (Merged) jenkins-bot: Replace Linker::link() with LinkRenderer [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/336010 (https://phabricator.wikimedia.org/T149346) (owner: Matěj Suchánek) [08:17:42] (PS5) Eileen: Forget activities [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445331 (https://phabricator.wikimedia.org/T196644) [08:17:44] (PS5) Eileen: Forget activity logs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445335 (https://phabricator.wikimedia.org/T196644) [08:17:46] (PS6) Eileen: Forget relationships [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445342 (https://phabricator.wikimedia.org/T196644) [08:17:48] (PS4) Eileen: Forget mailing provider data [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/445344 (https://phabricator.wikimedia.org/T196644) [08:17:50] (PS2) Eileen: Forget previous lives [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446536 (https://phabricator.wikimedia.org/T199748) [08:30:01] (CR) jerkins-bot: [V: -1] Forget previous lives [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/446536 (https://phabricator.wikimedia.org/T199748) (owner: Eileen) [15:11:02] (PS3) Mepps: Trying to simplify logic [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446187 [15:13:08] (PS6) Mepps: Change documentation variables and test to outline where changes are needed [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/437357 (https://phabricator.wikimedia.org/T193663) [15:20:42] (PS3) Ejegg: Delete obsolete parts of this extension. [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/446492 [15:20:50] fr-tech ^^^ [15:21:02] that extension was like 80% cruft [15:21:32] it included a bunch of settings pointing to foundationwiki, so Reedy had another patch updating some URLs [15:22:00] but those URLs were only used to send donors directly from fonudationwiki or donatewiki to the old Paypal integration [15:22:19] and included IPN listener addresses that we haven't used for at least 4 years [15:23:13] hmmm might be worth a chat to discuss how to test this, if it's needed before +2'ing, as I'm not familiar with testing this locally [15:24:37] also ejegg, I've been looking into the work involved in queue2civi to persist out opt_in flag. Did you say we're gonna re-purpose/rename an existing field or create a new? [15:25:17] jgleeson: it should (a) not break DonationInterface's use of contributionTracking (you should be able to save rows in that table and (b) show Special:FundraiserMaintenance if $wgContributionTrackingFundraiserMaintenance is set to true [15:25:31] ack, unclosed paren [15:25:40] * ejegg core dumps [15:25:47] loll [15:25:59] jgleeson: let's add a new field there [15:26:16] the old optout stuff might still be used in some imports [15:26:24] we can do a cleanup task another day [15:26:43] fair enough [15:26:46] Lemme add a little bit to the ticket [15:27:57] am I right in guessing that we add schema updates to the drupal module install file? [15:28:22] yep! [15:30:09] and it looks like we have a civi api we can use for updating our civicrm db [15:33:13] jgleeson: ok, just put a comment on the ticket outlining the Civi work. [15:33:15] for creating civi custom fields [15:33:21] ok cool will check it out [15:33:21] yep! [15:33:51] ok well if you haven't already started that work, I can carry on with it and try to get something working [15:35:30] sure, jgleeson, go right ahead [15:35:48] awesome! [15:36:20] mepps: do you have any suggestions for data_type and html_type to represent a trinary (yes/no/don't know or 0/1/null) field in Civi? [15:36:59] ideally without resorting to another join on civicrm_option_value [15:39:31] hmm this feels like it should exist, definitely radio for html [15:40:26] couldn't that be a boolean ejegg? [15:42:00] mepps nullable boolean is what we're looking for [15:42:15] we want to record a value if we presented the opt-in box [15:42:20] and leave it null if they never saw it [15:43:21] my fear about using civicrm_option_value is that it would make the silverpop export that much slower [15:44:08] ejegg i think booleans are nullable in civi [15:44:47] for example if you look at the communication fields [15:45:01] they are all radio yes/no but you can also click an X to null the value entirely [15:45:26] mepps Oh, nice! [15:45:41] I was worried that re-saving would always force either true or false [15:45:53] but if there's UI support for nulling the field, that's great [15:46:33] * ejegg shoulda just looked at the UI [15:47:23] jgleeson: ^^^ so that Civi stuff might be really straightforward! [15:48:40] great, just reading about custom fields now in civi as I was curious what type it is using at the db level [15:49:14] right on [15:49:33] guessing it's using some int-based type [15:49:42] or maybe an enum? [15:49:58] for the importer update, I see an array of custom fields in wmf_civicrm.module on line 1237 that might be all we need handle the mapping [15:50:00] actually, I can check out existing table fields I guess [15:50:09] jgleeson: yeah, I think it's a tinyint [16:05:26] fr-tech one sec! [16:17:03] (CR) Reedy: [C: 1] "DELETE ALL THE THINGS" [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/446492 (owner: Ejegg) [17:11:43] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: restore Merge option to Actions dropdown for Search results - https://phabricator.wikimedia.org/T199929 (MBeat33) [17:13:01] fr-tech anything you want me to bring up for scrum of scrums? [17:13:42] XenoRyet: lmk if you have time for some code review [17:13:58] there are some things with a +1 that still need someone else to confirm them [17:14:09] ejegg: nothing here, thanks! [17:17:09] ejegg: Point me at 'em and I'll take a look when I can. [17:18:51] ejegg: I guess just if it comes up, the core bug fix has been deployed, and everything w/ banner messages is back to its (slow) normal [17:19:00] Thanks XenoRyet. Would be great to see these smashpig things reviewed: https://gerrit.wikimedia.org/r/#/projects/wikimedia/fundraising/SmashPig,dashboards/default [17:19:02] so thx again to the core team on that! [17:19:13] AndyRussG|ish: rockin, will pass along your thx [17:19:17] 10-4 [17:19:26] thx4thx [17:19:32] yeah that banner slow-to-save isn't not a pain [17:20:11] I'll also update the ticket in a bit [17:32:49] Fundraising Sprint Karma chameleons hide amongst us, Fundraising Sprint Lactose is unusually tolerant, Fundraising Sprint Matt Damon to head up Space Force, Fundraising Sprint Naming Sprints Is Not Important, and 4 others: Update Ingenico WX audit parser to... - https://phabricator.wikimedia.org/T195337 [17:33:03] Fundraising Sprint Karma chameleons hide amongst us, Fundraising Sprint Lactose is unusually tolerant, Fundraising Sprint Matt Damon to head up Space Force, Fundraising Sprint Naming Sprints Is Not Important, and 4 others: Update Ingenico WX audit parser to... - https://phabricator.wikimedia.org/T195337 [17:46:56] (CR) XenoRyet: [C: 2] "Yep, it's a step forward." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/441250 (https://phabricator.wikimedia.org/T195488) (owner: Ejegg) [17:47:32] (Merged) jenkins-bot: JobQueueConsumer can hydrate non-JsonSerializableObjects [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/441250 (https://phabricator.wikimedia.org/T195488) (owner: Ejegg) [17:53:13] (CR) XenoRyet: [C: 2] "Agreed that the new way seems simpler, and preferable for that reason." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/441583 (owner: Ejegg) [17:53:49] (Merged) jenkins-bot: Rewrite DeletePendingJob as Runnable [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/441583 (owner: Ejegg) [18:09:20] (CR) XenoRyet: [C: 2] "Looks solid, and it worked for me as well, following Jack's steps." [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/441251 (https://phabricator.wikimedia.org/T195488) (owner: Ejegg) [18:10:20] (Merged) jenkins-bot: Job to add tokens to Ingenico payments [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/441251 (https://phabricator.wikimedia.org/T195488) (owner: Ejegg) [18:14:40] (CR) XenoRyet: [C: 2] Make sure to send right timestamp format to antifraud [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/445447 (https://phabricator.wikimedia.org/T199468) (owner: Ejegg) [18:15:11] (Merged) jenkins-bot: Make sure to send right timestamp format to antifraud [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/445447 (https://phabricator.wikimedia.org/T199468) (owner: Ejegg) [18:17:36] (CR) XenoRyet: [C: 2] Make sure we're adding numbers in AdyenAudit [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/439634 (owner: Ejegg) [18:18:11] (Merged) jenkins-bot: Make sure we're adding numbers in AdyenAudit [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/439634 (owner: Ejegg) [18:24:29] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446636 [18:24:37] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446636 (owner: Ejegg) [18:24:58] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446636 (owner: Ejegg) [18:42:03] fundraising-tech-ops, Operations, netops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (herron) p:Triage>Normal [19:32:39] fr-tech if anybody does want to talk about random tech stuff, you can find me in the meet room for today's tech-talk-revival [19:33:08] maybe we can work through the failed recurring or something [20:27:28] (CR) Raimond Spekking: "@Ejegg: Removal of the qqq happens again and again. There must be an error in the en.json with this message. The scripts of translatewiki." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446657 (owner: L10n-bot) [20:38:30] (CR) Raimond Spekking: "And the message itself is unknown to translatewiki.net" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446657 (owner: L10n-bot) [20:39:23] (CR) Ejegg: "Hi Raymond, the message in en.json is right here: https://phabricator.wikimedia.org/diffusion/EDOI/browse/master/gateway_common/i18n/inter" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446657 (owner: L10n-bot) [20:42:45] (PS4) Mepps: Trying to simplify logic [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446187 [20:43:16] made a teensy bit of progress fr-tech ^^ [20:45:52] (CR) jerkins-bot: [V: -1] Trying to simplify logic [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446187 (owner: Mepps) [20:48:08] (PS1) Ejegg: Update libraries [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/446691 [20:48:23] (CR) jerkins-bot: [V: -1] Update libraries [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/446691 (owner: Ejegg) [20:52:35] (PS2) Ejegg: Update libraries [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/446691 [21:28:46] (CR) Ejegg: [C: 2] "Code looks good, and it works! Would be nice to have the specific logging table names or entities visible somewhere. Maybe the first colum" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444482 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [21:32:21] (CR) Ejegg: [C: 2] "Oh hey, you added table names in the next patch. Nice!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444482 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [21:37:30] (Merged) jenkins-bot: Add showme for logging [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444482 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [21:52:10] (CR) Ejegg: [C: 2] "Looks good, and works well... but only because 'Logging' is the last on the list of array_merge($entitiesToDelete, $forgets). Can we disab" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444484 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [21:56:43] I see you movedd tech talk - I didn’t get up for it- I think it makes sense to have them either in Jacks tz or mine not at a time that is on the margin because it doesn’t feel to me like I can focus on it first thing & I expect Jack is the same on last thing [21:57:27] (Merged) jenkins-bot: Add forgetme for logging [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/444484 (https://phabricator.wikimedia.org/T196644) (owner: Eileen) [22:01:07] eileen: hi! I' was thinking about maybe right after standup tomorrow [22:01:09] happiness is when you kids can operation and want to operate the expresso machine [22:01:33] ejegg: ok - tomorrow I don’t normally join standup [22:01:39] hehe, nice! [22:01:45] m feeling is maybe do 2 times fortnight about [22:01:48] OK, I guess tomorrow's would be for Jack then [22:01:58] & have one that is in Jack’s normal work day & one in mine [22:02:43] I don’t mind early meetings but for a tech talk pre- coffee I doubt I would really fully engage my brain [22:03:47] ccogdill: are you around [22:05:36] (PS1) Brian Wolff: Add an extra message to allow users to add custom navbar [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/446698 (https://phabricator.wikimedia.org/T138284) [22:07:31] (CR) Ejegg: [C: 2] Update libraries [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/446691 (owner: Ejegg) [22:08:05] (Merged) jenkins-bot: Update libraries [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/446691 (owner: Ejegg) [22:08:43] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446699 [22:08:54] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446699 (owner: Ejegg) [22:09:22] ccogdill: ? [22:10:21] hey eileen! [22:10:36] I know I owe you GDPR info. sorry, I've been tracking down the documentation [22:10:54] (PS1) Ejegg: Update libraries [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/446700 [22:11:14] I have a few things I'm looking through now and checking to make sure it matches up with what you found [22:11:14] (CR) Ejegg: [C: 2] Update libraries [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/446700 (owner: Ejegg) [22:11:24] (CR) Ejegg: [V: 2 C: 2] Update libraries [wikimedia/fundraising/SmashPig/vendor] - https://gerrit.wikimedia.org/r/446700 (owner: Ejegg) [22:13:03] yeah, that's the most recent info I have from IBM re: an API, eileen [22:13:25] I don't have the documentation on this one, but what I can use manually is a button in the UI [22:13:33] (PS2) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446699 [22:13:42] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446699 (owner: Ejegg) [22:13:43] and the button is a delete one on a per-contact basis? [22:13:58] nope I can create lists as new databases and mass purge [22:14:09] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - https://gerrit.wikimedia.org/r/446699 (owner: Ejegg) [22:14:10] so that's what I figured we would do with a potential checkbox [22:14:30] is feed those people as "opt-ins" to the GDPR purge list [22:14:38] ccogdill: ah ok - so th approach of creating a list makes sense [22:14:49] & that would be some field we pass up to GDPR [22:14:57] sorry to silverpop? [22:15:09] ie it looks like [22:15:19] 1) you create a field in silverpop [22:15:39] 2) we update our export or in some other way populate that field based on DS deletions [22:16:02] 3) you create a query off that field [22:16:31] 4) we configure a regular purge of that list (that scares me as an automated step ….) [22:16:46] !log updated SmashPig payments listener from 2b4186715e to 2736e41045 [22:16:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:18:02] pretty much, except I would ask that we change step 3 to: you set up a specific import for the DS deletions field/table into a GDPR database in silverpop, and we just keep purging that main DB regularly [22:18:38] we have 2 concurrent imports for the main DB and unsubscribes DB right now... so I was hoping we could treat this as the 3rd [22:19:23] ccogdill: so this is a process like the nightly one [22:19:39] ie we import to unsubscribes db nightly? [22:19:39] that would work for me [22:19:55] though I don't know that this job needs to happen nightly [22:20:09] could be weekly or something? I'd want to ask Michael [22:20:15] (I find their use of ‘databases’ terminology confusing - ie. it’s the same contact so it’s the same DB) [22:20:30] re frequency - if it’s a small job we’ll do often [22:20:50] but, the way it works for unsubscribes kinda thing [22:21:14] yep pretty much [22:21:39] I think that would still require someone to come and manually hit the erasure button at some regular interval [22:21:44] in silverpop [22:21:53] but I can think of if there's a way to automate that... [22:22:31] it's so easy though, if it's just regularly importing into a list. I don't think it would be a burdonsome process [22:23:03] ccogdill: ok - I mean there is a possibility to create a purge job [22:23:33] in silverpop? [22:23:39] APi call [22:23:52] (CR) Ejegg: "@Raimond Spekking the en.json and qqq.json messages were added in this patch: https://gerrit.wikimedia.org/r/445451" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/446657 (owner: L10n-bot) [22:23:54] oh right :P [22:23:56] so from your end the first steps are 1) you create a new custom field e.g privacy_requested [22:24:04] & then I can try to populate that [22:24:43] okay so not going the separate list route [22:24:47] is that right? [22:24:58] ccogdill: isnt the list based off a new field? [22:26:07] ah I was thinking I could get one "database" to delete the data from another but I guess it wouldn't work like that [22:26:18] so yeah it does have to be a new field. I was thinking I could simplify it [22:29:09] ccogdill: I don’t fully get the database concept in silverpop [22:29:41] I think it's just their way of saying a master list [22:29:48] it's about folder hierarchy more than anyway [22:33:32] OK - do you want to go ahead & create a field? (this is a generic privacy project not associated with a specific legislation so use a generic name) [22:34:01] & perhaps a query based off that field too now [22:34:51] I'm testing something out right now [22:38:56] huh okay sorry, I just needed to try that. the tool in the UI isn't quite what I thought it was. requires csv upload currently [22:39:01] I can create the field [22:40:51] cool [22:41:13] I'm guessing this is a text field? [22:42:03] also FYI there is a Right of Access API as well https://developer.ibm.com/customer-engagement/tutorials/performing-gdpr-right-access-wca-apis/ [22:42:40] ccogdill: is there a checkbox field? [22:42:46] not yet [22:43:12] I need to talk to ejegg about what the checkbox is doing to the database [22:43:34] I don't want Right to Erasure people ending up on the unsubscribes list... [22:43:46] or checkbox people [22:44:11] hi ccogdill! [22:44:18] * ccogdill waves [22:44:34] So, for CiviCRM, we're going to add (yet another) custom field [22:44:53] it'll be null by default, and only contain Yes or No for people who have been shown the checkbox [22:45:19] okay cool. it won't do anything else to their communication preferences? [22:45:24] i.e. on hold, opt out, etc [22:45:27] Right [22:45:35] [:) [22:45:43] cool cool [22:45:54] ccogdill: err, we're talking about the opt-in checkbox, right? :) [22:46:11] (I’m expecting no new field in Civi because we’ll use activity info but a new field in silverpop ‘privacy_requested' [22:46:34] ohhhh [22:46:36] ah - that’s another again [22:46:44] yes! I was afraid of these people getting opted out or something which would permanently opt them out in silverpop, even if they checked the box again [22:46:46] no this is different [22:46:47] you mean a field for ‘op in’ [22:46:56] the checkbox and Right to Erasure people are different [22:47:06] so we have [22:47:12] okay sorry, I am talking to ejegg about one thing and eileen about another thing. to be clear :D [22:47:18] 1) checkbox for erasure / privacy & [22:47:26] 2) checkbox for optin [22:47:51] are you talking about checkboxes in civi? [22:47:59] #1 should be in civi for DS to use [22:48:18] #2 should be on payments/banners/etc -- for the user to use [22:48:24] (not in civi) [22:49:10] ccogdill: so in civi we create an activity when people request privacy - so the presence of that activity is effectively our checkbox [22:49:34] but in spop I’m expecting an actual field for it [22:49:56] okay, the opt-in checkbox is just different from "requesting privacy" IMO. I want to make sure we are distinguishing the intent between erasure and not opting in to future email [22:50:35] cool [22:50:53] so requesting privacy gets its own field [22:51:35] opt-in checkbox gets its own field. I think that works. I think we will have to build a mandatory rule in all our silverpop queries to remove people with Y in the opt-in field, and do regular purge of our database to clear them out... [22:51:55] (our database meaning the Big List in silverpop) [22:53:02] ccogdill: OK, so just exporting the opt-in value as a new column in the silverpop csv will work for you? [22:53:38] I'm not sure if there's a better way to handle it... [22:53:42] do you have any other ideas? [22:54:25] I just want to keep them off silverpop's "Master suppression list" which is where the unsubscribes import goes. it means we can never opt them back in (unless we purge them from that suppression lsit) [22:54:28] ccogdill: is this a typo “to remove people with Y in the opt-in field, and do regular purge of our database to clear them out" [22:54:40] sorry I guess I meant N [22:54:42] I was hoping that would be enough! Anything else would make the silverpop export even more complicated than it is... Like doing some kind of custom country logic [22:54:43] :-) [22:55:00] no no I don't think that's necessary [22:55:18] yeah actually this is simple enough [22:55:47] we can just do a big purge before each campaign and that will mostly cover it, and then keep our queries updated [22:56:36] so I vote yes on new column in csv [22:56:45] let me know when it's getting added [22:57:09] eileen, privacy_requested field added [22:57:19] ccogdill: ok cool [22:57:48] does it make sense to create a query off it now - that way I can find out what I can ‘see’ from the api [22:58:31] so I'm not sure query is actually part of this... [22:58:53] or I don't see that in the documentation [22:59:22] is it just me running a query of anyone with "Y" in that privacy_requested field? [23:00:51] I'm setting privacy_requested as a "GDPR lookup field" which is what is supposed to distinguish it, right? [23:02:25] ccogdill: well from what I can tell we can “regularly purge a pre-saved query in Silverpop" [23:02:42] & that seems to mean that we could purge the contacts that were in a query set [23:03:54] are you quoting me... ? :) [23:05:01] I don't know that I totally understand the process but I will just go ahead and create the quey [23:05:03] *query [23:06:49] ccogdill: ok cool [23:07:15] ccogdill: is Brian still the silverpop man [23:07:28] for dev stuff, yeah [23:07:32] the thing is - the api docs say I can purge by query [23:07:45] and it’s easy code to write [23:07:47] is this "purging" or GDPR erasure? [23:07:59] by the docs ‘purging' [23:08:08] I think those might mean different things [23:08:13] in terms of what IBM does with the data [23:08:20] I’ll find the ref [23:08:44] because "purge" is also an action you can perform on a database [23:08:50] or list [23:09:51] so - page 18 on the pdf on https://phabricator.wikimedia.org/T199747 [23:09:52] purging does remove a contact from your list, but I wonder if IBM actually clears records from its own systems when you perform the GDPR lookups [23:10:05] I'm having a phab access problem right now :( [23:10:26] as of this morning [23:10:43] is this IBM documentation? can you send me the title? I should be able to find it. [23:11:28] and when I said "I wonder" I meant "I understand that...". I can confirm this with Brian unless you're talking to him already. [23:11:44] ccogdill: I just copied the section to you [23:12:13] this is from the api xml doc - page 18 [23:12:36] I’m starting to think you are right though “Create a new database or list containing copies of the purged records. " [23:13:35] ccogdill: I can send an email to Brian to confirm [23:13:56] so re: Michael's question in phab (I got the email so I can see it) [23:14:00] "I'm not sure if this is in the works, or not doable, but if the Forget Me button could send a deletion request to Silverpop when pushed, that would be ideal from a workflow perspective for DS & email teams." [23:14:01] from what I can tell the GDPR feature takes an email or phone not a contact ID [23:14:25] would it be possible to simply ping silverpop's api with the contact ID or email of a person whenever DS hits that button? [23:14:39] so we can’t delete the contact and then tell Spop to erase them - because we will have already deleted the email [23:14:59] are we keeping the contactid around? [23:14:59] we could possibly send spop a gdpr request BEFORE we delete them [23:15:01] can we use that? [23:15:17] currently we have email set as a GDPR lookup key, but we could make contactid one [23:15:37] ah [23:15:38] https://developer.ibm.com/customer-engagement/tutorials/performing-gdpr-right-erasure-wca-apis/ [23:16:05] yeah! [23:16:10] wait you saw that before, right? [23:16:10] yes - if contact id was a gdpr lookup key that makes sense [23:16:17] I thought that was the documentation we were looking at [23:16:25] there is the xml doc & the rest doc [23:16:43] we actually use the xml stuff so far - so there is possibly some new config to also use rest [23:16:54] but I can dig into that [23:17:08] cool. so I just need to make contactid a GDPR lookup key and that's it, right? [23:17:26] yeah - that was the missing piece for me [23:17:41] I was focussed on the examples all expecting email or phone [23:18:01] ah [23:18:08] yeah, we can pick any field we like [23:19:26] ccogdill: ok - if you do that I’ll try to figure how to use it! [23:21:18] okay annoyingly enough I'm having trouble setting up a new key [23:21:21] don't know why [23:21:23] emailed Brian [23:21:26] :D [23:21:27] sigh [23:21:53] you can play with email now if that's helpful [23:26:21] ccogdill: ok - well I’ll play with the rest inteface for now [23:26:28] because we haven’t used it yet [23:28:02] cool [23:28:04] phew! [23:30:00] ccogdill: from memory I have to configure a new user or permission in spop to make it work & I wasn’t sure last time if it would impact on our existing auth…. but will di [23:30:02] dig [23:30:11] & refresh my memory [23:30:23] okay