[00:38:12] (PS7) Eileen: Civiproxy preferences shell [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/649729 (https://phabricator.wikimedia.org/T268495) [00:53:41] (CR) jerkins-bot: [V: -1] Civiproxy preferences shell [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/649729 (https://phabricator.wikimedia.org/T268495) (owner: Eileen) [01:05:39] (PS4) Eileen: Move preferred_language [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/676181 (https://phabricator.wikimedia.org/T270679) [01:15:52] (CR) Eileen: "I updated this to use apiv4 & to pre-parse the interpretation of opt_in & to have tests. It's failing now because I incorporated some synt" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/676181 (https://phabricator.wikimedia.org/T270679) (owner: Eileen) [01:17:32] (PS1) Eileen: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677058 [01:20:44] (CR) Eileen: [C: +2] Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677058 (owner: Eileen) [01:35:09] (Merged) jenkins-bot: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677058 (owner: Eileen) [01:52:56] (PS1) Eileen: Add additional check on data before tearDown [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677061 [02:07:11] (PS1) Eileen: WIP - see what fails [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677062 [02:13:15] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/677065 [02:13:35] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/677065 (owner: Eileen) [02:18:06] !log civicrm revision changed from 740e49d868 to eb9379daa3, config revision is 6779e3829a [02:18:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:18:40] Fundraising Sprint Esperantoland, Fundraising Sprint File Systems Stage Show, Fundraising Sprint Git Rebase Jump, Fundraising-Backlog, and 3 others: Civi: Find Matches Using Fishing Net is hanging - https://phabricator.wikimedia.org/T277338 (Eileenmcnaughton) @MBeat33 this should be fixed now [02:20:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Fr-drupal-upgrade-2021: Move wmf_eoy_reports to wmf-thankyou extension - https://phabricator.wikimedia.org/T270683 (Eileenmcnaughton) [02:20:14] Fundraising Sprint Airline Passenger Experience, Fundraising Sprint Git Rebase Jump, Fundraising Sprint Zeitgeistbusters, Fundraising-Backlog, and 3 others: Determine if wmf_reports is used by anyone & possibly remove - https://phabricator.wikimedia.org/T270684 (Eileenmcnaughton) Open→Reso... [02:21:50] (CR) jerkins-bot: [V: -1] WIP - see what fails [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677062 (owner: Eileen) [02:30:49] (PS5) Eileen: Move skipping of wmf-donor fields to deduper [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/676175 (https://phabricator.wikimedia.org/T270679) [02:30:51] (PS3) Eileen: Move handling for do_not_solicit field to extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/676180 (https://phabricator.wikimedia.org/T270679) [02:30:53] (PS5) Eileen: Move preferred_language [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/676181 (https://phabricator.wikimedia.org/T270679) [02:30:55] (PS2) Eileen: Add additional check on data before tearDown [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677061 [02:30:57] (PS1) Eileen: Additional check pre-tearDown for MergeTest [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677068 [02:44:24] (CR) jerkins-bot: [V: -1] Additional check pre-tearDown for MergeTest [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/677068 (owner: Eileen) [02:46:50] eileen fr-tech so looking at this task for consuming email preference queue... any quick pointers to the existing unsubscribe queue consumer that I could follow the pattern on? thanks!!! :) [02:47:53] AndyRussG: so the existing unsubscribe uses drush - but its probably more future proof to create a civicrm api [02:48:59] Fundraising Sprint Esperantoland, Fundraising Sprint File Systems Stage Show, Fundraising Sprint Git Rebase Jump, Fundraising-Backlog, and 3 others: Civi: Find Matches Using Fishing Net is hanging - https://phabricator.wikimedia.org/T277338 (MBeat33) Open→Resolved Thank you @Eileenmcnaugh... [02:49:06] eileen: ahh ok that sounds reasonable [02:49:07] the drush file is unsubscribe_queue_consume_drush.inc - but that is just a wrapper around unsubscribe_batch_process [02:50:22] do you have an existing extension or module in civi-land you are adding to? [02:51:06] eileen: no I have actually no idea what I'm doing hehehe :) [02:51:08] I guess it can go in https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/649729 [02:51:13] :-) [02:51:26] also that file doesn't seem to be anywhere on my computer (unsubscribe_queue_consume_drush.inc) [02:53:51] hmm - that should be in crm/drupal/sites/all/modules/queue2civicrm/unsubscribe [02:55:10] I feel like you should probably give yourself some space to familiarise [02:56:24] So one example of where we have a scheduled job that calls an api is the one that checks if attempts to forget emails at silverpop have stalled [02:57:29] https://www.irccloud.com/pastebin/FmNUdR6P/ [02:58:03] so that means call the v4 api with entity = OmnimailJobProgress and action = check [02:58:16] (default would be v3 api with no version ) [02:59:12] that action is under drupal/sites/default/extensions/org.wikimedia.omnimail/Civi/Api4/Action/OmnimailJobProgress/Check.php [02:59:39] & the entity file that defines the action is one level higher [02:59:59] - if you haven't played around with the api explorer you should [03:00:00] eileen: ok hmm that makes sense... btw I did find the file... just a _ was a . [03:00:32] eileen: I did fiddle with the API explorer yesterday, but in fact it didn't give me what I was hoping for, in terms of getting the country ISO code [03:00:38] but I guess it could also be a version thing [03:00:45] should I use v4 then? [03:01:13] well for v3 it takes a bit of 'knowing' -for v4 once you know the rules (less well documented) it's more consistent [03:01:55] I actually switched the patch over to use v4 api https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/649729/7/drupal/sites/default/civicrm/extensions/wmf-civicrm/api/v3/Civiproxy/Getpreferences.php#37 to grab country [03:03:14] this is the civi docs https://docs.civicrm.org/dev/en/latest/api/ - but I find that there is more to navigate than is obvious [03:03:35] ie it sorta 'lands' not on 'api' as a topic but on the first subpage 'api intro' [03:04:13] but the hamburger at the top left works better [03:04:16] ahh hmmm hey cool thanks for updating the patch! [03:04:35] shouldn't it now be in a different directory (not api/v3)? [03:06:22] Wikimedia-Fundraising-Banners, FR-Italy: Notifications overlay on top of WYDG for logged in users - https://phabricator.wikimedia.org/T279404 (jbolorinos-ctr) [03:09:55] eileen ^ [03:15:57] AndyRussG: ah - I actualy updated it to call v4 internally [03:16:00] it's still a v3 api [03:16:30] there are some limitations around v4 - especially around calling via rest and passing parameters via drush [03:16:42] so probably I'd leave that as v3 for now [03:18:57] hmmm ok interesting..... :) [03:19:13] K gonna try it out! thanks again btw :) [03:32:29] eileen: country code works! however now it's not returning the is_opt_out field [03:32:52] AndyRussG: hmm - so I did something hopefully clever but maybe wrong there.... [03:33:30] it's not passing tests right now because it's adding an inner join that it won't when the same code runs against 5.37 (which we can deploy later this week) [03:40:04] ah hmm ok [03:41:40] eileen: ahh another question, are there legit times when we might get contacts with null country or null preferred language? [03:42:01] yes [03:42:06] hmmm ok [03:42:23] probably not super common but legit [03:42:57] do you know if users offhand are geolocated with that data point available inside our Mediawikis? [03:43:11] (for the main cluster they are, but not sure in e-mail pref ctr then) [03:50:58] I feel like I should put in Smashpig two helpers, one for mapping back and forth language codes, and another for mapping back and forth opt-in status [03:52:16] yeah not too sure [03:52:27] those helpers make sense [03:53:23] K cool [03:53:35] AndyRussG: I guess I don't know what you mean actually [03:53:58] eileen: like we have a bunch of civi-allowed languages [03:54:03] I mean - we do have fields in civicrm_address for lat long but they would really only be populated in the US [03:54:14] civi allowed languages is a mess :-( [03:54:24] ah you mean you don't know what I mean about the geolocation [03:54:40] eileen: rather I mean ip-based geolocation [03:54:48] where we can know what country someone's in based on their IP address [03:54:49] I mean we 'accept' de_US and en_UG for no good reason [03:54:54] IMHO [03:55:05] no we don't have ip based geolocation in civi [03:55:31] but in payments do we? [03:55:36] not sure [03:55:40] okok [03:55:55] so for languages - if you look at select * FROM civicrm_option_value WHERE option_group_id = 86; [03:56:02] you will see the look up options [03:56:44] but they are pretty contaminated with 'not real' options - that contamination is a WMF thing not a generic civi thing & it goes back before I started [03:57:59] ok hmm [04:00:57] eileen: so for now I'm limiting to this list, which is what ejegg said was available for ty e-mails: https://phabricator.wikimedia.org/T268510#6944167 [04:01:28] ok [04:01:46] but when a donor with a different language lands on the e-pref-ctr I guess some mapping will have to occur [04:02:19] yeah - I mean that list is narrow enough to hand-map [04:03:06] hmmm [04:03:52] eileen: so here's a question.... how are we currently mapping it for e-mail sends? i.e., if we have contacts with any number of odd preferred languages, but we're only sending in a small subset of that, there must already be lang mapping happening somewhere? [04:46:59] Yeah I think we have some cludgey code [04:47:37] but basically they have to be recognisably close to one of the translations or they get English [04:47:52] right that sounds reasonable [04:48:25] eileen: I feel like I should at least look at that code... got a quick pointer (if not it's ok, I think it's already getting late 4 u :) ) [04:49:45] search for wmf_common_locale_civi_to_mediawiki [04:50:39] looks like we are just using 2 characters [04:51:50] ahhh cool this is great thanks! yeah it's in drupal/sites/all/modules/wmf_common/wmf_common.module [04:55:44] https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/crm/+/fff771fe020a25722647a01d9f928af00e714187/drupal/sites/all/modules/wmf_common/wmf_common.module#140 [04:55:57] eileen: huh right so it's just first two characters, end of story? [04:56:27] yeah I guess - I mean I'm not sure that's great but it seems to be the case for now [04:56:47] but if they update ideally they would update to 'the real value' [05:00:45] eileen: yeah I think we want something fancier [05:02:19] hmmm now I feel weird putting it in SmashPig because there's no other such specific unusual-helper-class config like what we'd need anywhere else in the SP config [06:06:17] the first place isn't the final place :-) [08:55:50] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Cleanup our language mess - https://phabricator.wikimedia.org/T279389 (Peachey88) [14:53:14] Fundraising Sprint Bee Wheel, Fundraising Sprint Downed power line jump rope, Fundraising Sprint Esperantoland, Fundraising Sprint File Systems Stage Show, and 5 others: Use Guzzle to make API request to CiviProxy to retrieve opt-in / opt-out fields. - https://phabricator.wikimedia.org/T268497 (Ej... [14:53:19] Fundraising Sprint File Systems Stage Show, Fundraising Sprint Git Rebase Jump, Fundraising-Backlog, FR-AutoTY-Email, Patch-For-Review: Production of New Annual Fund Thank You Email (due to ED transition) - https://phabricator.wikimedia.org/T278363 (CDenes_WMF) Quick Update: Japanese, Hebrew,... [16:18:38] (PS1) Ejegg: Add config defaults for email-prefs queue [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/677302 (https://phabricator.wikimedia.org/T268510) [16:32:09] Fundraising-Backlog, MediaWiki-extensions-Other, Quality-and-Test-Engineering-Team (QTE), Documentation: [FundraisingTranslateWorkflow] Create https://www.mediawiki.org/wiki/Extension:FundraisingTranslateWorkflow - https://phabricator.wikimedia.org/T167996 (thcipriani) [16:53:39] (PS1) AndyRussG: Add settings for email-pref-ctr connection to civiproxy [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/677308 (https://phabricator.wikimedia.org/T268497) [17:22:47] jgleeson hey (for when you're back from your errand, no rush) are you ready for some review on your CiviProxy Gerrit changes? (I see them marked as WIP, so I wasn't sure... also I rebased one of them) :) thx! [17:26:43] ahhh almost time again for my biannual, two-week coffee-fast [17:34:26] (PS2) AndyRussG: Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) [17:37:02] (CR) jerkins-bot: [V: -1] Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) (owner: AndyRussG) [17:41:48] (PS3) AndyRussG: Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) [17:47:23] hey fr-tech. just checking in on the civiproxy stuff. Are folks still carrying out the manual steps from the etherpad to work with stuff? if so I'm pick back up on trying to automate those in the build process [17:47:47] jgleeson: hi! yeah I got it working like that, with just one additional tweak [17:48:12] which is to add a ServerAlias line to the apache config for civicrm [17:48:15] (CR) jerkins-bot: [V: -1] Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) (owner: AndyRussG) [17:48:47] jgleeson: I wasn't sure whether to post any further comments, since the patches are marked WIP [17:49:09] didn't we get rid of the need for the server alias last week? [17:49:26] jgleeson it didn't work on my setup since I also have civicrm's dmaster installed [17:49:27] is it the same one as item 2 in the etherpad, https://etherpad.wikimedia.org/p/adfadf-civiproxy [17:49:51] oh strange. I thought I had that installed also [17:50:23] jgleeson so the alias directive in docker-compose.yml works great! [17:50:36] this is in the 000-default apache config [17:50:56] basically, locally, calls to civiproxy from email-pref-ctr were going to dmaster [17:51:12] and adding that line fixed it [17:52:59] jgleeson: so this is the line I added: ServerAlias wmff.civicrm [17:53:17] in config/amp/apache.d/wmff.localhost_32353.conf [17:53:27] (which is generated, so we'd have to see how to generate it with that line) [17:53:50] I don't see any harm in adding a dmaster.civicrm line to the corresponding dmaster config file [17:53:58] so that should be easy enough [17:54:12] interesting [17:54:30] heheh yes my reaction exactly when I figured it out ;) [17:54:44] I know I had to add an alias to that file for the old way [17:55:03] I wonder if the wmff.localhost name wasn't working afterall [17:55:07] I think it has to be both places [17:55:35] I'm guessing that it has to do with if you have another site set up, and where Apache would then send things by default? [17:55:38] I think my local setup is still broken after the hacking to test out the site key install [17:56:22] ahhh right heh that can happen [17:56:46] jgleeson also to note, src/civi-sites/dmaster.sh programatically sets the CIVI_SITE_KEY environment variable [17:57:10] so presumably we can do the same in wmff.sh [17:58:48] hmm I wonder why it didn't work when I tried it [17:59:17] I tried out CIVI_SITE_KEY=TESTYMCTESTFACE during the install last wednesday [17:59:47] be good if that's all we need [17:59:51] ahh heh right... mmm no idea! we can also probably wait 'till eileen is on to get her feedback on that one, I imagine she'd know right away [17:59:57] heh hopefully [18:00:24] the key that is set in that .sh file is indeed the one that dmaster gets inside the site (confirmed via debugging) [18:00:50] yeah I just checked out civicrm.settings.php and it's there [18:04:53] so it looks like there is also a site_key set in src/civi-sites/wmff.sh [18:07:13] huh, I wonder if our env var was being overwritten by this one [18:07:32] so it might have worked afterall [18:09:40] Fundraising Sprint Git Rebase Jump, Fundraising-Backlog, FR-WMF-Audit: Paypal audit downloads failing - https://phabricator.wikimedia.org/T278878 (Ejegg) It's listing the files fine now, but when it tries to download anything over a few KB it spins for 10 minutes and dies with an IdleTimeoutException... [18:24:34] Fundraising Sprint Esperantoland, Fundraising Sprint File Systems Stage Show, Fundraising Sprint Git Rebase Jump, Fundraising-Backlog, and 2 others: Update Fundraising tech CI image to use upstream buildkit, no symlink for civicrm - https://phabricator.wikimedia.org/T277500 (jgleeson) @Eileenmcna... [19:07:34] (CR) Cstone: Email Preference Center: Guzzle connetor to Civiproxy (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) (owner: AndyRussG) [19:17:26] (PS7) AndyRussG: Initial general e-mail preferences prototype [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/672362 (https://phabricator.wikimedia.org/T268510) [19:20:37] (CR) jerkins-bot: [V: -1] Initial general e-mail preferences prototype [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/672362 (https://phabricator.wikimedia.org/T268510) (owner: AndyRussG) [19:22:11] (PS4) AndyRussG: Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) [19:24:17] (CR) jerkins-bot: [V: -1] Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) (owner: AndyRussG) [19:38:32] Fundraising Sprint Git Rebase Jump, Fundraising-Backlog, FR-Adyen: Break down tasks for Adyen reintegration, drop in web - https://phabricator.wikimedia.org/T277121 (jgleeson) Some related info on this here T271076 [20:16:56] Wikimedia-Fundraising-Banners, FR-Italy: Notifications overlay on top of WYDG for logged in users - https://phabricator.wikimedia.org/T279404 (HNordeenWMF) Thanks for surfacing this John! Since this only affects logged-in users, and we don't show banners to logged-in users: not a blocker. [20:33:09] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Forget me failing - https://phabricator.wikimedia.org/T271187 (DStrine) [20:36:24] (PS8) AndyRussG: Initial general e-mail preferences prototype [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/672362 (https://phabricator.wikimedia.org/T268510) [21:33:46] (PS5) AndyRussG: Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) [21:34:27] (PS9) AndyRussG: Initial general e-mail preferences prototype [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/672362 (https://phabricator.wikimedia.org/T268510) [21:37:59] (CR) jerkins-bot: [V: -1] Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) (owner: AndyRussG) [21:54:17] (PS6) AndyRussG: Email Preference Center: Guzzle connetor to Civiproxy [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/677018 (https://phabricator.wikimedia.org/T268497) [21:54:59] (PS2) AndyRussG: Add settings for email-pref-ctr connection to civiproxy [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/677308 (https://phabricator.wikimedia.org/T268497) [23:43:22] AndyRussG: is it new that the docker db image is listening on the host port 3306? I feel like i'd left that as default before and it wasn't conflicting with my host machine db but now it is [23:44:53] anyway, 3307 seems to work