[00:38:46] (PS10) Eileen: Upgrade CiviCRM to 4.7.21 [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/359898 [14:31:03] PROBLEM - Host benefactorevents.wikimedia.org.mgmt.eqiad.wmnet is DOWN: PING CRITICAL - Packet loss = 100% [14:35:22] hmmm [14:36:12] cwd: that's a red herring [14:36:15] I imagine see -ops [14:37:38] chasemp: thanks! [15:47:01] hi cwd, AndyRussG, and mepps! [15:47:13] hi ejegg! [15:47:21] How's it going? [15:48:47] howdy [15:50:17] cwd so, mepps merged the patch to have SmashPig read the location of the provider configs from /etc/smashpig/main.yaml [15:51:01] I can prep what main.yaml is supposed to look like on the deploy host, and get the provider configs ready [15:57:17] ok, i can look into getting that deployed [16:37:21] ejegg: would you be able to call into the fr-online standup in ~20 minutes? I want to figure out timing on the civi upgrade [16:37:55] dstrine: sure! [16:38:00] thanks! [16:52:18] ejegg: hi :) [16:53:00] Buenos diás [16:53:07] o tardes... [17:00:32] fr-tech: You never know how many friends you have until you rent a house on the [17:00:32] beach. [17:00:32] -- discuss. [18:02:00] cwd I think the main.yaml file in my homedir on boron should be ready to templatize and deploy to /etc/smashpig/main.yaml [18:02:09] just testing a bunch locally now [18:03:55] ejegg: what is the future of /etc/fundraising? [18:06:23] It's deprecated at least for SmashPig [18:07:01] should this file be replacing /etc/SmashPig.yaml? [18:07:35] cwd yep, but we can't remove that till DonationInterface and CRM are updated with the new version [18:09:37] back again soon... [18:09:40] can we do that at the same time? [18:10:01] i would like to not add any confusion to /etc [18:10:14] it's already a very confusing place [18:15:39] oof, sounds like a crazy amount to deploy at the same time [18:16:11] i'd much rather focus on makong sure listeners and job runners are ok first [18:16:31] all 3 at once would make problems harder to diagnose [18:18:19] i'm not planning to let the others linger long, tho [18:18:40] already have WIP patches for the payments update [18:19:15] and tbe crm update shouldn't be much code, though we'll want to thoroughly test it of course [18:20:29] could the config pointer just be added to SmashPig.yaml? [18:22:07] nah, it's a different structure altogether [18:22:20] dropping the topmost nesting level [18:22:45] hmm but isn't there only one file in /etc/smashpig? [18:22:51] i think you'll like the new main.yaml, though. way simpler! [18:23:38] cwd for now, yes [18:25:11] when will there be more? do they need to be deployable by fr-tech? [18:27:08] it could potentially hold account info if we want to have all the secrets in puppet [18:27:17] in am accounts.yaml file [18:36:09] ah ok, but that dir will be administrated with puppet? [18:41:44] couldn't you add provider-configuration-directory to SmashPig.yaml to make the smallest change possible? [19:03:04] cwd most of SmashPig.yaml is gone now, moved into the provider-specific files [19:05:57] did you see how tidy the new one is? [19:09:22] it wouldn't make much sense to smash that on with all the old stuff [19:12:59] yeah i feel that but is it a compelling enough practical concert to justify creating a potentially confusing situation? [19:13:08] where there are 2 things in /etc called smashpig [19:13:16] *concern [19:13:50] what other stuff is currently destined for /etc/smashpig? [19:18:38] cwd we can delete SmashPig.yaml vey soon [19:18:47] I just want to deploy one thing at a time [19:19:14] oh yeah i'm pushing for a smaller change if anything [19:19:23] just wondering what other stuff will go in /etc/smashpig? [19:19:30] cwd it'll probably be a while before we can move more generic DonationInterface functionality into SmashPig [19:19:54] I'm thinking some of the stuff from localsettings would be nice to have it its own file [19:23:03] even if that meant fr-tech could not deploy it? [19:23:39] yeah, not sure exactly what. stuff like minfraud credentials [19:23:44] maybe? [19:24:39] yeah there is some stuff i could see being more puppet oriented in there [19:24:58] but i just think why make a whole new dir if the file that's there will suffice at this point? [19:25:26] well, for anyone else, all the provider config would live in /etc as well [19:26:12] yeah but...nobody else is using it :) [19:26:44] baby steps... don't want to design it so they can't! [19:26:56] at this point if anyone was to adopt it they'd need to be a php hacker of some stripe [19:27:17] so i doubt that loading config would be the barrier [19:27:24] yeah, probably not [19:27:38] we do have to transition between the old and new formats, though [19:27:50] i totally agree with the impulse to lower the bar for usage [19:27:52] and it's either put the new stuff in the new file and delete the old file later [19:28:08] or smash it all up in the one file, then delete the old stuff later [19:28:18] which seems really yucky [19:28:27] and even more confusing [19:29:05] maybe a comment in the file could delineate sections? [19:29:27] well, we'd also have to watch out for duplicated <<* yaml references [19:30:20] things that exist in both files in the current version? [19:30:36] the redis credentials, at least [19:31:11] is it really so crazy for /etc/smashpig/main.yaml and /etc/fundraising/SmashPig.yaml to coexist for a couple weeks? [19:31:58] well, i have two opinions about it [19:33:17] first i think it does create kind of a confusing situation, esp with duplication between the files, imagine you get swept up into a mystery of international intrigue in the interrim [19:33:51] it's not like it's terrible, it's just that it's free to not create the situation at all [19:34:13] but it is slightly harder to parse for a newcomer [19:34:30] but everything duplicated in the same file is better? [19:35:11] sorry i'm confused, what is duplicated in the file? [19:35:16] also, I'd want to move the file from /etc/fundraising anyway [19:35:34] cwd main.yaml has all the stuff from SmashPig.yaml's default: section [19:35:34] my impression is this commit pushes all this config from one file into per provider? [19:35:45] only it's not under a default: section, it's all at the top level [19:36:10] so that's the duplicated stuff [19:36:31] also, the new config format isn't expecting all the provider config to be in the global file. [19:36:44] it probably won't mind all that stuff being there, but it's not designed for that [19:37:23] and the old code needs all the provider stuff to stay in /etc/fundraising/SmashPig.yaml [19:37:40] so even if we put to config pointer there and duplicated all the global stuff at the top level of that file [19:37:43] aah i see [19:37:54] we'd still be duplicating all the provider config in that file and /srv [19:38:46] did you take a look at the new one? [19:39:06] yep, agreed that it's nice and short [19:39:24] I'll commit the localsettings stuff in a sec [19:44:54] the other thing is doing it this way creates a situation where you have to have multiple different versions of smashpig running in different places [19:52:24] again it's not that huge of a deal but it seems easy to avoid creating a gap where things are muddled [20:06:14] cwd that's always the case, though [20:06:31] we never deploy smashpig, di, and crm all at the same time [20:07:02] except for superbig things like destroying activemq [20:12:58] yeah, but isn't it still better not to create an undeployable state when you can? [20:13:32] if we have no solid plans to add other stuff to /etc/smashpig [20:14:00] it seems like a messy config file is a preferable temporary situation [20:14:36] we could make the multiple global config files thing happen when there is a need for it [20:14:58] isn't it more of a baby step to do that on its own? [20:15:44] wait, what's the undeployable state? [20:16:07] as in you couldn't upgrade smashpig on crm etc [20:16:10] We deploy the new config file without touching anything else [20:16:27] then we can upgrade everything at our leisure [20:17:07] it'd be a huge pain to revert everything and rewrite the patches that are on top of it [20:17:26] i notice there is already a symlink from /etc/SmashPig.yaml to /etc/fundraising [20:17:51] then reqrite thwrite the whole config split later [20:18:11] oh weird, there should be no need for /etc/SmashPig.yaml to exist [20:18:32] the old code only looked in /etc/fundraising/SmashPig.yaml [20:19:35] that config split won't rebase if we revert it and do anything else [20:20:00] and it took more than a month to get it merged this time [20:20:32] alls i'm sayin is that symlink situation was probably a temporary thing that is now a confusing relic [20:20:46] ok, so let's clean it up! [20:20:57] sure [20:21:09] i'm just trying to do what we can to avoid creating another one [20:21:26] i was not imagining it would require any big rewrites [20:21:37] just to look at the current file instead of the new one? [20:22:55] it's a lot of new ones, and two different classes now [20:23:36] but doesn't it just get the location of the config files from the global config? [20:24:28] https://gerrit.wikimedia.org/r/355577 [20:25:05] it also moves all the stuff that's in the global condif [20:25:07] *config [20:25:20] either up from the default: node to the root level [20:25:27] or into other files entirely [20:26:35] I really don't like the thought of the new code having to deal with the old keys [20:26:42] and the old code having to deal with the new keys [20:27:12] and in either case, we have to do another /etc deploy to clean up after crm and di are updated [20:28:07] so for this deploy, let's just add the new file [20:28:22] and for the next /etc deploy, let's delete the old file and the symlink [20:28:26] way cleaner [20:28:31] and less risky [20:29:30] there are NO keys in the same place in the old config and the new global config [20:30:21] or, at least, the correct location of all of the keys has moved [20:32:07] there could potentially be collisions [20:35:41] alright, i have to run for a few but i'll get this prepped when i return, shouldn't take very long [20:35:46] thanks! [20:35:51] I'll keep testing locally [20:54:13] Fundraising-Backlog, FR-Email: Spike: Can we move bulk export back to running on a slave db? - https://phabricator.wikimedia.org/T152735#3402545 (Ejegg) Open>Resolved a:Ejegg This is switched! [21:02:18] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Silverpop db should know about donations as soon as possible. - https://phabricator.wikimedia.org/T169583#3402561 (Ejegg) [21:03:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Silverpop db should know about donations as soon as possible. - https://phabricator.wikimedia.org/T169583#3402573 (Ejegg) @CCogdill_WMF does this sound like something we could do with Silverpop? The partial update files would contain all t... [21:03:51] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Reflect all unsubscribes via Silverpop in CiviCRM - https://phabricator.wikimedia.org/T161760#3402577 (Ejegg) [21:03:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Import unsubscribe data from Silverpop - https://phabricator.wikimedia.org/T112674#3402579 (Ejegg) [21:05:37] Fundraising Sprint Judgement Suspenders, Fundraising Sprint Kickstopper, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: retrieve lists of contacts who received a particular mailing - https://phabricator.wikimedia.org/T161762#3402584 (Ejegg) [21:05:39] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: More communication between Silverpop and Civi - https://phabricator.wikimedia.org/T114671#3402586 (Ejegg) [21:06:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3402593 (Ejegg) [21:06:54] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Import email-only contacts from 'remind me later' links into CiviCRM - https://phabricator.wikimedia.org/T160949#3402592 (Ejegg) [21:07:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic, FR-Email: EPIC: Add key silverpop data into Civi - https://phabricator.wikimedia.org/T161757#3142095 (Ejegg) [21:36:02] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising-Backlog, fundraising-tech-ops, and 2 others: Make /etc/smashpig config dir - https://phabricator.wikimedia.org/T169585#3402647 (cwdent) [21:45:08] ejegg: I have one thing I wanted to raise with you - I put the tests for the omnimail within the extension - do you think we can add those to ci - or do I need to figure out another way to put testing in our ci [21:50:50] ejegg: that file is in place [21:50:54] eileen as long as they're phpunit, i think we can just add them to the test suite [21:51:01] cwd thanks! [21:51:05] np! [21:51:58] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising-Backlog, fundraising-tech-ops, and 3 others: Figure out a sane way to deploy SmashPig.yaml configuration - https://phabricator.wikimedia.org/T147503#3402674 (cwdent) [21:52:00] Fundraising Sprint Kickstopper, Fundraising Sprint Loose Lego Carpeting, Fundraising-Backlog, fundraising-tech-ops, and 2 others: Make /etc/smashpig config dir - https://phabricator.wikimedia.org/T169585#3402672 (cwdent) Open>Resolved main.yaml is in place [21:54:52] ejegg: as in by editing phpunit.dist? I think we need a few env vars declared but maybe that is fine [21:57:24] (PS7) Eileen: Initial commit, omnimail extension & extendedmailingreport [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/360610 (https://phabricator.wikimedia.org/T161758) [22:01:12] eileen: or just phpunit.xml in the root crm folder [22:01:53] and put the vars in sites/default/bootstrap-phpunit.php [22:16:19] ejegg[m]: trying [22:40:37] (PS1) Eileen: Add extension tests to phpunit.dist [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363099 [22:42:31] (CR) jerkins-bot: [V: -1] Add extension tests to phpunit.dist [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/363099 (owner: Eileen) [22:46:40] hmm - test failed on jenkins due to lack of 'cv' civi tool [22:46:42] 22:42:27 sh: 1: cv: not found [22:46:42] 22:42:27 RuntimeException: Command failed (cv php:boot --level=classloader): [22:46:42] 22:42:27 in cv() (line 30 of /srv/jenkins-workspace/workspace/wikimedia-fundraising- [22:50:07] hmm - it's downloaded /srv/jenkins-workspace/workspace/wikimedia-fundraising-civicrm/src/wikimedia/fundraising/civicrm-buildkit/bin/cv [22:50:08] but perhaps not in the path? [23:43:50] Fundraising-Backlog: SQL challenge: - https://phabricator.wikimedia.org/T169591#3402844 (CCogdill_WMF) [23:45:35] Fundraising-Backlog: SQL challenge: Mailing and donation data - https://phabricator.wikimedia.org/T169591#3402859 (CCogdill_WMF) [23:51:28] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 5 others: Add buildkit/bin to path on jenkins CI environmen... - https://phabricator.wikimedia.org/T169593#3402888 [23:53:16] Fundraising Sprint Gondwanaland Reunification Engine, Fundraising Sprint Homebrew Hadron Collider, Fundraising Sprint Ivory Tower Defense Games, Fundraising Sprint Judgement Suspenders, and 5 others: Add buildkit/bin to path on jenkins CI environmen... - https://phabricator.wikimedia.org/T169593#3402907