[00:03:42] <wikibugs>	 (CR) AndyRussG: Pare down email pref ctr config (2 comments) [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (owner: Ejegg)
[00:03:47] <wikibugs>	 (PS2) AndyRussG: Pare down email pref ctr config [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (owner: Ejegg)
[00:13:07] <wikibugs>	 (PS3) AndyRussG: Pare down email pref ctr config [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (https://phabricator.wikimedia.org/T268510) (owner: Ejegg)
[00:13:48] <AndyRussG>	 ejegg: you shouldn't be working today :) but if you happen to see this and would say it's OK for me to copy that civiproxi api command from fundraising-dev to wmff/install.sh in buildkit, and then +2 that patch, and abandon the other, I could definitely do that..... ;)
[00:23:05] <ejegg>	 oh sure, go ahead AndyRussG !
[00:23:15] <ejegg>	 I was about to do that myself and then the baby woke up
[00:24:10] <AndyRussG>	 ejegg: okok thanks, yeah I got it, baby is important :)
[00:28:47] <wikibugs>	 (CR) AndyRussG: [C: +2] Pare down email pref ctr config [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (https://phabricator.wikimedia.org/T268510) (owner: Ejegg)
[00:28:53] <wikibugs>	 (CR) AndyRussG: [V: +2 C: +2] Pare down email pref ctr config [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683725 (https://phabricator.wikimedia.org/T268510) (owner: Ejegg)
[00:34:22] <wikibugs>	 (PS4) AndyRussG: Delete drupal settings override [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683995 (https://phabricator.wikimedia.org/T281609) (owner: Ejegg)
[00:54:04] <wikibugs>	 (CR) AndyRussG: [C: +2] "Works great, thanks!" [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/683994 (https://phabricator.wikimedia.org/T281609) (owner: Ejegg)
[00:54:35] <wikibugs>	 (CR) AndyRussG: [C: +2] "Wooohooo thanks for this!!" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683995 (https://phabricator.wikimedia.org/T281609) (owner: Ejegg)
[00:54:37] <wikibugs>	 (CR) AndyRussG: [V: +2 C: +2] Delete drupal settings override [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/683995 (https://phabricator.wikimedia.org/T281609) (owner: Ejegg)
[01:14:20] <wikibugs>	 (CR) AndyRussG: [V: +2 C: +2] Restore fredge db, move drupal extra settings [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/683994 (https://phabricator.wikimedia.org/T281609) (owner: Ejegg)
[02:43:34] <wikibugs>	 (PS2) Ejegg: Set wmff CIVI_SITE_KEY for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684121 (https://phabricator.wikimedia.org/T281651) (owner: AndyRussG)
[02:44:15] <AndyRussG>	 ejegg: ah btw getting an error on the fredge schema thing (already merged)
[02:44:36] <AndyRussG>	 only happens when I run buildkit without having reset the persistent storage
[02:44:39] <ejegg>	 oh hmm, let me see if I have any more changes locally that were making it work
[02:44:42] <ejegg>	 ohhh
[02:44:46] <AndyRussG>	 DatabaseSchemaObjectExistsException: Table payments_initial already exists. in /srv/civi-sites/wmff/drupal/includes/database/schema.inc:731
[02:44:59] <ejegg>	 i was always resetting
[02:45:09] <AndyRussG>	 yeah me too
[02:45:20] <AndyRussG>	 I think it just needs a "if exists" somewhere
[02:45:21] <ejegg>	 ah, probably buildkit drops all the tables in civicrm.* and drupal.* ?
[02:45:36] <AndyRussG>	 ah also yes true I think
[02:46:12] <ejegg>	 hmm, that 'if exists' would be in drupal 7 core code
[02:46:31] <AndyRussG>	 ejegg: here's the full error: https://paste.toolforge.org/view/e091f048
[02:48:14] <ejegg>	 if buildkit really is blowing away all those tables, I guess the solution is to have the wmff config also blow away the fredge.* tables when it starts
[02:48:53] <ejegg>	 oh, and we should make a PR upstream for anything we locally merge to buildkit
[02:49:10] <ejegg>	 and then pester eileen to merge them :)
[02:49:24] <ejegg>	 so stuff doesn't break when we switch to upstream
[02:50:20] <AndyRussG>	 ejegg: oh oooops
[02:50:35] <ejegg>	 oops?
[02:50:51] <AndyRussG>	 yes ooops
[02:51:03] * AndyRussG merged
[02:51:35] <ejegg>	 that's fine for now, just so long as we get those things upstream before we switch repos
[02:52:41] <AndyRussG>	 ah hmmmm okok
[02:56:23] <AndyRussG>	 K I can add that in the same place in install.sh
[02:56:41] <ejegg>	 cool cool!
[02:56:46] <ejegg>	 k, see you later
[03:13:48] <wikibugs>	 (PS1) AndyRussG: WMFF: Drop fredge db before creating it [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684138 (https://phabricator.wikimedia.org/T281609)
[03:30:31] <wikibugs>	 (PS3) AndyRussG: WMFF: sync drupal users and set API key for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684003 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg)
[03:49:32] <wikibugs>	 (PS4) AndyRussG: WMFF: sync drupal users and set API key for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684003 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg)
[03:52:31] <wikibugs>	 (PS3) AndyRussG: Set Civicrm API key for Civiproxy [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684006 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg)
[05:09:12] <wikibugs>	 (PS3) AndyRussG: WMFF: Static CIVI_SITE_KEY for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684121 (https://phabricator.wikimedia.org/T281651)
[05:34:04] <wikibugs>	 (PS4) AndyRussG: Set static CIVI_SITE_KEY for Civiproxy [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684122 (https://phabricator.wikimedia.org/T281651)
[07:20:21] <wikibugs>	 Fundraising Sprint File Systems Stage Show, Fundraising Sprint Git Rebase Jump, Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, and 2 others: Production of New Annual Fund Thank You Email... - https://phabricator.wikimedia.org/T278363
[08:12:49] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 1 [=1]
[08:17:49] <icinga-wm>	 RECOVERY - check_log_messages on frav1002 is OK: OK
[16:03:25] <wikibugs>	 Fundraising-Backlog: Spanish Latam donors seeing English survey - https://phabricator.wikimedia.org/T281731 (krobinson)
[16:08:32] <wikibugs>	 Fundraising-Backlog: Spanish Latam donors seeing English survey - https://phabricator.wikimedia.org/T281731 (DStrine) @krobinson would this be the survey on the TY page or something else?   @KHaggard @MNoorWMF @TSkaff @Pcoombe FYI. Have there been any surveys emailed by the email team? Has the TY page survey...
[16:14:59] <wikibugs>	 Fundraising-Backlog: Spanish Latam donors seeing English survey - https://phabricator.wikimedia.org/T281731 (KHaggard) @DStrine TY page survey only - no surveys sent through email recently and no TY page survey updates recently. I've checked the es-419 and es TY pages and the survey links are correct and in...
[16:22:32] <wikibugs>	 Fundraising-Backlog: Spanish Latam donors seeing English survey - https://phabricator.wikimedia.org/T281731 (DStrine) Per slack this is the TY page survey. The donors have not mentioned that the Ty page is in english so it might just be the survey. Fr-tech can't do anything with this. It's possible survey mo...
[16:25:43] <wikibugs>	 Fundraising-Backlog, AbuseFilter, Analytics-Radar, BetaFeatures, and 46 others: Prepare User group methods for hard deprecation - https://phabricator.wikimedia.org/T275148 (AndyRussG)
[17:53:14] <dwisehaupt>	 howdy fr-tech: as of yet, do we have an idea of what the http requests to civiproxy will look like? i'm looking at possibilities for request regexes to make sure only the enpoints/queries we want hit the proxy layer.
[17:54:38] <ejegg>	 dwisehaupt: we have that all set up in dev so it shouldnt be too hard to capture a couple examples
[17:55:22] <dwisehaupt>	 cool. not a rush on this but it would be good to get in from the start if possible.
[17:56:07] <wikibugs>	 Fundraising-Backlog, FR-Email: Civi: Contact records can be opted in even when on the Master Suppression List - https://phabricator.wikimedia.org/T274146 (KHaggard) @DStrine and @Eileenmcnaughton - Brian found a lot of things in the MSL purge, so I've summarized it a Google Doc for now. Please let me kno...
[18:00:17] <AndyRussG>	 dwisehaupt ejegg https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/DonationInterface/+/refs/heads/master/extras/civiproxy/CiviproxyConnect.php#18
[18:00:36] <AndyRussG>	 contact_id is numerical and hash is [0-9a-f]
[18:03:38] <AndyRussG>	 Here is the civi API that eventually is hit: https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/crm/+/e08f34fba4db3c2e066833b538f62dd698c48445/drupal/sites/default/civicrm/extensions/wmf-civicrm/api/v3/Civiproxy/Getpreferences.php
[18:50:42] <dwisehaupt>	 thanks. i'll have a look.
[20:22:53] <AndyRussG>	 fr-tech cstone here are the last patches in the chain for fundraising-dev, which should get you the civipreference setup automagically
[20:23:46] <AndyRussG>	 for buildkit: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm-buildkit/+/684121
[20:24:01] <AndyRussG>	 for fundraising-dev: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/684122
[20:24:08] <cstone>	 thanks AndyRussG
[20:24:14] <AndyRussG>	 cstone: thank u :)
[20:24:34] <AndyRussG>	 cstone fr-tech don't forget to do rm -rf src/civ-sites/wmff/vendor first
[20:24:45] <AndyRussG>	 and also docker-compose down before updating fundraising-dev
[20:25:07] <AndyRussG>	 after those steps, if you run it through setup.sh it should just work
[20:25:37] <AndyRussG>	 or if you want to run the civi build step directly on the command line in the container, first do rm /srv/civi-sites/wmff/wmff.sh
[20:25:46] <AndyRussG>	 and then you can do civibuild create wmff
[20:27:12] <AndyRussG>	 you'll get the proper fredge database created with all that, too, btw
[20:27:17] <AndyRussG>	 ahhhhh hopefully anyway ;p
[20:40:11] <cstone>	 woo AndyRussG got further than friday
[20:40:42] <AndyRussG>	 ahhh hmmmm and.....?
[20:41:42] <cstone>	 still installing
[20:41:48] <AndyRussG>	 ahhh okok good news!
[20:42:01] <AndyRussG>	 dwisehaupt Jeff_Green I just confirmed locally that I can set Redis port on a per-queue basis
[20:42:02] <AndyRussG>	 works fine
[20:42:10] <Jeff_Green>	 great!
[20:42:14] <AndyRussG>	 yeah!
[20:42:19] <AndyRussG>	 so I think no need to even make a task for that!
[20:42:45] <AndyRussG>	 Jeff_Green dwisehaupt what I'm not clear on is who does what to make the email-pref-ctr-specific SmashPig config?
[20:43:00] <Jeff_Green>	 wellllllllll
[20:43:26] <AndyRussG>	 Jeff_Green is it puppetized or in localsettings? hmmm I guess I can check heheh
[20:43:28] <Jeff_Green>	 ideally we'll be able to have that fully served by puppet
[20:43:31] <Jeff_Green>	 it is right now
[20:43:38] <AndyRussG>	 Jeff_Green yeah that sounds reaosnable
[20:44:10] <Jeff_Green>	 it's in puppet, deployed to /etc/fundraising and f_c_u deploys a symlink to it
[20:44:33] <AndyRussG>	 hmmm
[20:44:52] <AndyRussG>	 Jeff_Green if you point me to the current puppet template I can propose a one for email-pref-ctr and put it in /tmp like e jegg did for localsettings
[20:44:55] <Jeff_Green>	 if it turns out there's something that has to be outside of puppet, I think there are other contexts where we source the /etc/fundraising file from one in localsettings
[20:45:15] <AndyRussG>	 I don't think there is anything
[20:45:22] <Jeff_Green>	 ok on sec
[20:45:57] <AndyRussG>	 also I don't know how the symlink works, but basically the requirement is just that it be different for the boxes that run e-p-c
[20:46:23] <Jeff_Green>	 there's a template that is specific to donorwiki in the puppet tree
[20:46:41] <Jeff_Green>	 puppet/modules/fundraising/templates/donorwiki/LocalSettings.php.erb
[20:47:10] <AndyRussG>	 Jeff_Green: but for SmashPig, it's like a main.yaml file that ends up in /etc/smashpig
[20:47:25] <Jeff_Green>	 looking
[20:48:58] <Jeff_Green>	 the regular smashpig config is puppetized, but not (yet) deployed to frdata servers
[20:49:52] <Jeff_Green>	 we can configure it differently as needed, without any template modification we can tweak db/queue settings for it
[20:50:31] <Jeff_Green>	 that template is puppet/modules/fundraising/templates/smashpig/main.yaml.erb
[20:51:31] <Jeff_Green>	 if we need to vary the structure of the template otherwise that's pretty straightforward puppetwise
[20:56:36] <AndyRussG>	 Jeff_Green: ok so we need to add another key after unsubscribe:
[20:56:55] <cstone>	 woohoo AndyRussG !!! fredge db and api key works
[20:57:09] <AndyRussG>	 cstone ahhhhh fantasmic yaaaay
[20:57:51] <cstone>	 running to post office now ill be back
[20:58:07] <AndyRussG>	 cstone though I'm not sure we should merge the fundraising-dev patches until the buildkit patches get through, which would be I guess more in eileen's corner?
[20:58:21] <AndyRussG>	 Jeff_Green  the new key will be 'email-preferences'
[20:58:40] <AndyRussG>	 and instead of saying     <<: *REDIS right below it, like the others
[20:59:22] <AndyRussG>	 it'll be followed by entries that are similar to what's right under the top-level redis-servers key
[20:59:25] <Jeff_Green>	 ok
[20:59:37] <AndyRussG>	 i.e. class: PHPQueue.... etc
[20:59:53] <AndyRussG>	 and then with the e-p-c-specific port and password and such
[21:00:23] <AndyRussG>	 does that make sense?
[21:00:51] <Jeff_Green>	 I think so? I'll try to mock it up
[21:01:33] <AndyRussG>	 Jeff_Green: cool beans, thanks much!
[21:02:17] <AndyRussG>	 Jeff_Green: I think potentially we can also leave off all those other queues, but I think also it won't make much difference, since they're there by default in SmashPig in any case, IIRC
[21:03:05] <Jeff_Green>	 what I may do is change the structure of the template to only include a stanza if there's config for it
[21:04:15] <Jeff_Green>	 i.e. if we don't specify fredge_db_reader we won't add the fredge-db data store block
[21:04:44] <AndyRussG>	 Jeff_Green: right... ah also no provider-configuration-directory needed
[21:04:50] <AndyRussG>	 or sequence-generator
[21:27:20] <Jeff_Green>	 AndyRussG: I'm a little unclear what the config would look like on the civicrm server
[21:27:50] <Jeff_Green>	 i.e. where we would want two queue server stanzas
[21:28:50] <AndyRussG>	 Jeff_Green: so what worked locally is just adding email-preferences as a queue name, and putting the specific Redis config under that, instead of the  <<: *REDIS backreference
[21:29:32] <Jeff_Green>	 can you put a copy of that config in tmp on frpm1001 or something?
[21:29:58] <AndyRussG>	 Jeff_Green: yep one sec
[21:31:11] <AndyRussG>	 Jeff_Green: oh I just noticed it already has an email-preferences key
[21:31:18] <AndyRussG>	 just not where I was expecting it should go
[21:31:56] <Jeff_Green>	 oh, right, that must be for the unsubscribe stuff in paymentswiki
[21:34:06] <AndyRussG>	 Jeff_Green: no I think someone just already added it?
[21:34:16] <AndyRussG>	 there's the unsubscribe queue for that I think?
[21:34:20] <AndyRussG>	 I might be wrong tho
[21:34:33] <AndyRussG>	 Jeff_Green: so, see frpm1001:/tmp/new-draft-main.yaml.erb
[21:37:03] <Jeff_Green>	 AndyRussG: you're right, git says Dallas added it a few days ago
[21:37:06] <Jeff_Green>	 looking
[21:37:19] <AndyRussG>	 ahh cool beans :)
[21:37:58] <Jeff_Green>	 ok I see
[21:38:23] <AndyRussG>	 Jeff_Green: so also I added the new template parameters that I think would be needed?
[21:38:26] <AndyRussG>	 like @queue_redis_epc_port etc
[21:38:39] <Jeff_Green>	 yup
[21:38:50] <AndyRussG>	 :)
[21:38:51] <dwisehaupt>	 yeah. i added it in for T268512, but that was before we discussed the separate instance.
[21:38:52] <stashbot>	 T268512: Add new queue settings for email-preferences - https://phabricator.wikimedia.org/T268512
[21:38:58] <Jeff_Green>	 I'm on the fence about making two separate templates
[21:39:28] <AndyRussG>	 Jeff_Green: welp we definitely don't want the provider directory on the new wiki
[21:39:36] <Jeff_Green>	 right
[21:39:38] <AndyRussG>	 and it feels kinda odd to have that in the config pointing to nothing
[21:39:50] <AndyRussG>	 though I guess technically it might not break anything
[21:40:11] <Jeff_Green>	 it's easy to leave out stuff like that in puppet, once you can envision all the variants you need
[21:40:42] <AndyRussG>	 ah right I see
[21:40:44] <Jeff_Green>	 I'm not entirely clear on how this config file works though
[21:41:00] <Jeff_Green>	 the top stanza breaks my brain
[21:41:03] <AndyRussG>	 so  still two different configs, just created from a single template with different parameters
[21:41:07] <Jeff_Green>	 right
[21:41:21] <Jeff_Green>	 three, actualy
[21:41:28] <AndyRussG>	 hmmmm
[21:41:32] <AndyRussG>	 okok
[21:41:36] <Jeff_Green>	 payments/payments-listeners vs civicrm vs frdata
[21:41:40] <AndyRussG>	 right
[21:41:58] <AndyRussG>	 Jeff_Green: wrt the stanza, I'm also fuzzy on the details of how yaml backreferences or whatever they're called work
[21:42:01] <AndyRussG>	 but basically it works
[21:42:14] <AndyRussG>	 and that's about as far as I get
[21:42:17] <Jeff_Green>	 right
[21:43:32] <Jeff_Green>	 I'll see if I can figure it out, if I can standardize on "all the queue-server stuff goes in redis-servers: and all the queues get their config from there by a backreference, I think it will be easier/more logical to include/exclude stuff per server role
[21:44:06] <Jeff_Green>	 I'll have to work on this tomorrow though, it's late here!
[21:44:39] <AndyRussG>	 Jeff_Green: okok.... just one quick note, definitely for the civicrm we need two different versions of the queue config
[21:44:45] <AndyRussG>	 or at least two different ports, in a single file
[21:45:07] <Jeff_Green>	 yep!
[21:45:10] <AndyRussG>	 since that's the single box that needs to consume queue messages from both redis instances
[21:45:13] <AndyRussG>	 :) cool :)
[21:49:58] <ejegg>	 Jeff_Green: I think we need the provider-defaults.yaml file still on the email pref center wiki
[21:50:41] <ejegg>	 oh wait, two different redis servers?
[21:50:51] * ejegg needs to read more backscroll
[21:51:21] <AndyRussG>	 ejegg: yeah the proposal is to have a different redis server on a different port for email-pref-ctr
[21:52:12] <ejegg>	 oh ok
[21:52:56] <AndyRussG>	 ejegg: btw I haven't tested any of the provider-defaults stuff with the current patches
[21:53:03] <AndyRussG>	 that file is fully commented out on Docker
[21:53:20] <ejegg>	 so yah, you can set the queue params on a per-queue basis in the main.yaml no problem
[21:53:35] <AndyRussG>	 ejegg: yep!
[21:53:42] <ejegg>	 AndyRussG: oh, if it's all commented out in docker, then we're just using the defaults from the SmashPig repo
[21:53:48] <AndyRussG>	 okok
[21:54:10] <AndyRussG>	 ejegg: I added some notes on the localsettings.php.erb in temp on the etherpad
[21:54:12] <ejegg>	 https://phabricator.wikimedia.org/diffusion/WFSP/browse/master/config/provider-defaults.yaml
[21:54:44] <AndyRussG>	 ejegg: hmmm
[21:55:02] <ejegg>	 so just logging to syslog, no failmail
[21:55:20] <AndyRussG>	 ejegg: so not sure if any of the logging calls for e-p-c go through SP?
[21:55:34] <Jeff_Green>	 ejegg: I don't really understand how the backreferences work in the existing file, is that quick to explain? otherwise I can research it tomorrow
[21:55:34] <AndyRussG>	 though I do see the email entry there
[21:56:34] <ejegg>	 Jeff_Green: they're standard yaml backreferences
[21:57:06] <AndyRussG>	 Jeff_Green: it's basically just that the part below &REDIS is considered to be copied and inserted in all the places that it says <<: *REDIS
[21:57:06] <ejegg>	 the reference just populates the node it's under with the content of the node it's referring to
[21:57:34] <AndyRussG>	 it's just interpreted as such by the yaml interpreter
[21:58:04] <AndyRussG>	 so, completely outside the domain of erb/puppet stuff, if that's a question
[21:58:11] <Jeff_Green>	 ejegg: so without familiarity with yaml backreferences yet, what I'd like to do is separately name something to the effect of *DONATIONQUEUE vs *EMAILPREFSQUEUE
[21:58:48] <ejegg>	 Jeff_Green: if you're going to just use the different settings once, then just add them directly under the email-preferences node
[21:58:54] <ejegg>	 and don't use any backreference ther
[21:58:56] <ejegg>	 *there
[21:59:09] <AndyRussG>	 fr-tech just gonna accompany Doggo on his "errands", back in about 20 min :)
[21:59:23] <Jeff_Green>	 right, I'm just thinking in terms of repeatability in the puppet template
[22:00:00] <Jeff_Green>	 and thinking ahead to when we decide we need more than one queue in the EMAILPREFSQUEUE redis instance
[22:00:06] <ejegg>	 repeatability? so this second redis instance will have more than just the email-preferences queue?
[22:00:49] <Jeff_Green>	 not today, but if I can design the template well now maybe I can save pain in the future
[22:00:59] <ejegg>	 hmm
[22:01:13] <Jeff_Green>	 in redis-servers:
[22:01:53] <ejegg>	 I'd say go for simplicity for now and just do it without a backreference till we come up with that second queue
[22:01:54] <Jeff_Green>	 what is &REDIS and everything under it? is it all one value?
[22:02:16] <ejegg>	 &REDIS 'captures' all of the stuff under it
[22:02:29] <ejegg>	 and stores it as the REDIS set of nodes
[22:02:40] <ejegg>	 to be pasted in anywhere you see <<< REDIS
[22:02:47] <Jeff_Green>	 ok
[22:05:29] <Jeff_Green>	 so is value "redis-servers" arbitrary, just there so that the &REDIS part will be interpreted? or is redis-servers called anywhere else?
[22:05:59] <ejegg>	 yep, exactly, that node is just a placeholder
[22:06:07] <ejegg>	 oop, gotta head out
[22:06:12] <Jeff_Green>	 ok thanks!
[22:14:22] <eileen>	 cstone: you are reviewing that last patch? do you need any help?
[22:14:30] <eileen>	 I'm gonna check about monolog deployment
[22:15:08] <cstone>	 eileen: im getting some food right now but im going to look at it after that, i got my local setup behaving which was the biggest issue on friday
[22:15:17] <eileen>	 cool
[22:20:11] <wikibugs>	 (PS1) Eileen: Vendor update, phpmailer, monolog handler [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510
[22:20:25] <wikibugs>	 (CR) Eileen: [C: +2] Vendor update, phpmailer, monolog handler [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510 (owner: Eileen)
[22:22:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: minFraud_endpoint_critical 2 [=1]
[22:23:36] <wikibugs>	 (CR) jerkins-bot: [V: -1] Vendor update, phpmailer, monolog handler [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510 (owner: Eileen)
[22:25:33] <wikibugs>	 (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510 (owner: Eileen)
[22:25:46] <cstone>	 woo consuming and updating !! :)
[22:27:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 1 [=1],minFraud_endpoint_critical 3 [=1]
[22:28:11] <wikibugs>	 (CR) jerkins-bot: [V: -1] Vendor update, phpmailer, monolog handler [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510 (owner: Eileen)
[22:28:11] <dwisehaupt>	 hmmm... looking at those check_log_messages errors.
[22:31:13] <dwisehaupt>	 looks like it may be transient. i'll keep an eye on it.
[22:32:30] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 1 [=1],cURL_error 5 [=5],minFraud_endpoint_critical 4 [=1]
[22:34:03] <AndyRussG>	 cstone yaaaay :) :)
[22:35:02] <cstone>	 question on the part when it logs when none are processed, is there a reason were logging that?
[22:37:15] <AndyRussG>	 cstone: hmmm not sure that might have come form eileen's initial version of the patch, but I don't see why it has to be a conditional like that
[22:37:32] <cstone>	 Just seems it would cause a lot of spam
[22:37:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 1 [=1],Paypal_endpoint_critical 3 [=1],cURL_error 5 [=5],minFraud_endpoint_critical 3 [=1]
[22:37:37] <AndyRussG>	 as in, it seems fine if it just says "Processed 0 messages"
[22:37:44] <AndyRussG>	 cstone: ah I see right
[22:37:49] <AndyRussG>	 hmmm
[22:37:59] <cstone>	 Cause if it's running at whatever frequency and no one is unsubing at most times
[22:38:01] <AndyRussG>	 we can also get rid of it
[22:38:03] <AndyRussG>	 yeah
[22:38:18] <AndyRussG>	 so we wouldn't need something like that for debugging?
[22:38:33] <cstone>	 Hmm like a general it did x thing ?
[22:38:46] <cstone>	 Is there a success one sorry I'm on phone now walking back
[22:39:03] <cstone>	 If it's just the results of the run then that makes sense
[22:41:51] <AndyRussG>	 cstone: I think that's it? looking in other queue consumers, it's the same pattern, see queue2civicrm drupal module
[22:42:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 2 [=1],Paypal_endpoint_critical 2 [=1],cURL_error 7 [=5],minFraud_endpoint_critical 2 [=1]
[22:47:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 2 [=1],minFraud_endpoint_critical 1 [=1]
[22:50:34] <cstone>	 ahh i see AndyRussG I was thinking it was doing it an extra time but nope
[22:50:59] <cstone>	 ok that makes sense
[22:52:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: Paypal_endpoint_critical 2 [=1],cURL_error 6 [=5],minFraud_endpoint_critical 4 [=1]
[22:54:11] <eileen>	 there is a problem in vendor just updating the phpmailer code - which is odd
[22:54:31] <eileen>	 since the monolog extension is not in that & I can't think what else
[22:55:28] <AndyRussG>	 eileen: did you see that when the vendor directory is there and you run civibuild create wmff, it doesn't create the civicrm_monolog table?
[22:55:41] <eileen>	 AndyRussG: sorry this is gerrit
[22:55:49] <eileen>	 & without the monolog patch
[22:56:02] <AndyRussG>	 ahh okok
[22:56:09] <eileen>	 https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/684510
[22:56:27] <eileen>	 (that should merge to vendor & not have much to do with monolog)
[22:56:32] <AndyRussG>	 ah okok
[22:56:37] <AndyRussG>	 eileen: I was looking at https://phabricator.wikimedia.org/T281647
[22:56:49] <AndyRussG>	 but maybe that's the fix'
[22:56:51] <AndyRussG>	 ?
[22:57:17] <eileen>	 AndyRussG: yeah - I wasn't going to look at that until we had merged monolog to deployment - but this is pre-monolog in deployment
[22:57:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 1 [=1],minFraud_endpoint_critical 2 [=1]
[22:57:40] <AndyRussG>	 hmmm
[22:57:46] <AndyRussG>	 that's a hefty patch
[22:59:00] <wikibugs>	 (CR) Eileen: "recheck" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/683482 (owner: Eileen)
[22:59:10] <eileen>	 the composer update one?
[22:59:38] <eileen>	 I'm gonna retry merging just the phpmailer - my current theory is that maybe the other package update touches symfony
[22:59:54] <eileen>	 if this still passes https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/vendor/+/683482 - I'll merge it
[23:00:05] <eileen>	 & get the phpmailer update done first
[23:01:48] <AndyRussG>	 eileen: K I'm not understanding much but pls lmk if I can help w stuff :)
[23:01:57] <wikibugs>	 (CR) jerkins-bot: [V: -1] phpmailer update to 6 [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/683482 (owner: Eileen)
[23:02:07] <eileen>	 lol ok - I'll see what happend
[23:02:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 1 [=1],Paypal_endpoint_critical 2 [=1],minFraud_endpoint_critical 6 [=1]
[23:02:38] <eileen>	 nope phpmailer patch failing now too - even though it passed last week - same patch
[23:07:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: cURL_error 6 [=5],minFraud_endpoint_critical 3 [=1]
[23:12:34] <icinga-wm>	 PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 2 [=1],Paypal_endpoint_critical 1 [=1],minFraud_endpoint_critical 3 [=1]
[23:13:39] <dwisehaupt>	 these alerts are annoying but not fatal. i'm going to ack them for a while as i have to kid taxi in a few.
[23:14:54] <icinga-wm>	 ACKNOWLEDGEMENT - check_log_messages on frav1002 is CRITICAL: CRITICAL: GlobalCollect_endpoint_critical 2 [=1],Paypal_endpoint_critical 1 [=1],minFraud_endpoint_critical 3 [=1] Dwisehaupt transient errors on a few providers from different hosts. acking for now to give it time to settle.
[23:21:58] <eileen>	 Hmm Ok I think it's the changes made in https://phabricator.wikimedia.org/T281609
[23:22:41] <cstone>	 hmm gerritt isn't loading for me
[23:24:07] <AndyRussG>	 eileen: oh hmmm which of those?
[23:24:08] <eileen>	 I'm finding it does after a bit
[23:25:06] <eileen>	 AndyRussG: I'm pretty sure the reason this line was commented out https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm-buildkit/+/683994/2/app/config/wmff/install.sh is because it caused the problem we are hitting
[23:26:32] <eileen>	 for history - that whole precreated.sql thing is because it used to be the case that we created our db on vagrant and had to let vagrant pre-create because of internal wmff setup
[23:28:21] <AndyRussG>	 hmmm okok
[23:28:32] <AndyRussG>	 eileen: I can test commenting that out again...
[23:28:39] <AndyRussG>	 any suggestions then on another way to do that?
[23:28:57] <eileen>	 ideally unravel the whole CI pre-create dbs thing
[23:28:59] <eileen>	 (somehow)
[23:29:09] <eileen>	 under wmff there is a bin/ci-create-dbs and that sets up the databases when jenkins is running it
[23:29:50] <AndyRussG>	 hmmm
[23:29:56] <AndyRussG>	 also I don't see any precreated.sql anywhere
[23:31:21] <eileen>	 no I'm trying to recall where our script causes that to be used
[23:31:42] <eileen>	 in the past the fredge thing has never worked quite right because we've had to prioritise CI over local
[23:32:14] <eileen>	 but if we could get the magic out of ci it would help - I was hoping that once we had the repo phab done we might figure that out
[23:34:19] <wikibugs>	 (PS1) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537
[23:34:38] <eileen>	 AndyRussG: I don't think ^^ is magically going to work but it's the right code areas
[23:37:27] <wikibugs>	 (PS2) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537
[23:38:01] <eileen>	 AndyRussG: in the meantime I think we have to comment out the line again because we have to prioritise CI
[23:41:22] <wikibugs>	 (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen)
[23:44:40] <wikibugs>	 (PS3) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537
[23:46:12] <AndyRussG>	 eileen: before you dive too deep, what about if we just add directly create the db in setup.sh?
[23:46:23] <AndyRussG>	 and leave that stuff commented out then?
[23:46:27] <AndyRussG>	 would that be easier for now?
[23:47:37] <eileen>	 well we Do need to get rid of the precreated stuff at some point because it is a pain - but yeah unless it turns out to be easier than I think we might need to do that.
[23:47:38] <AndyRussG>	 also cstone eileen not meaning to distract or change topic here, just a thought, if we could get the queue consumer merged today that'd prolly be nice? :)
[23:47:57] <AndyRussG>	 eileen okok yeah I think that's fine eh
[23:48:00] <wikibugs>	 (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen)
[23:48:13] <cstone>	 AndyRussG: its looking good just trying to break it haha
[23:48:27] <AndyRussG>	 eileen without having looked, it sounds like it's a pretty labyrinthine rabbit hole
[23:48:31] <AndyRussG>	 cstone ah cool beans thx!
[23:48:46] <cstone>	 i guess what i was trying to do was get to these messages https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/682349/13/drupal/sites/default/civicrm/extensions/wmf-civicrm/CRM/Queue/PreferencesQueueConsumer.php
[23:48:51] <cstone>	 cause i dont see them in the logs
[23:51:23] <AndyRussG>	 cstone: right... well the one on line 21 ("No records updated from e-mail preferences message") I think would only happen if somehow a record wasn't written, or multiple records were written, and an exception wasn't thrown
[23:51:41] <AndyRussG>	 you could change the !== 1 conditional just to test that it gets to the log
[23:52:52] <AndyRussG>	 I have seen the error message about invalid data appear somewhere, I think on the command  line when the test was broken... but I'm not positive that it goes to all the right places, I guess
[23:52:54] <cstone>	 ok cool that worked
[23:53:04] <AndyRussG>	 cool! :)
[23:54:14] <cstone>	 i think this looks good if gerrit would behave
[23:56:09] <cstone>	 i guess the only thing is the log messages have a empty [] after them
[23:56:23] <cstone>	 but thats probably a log issue?
[23:56:49] <wikibugs>	 (PS4) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537
[23:58:10] <AndyRussG>	 cstone: ah yeah I think that's a thing with the new monolog stuff, where you can send in an array with the values of variables you want to log, or something?
[23:58:27] <cstone>	 ah nice
[23:58:48] <AndyRussG>	 dunno if there's a different value we can send in to prevent the []? I think it's an argument we're not sending
[23:59:53] <cstone>	 yeah its fine for now