[00:00:24] gerrit wont let me +2 hahaa [00:00:26] come onn [00:00:53] (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen) [00:01:33] awwww [00:01:41] ehhh thanks also!! [00:01:42] (CR) Cstone: [C: +2] "This looks good! Updates the contact, logs (both when there are messages on the queue and when there are none). I got a bad message onto t" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [00:01:48] yayyyy [00:01:52] haha went to other computer [00:02:48] woohooo glad that worked heheh :) :) [00:04:32] (CR) jerkins-bot: [V: -1] E-mail pref ctr queue consumer using api [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [00:07:40] RECOVERY - check_log_messages on frav1002 is OK: OK [00:18:03] (CR) AndyRussG: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [00:18:15] (PS5) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 [00:21:51] (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen) [00:24:59] (PS6) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 [00:28:21] (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen) [00:29:55] (PS7) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 [00:33:31] (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen) [00:42:57] (PS8) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 [00:43:13] cstone: it didn't merge :| [00:46:28] (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (owner: Eileen) [00:48:21] AndyRussG: I'm close to getting that precreate stuff fixed - it's falling over on the grant line now which is much later [00:50:43] eileen: ah okok cool! I haven't looked at it yet, but I could in about 1 hr [00:53:18] eileen: ah oops looks like for this same reason, the e-mail pref queue consumer is not merging! [00:53:20] https://paste.toolforge.org/view/47c2b576 [00:53:46] (PS9) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 [00:53:48] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/5105/console [00:55:16] (PS1) Eileen: Do not attempt to set grant permissions [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684568 (https://phabricator.wikimedia.org/T281609) [00:55:36] AndyRussG: yeah - because I don't think updating buildkit triggers Ci so it wasn't obvious the changes broke it [00:56:13] I think if we roll back the grant permission line for now I can get the rest through - ie with 684568 merged I think 684537 will merge [00:58:24] Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: fix fredge database setup - https://phabricator.wikimedia.org/T281609 (Eileenmcnaughton) Just noting that this broke CI because of the weird mysql precreated stuff - I think if we can get this through then we can whack this mole dead... [01:10:51] Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: fix fredge database setup - https://phabricator.wikimedia.org/T281609 (Eileenmcnaughton) I've logged this upstream https://github.com/amp-cli/amp/issues/67 to see the response to making db names specifiable (the reason we currently p... [01:19:15] eileen: I can +2 this one https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm-buildkit/+/684568/ [01:19:23] (removes the grant permissions line) [01:20:12] eileen: I guess you also saw this one? [01:20:14] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/civicrm-buildkit/+/684138 [01:20:32] Now recalling, the current master of buildkit was also failing locally if the db already existed [01:20:43] so that other patch drops the whole database too [01:20:49] I guess that doesn't help tho? [01:21:30] anyway happy to +2 and merge the removal of the gran permissions line... [01:21:33] AndyRussG: well not with the CI problem [01:21:54] If you +2 the one you marked I can see if that will fix the CI problem [01:22:26] the problem with this line [01:22:26] echo "GRANT ALL ON fredge.* TO 'drupal'@'%'"| amp sql -N civi -a [01:22:36] is that on ci the user is not called 'drupal' [01:22:50] ahhhh hmmm ok gotcha [01:22:51] - but we can come back to that if we get CI working & unblocking [01:23:04] K fun times heheheh [01:23:28] yeah -but maybe this time we will get a more durable fix [01:23:53] (CR) AndyRussG: [C: +2] Do not attempt to set grant permissions [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684568 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [01:23:56] (CR) AndyRussG: [V: +2 C: +2] Do not attempt to set grant permissions [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684568 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [01:25:14] (PS10) Eileen: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) [01:25:53] ok - retrying [01:30:39] (CR) AndyRussG: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [01:32:11] (CR) jerkins-bot: [V: -1] Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [01:32:50] dang CI failing on the permissions [01:36:46] eileen: hmm also this one dying as before: [01:36:48] 20:32:57 +++ echo 'CREATE DATABASE IF NOT EXISTS fredge' [01:36:50] 20:32:57 +++ amp sql -N civi -a [01:36:52] 20:32:57 [01:36:54] 20:32:57 Fatal error: Uncaught Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException: Circular reference detected for service "db.mysql_precreated", path: "db.mysql_precreated -> instances". in phar:///src/wikimedia/fundraising/crm/civicrm-buildkit/bin/amp/vendor/symfony/dependency-injection/Symfony/Component/DependencyInjection/Container.php:296 [01:37:02] (the email pref ctr api one) [01:37:20] eileen: we should probably abandon the rabbi thole pretty soon....? [01:39:51] AndyRussG: we could maybe look up the user name in mysql ? [01:41:42] the mysql docs suggest this should work but it doesn't GRANT ALL ON civicrm.* TO ''@'localhost' [01:43:41] (CR) DannyS712: [C: +2] build: Updating composer dependencies [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/684586 (owner: Libraryupgrader) [01:46:43] eileen: I suggest just reverting the whole create fredge db thing for now, and for fr-dev just making that run in setup.sh [01:47:07] I think the progress made by you and e jegg on fixing the db stuff is great [01:47:16] and isn't work that'd be lost if we do that [01:47:20] Hmm - I think there is a different part we can revert & go back to not having a separate db on fredge [01:47:24] on dev [01:47:47] or a for now anyway [01:48:12] hmmm dunno, I think we do want the db setup to be as similar as possible? that's why E lliott dug into it I think? i don't know the implications [01:48:24] but I also don't imagine it's a blocker for now to do it like you say, either [01:52:02] (PS1) Eileen: Use drupal db for fredge [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684600 (https://phabricator.wikimedia.org/T281609) [01:52:28] AndyRussG: yeah - we can switch back later - but ^^ should get us back to CI working [01:52:52] & we should be in a better place to re-solve it later because we are down to just one problem with it [01:56:01] eileen: I don't understand that one [01:56:09] install.sh will still create the db then? [01:56:25] shouldn't we just revert there instead? or at least, also? [01:56:41] yeah - we can still create there because the goal is to re-instate that working [01:57:07] & by leaving it there we lock in the things that stop us breaking that part again [01:58:05] But that bit of code tells it to use the same db for fredge so we won't have permissions issues [02:00:24] eileen: ok [02:00:32] what we want is to make sure that the parts of the puzzle we do solve are the same for CI & local dev environments & we managed to get the create part solved for both (finally) I think - we need to confirm once that one is merged [02:00:59] eileen: you saw on the ci for the e-p-c queue consumer it was complaining about some symfony circular dependency, right? [02:01:13] I guess you're understanding that as a side effect and not the real error? [02:01:14] AndyRussG: yep - that is the precreate db stuff [02:01:27] ok [02:02:04] do you want to merge the "Use drupal db for fredge" patch (https://gerrit.wikimedia.org/r/684600) or should I test it first somewhere? [02:02:12] *merge blindly [02:02:30] lol - you can merge blindly since CI will spit the dummy if it fails :-) [02:03:20] okok yis [02:03:56] (CR) AndyRussG: [C: +2] "aaahhhhhhh" [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684600 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [02:04:02] (CR) AndyRussG: [V: +2 C: +2] Use drupal db for fredge [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684600 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [02:04:33] (CR) AndyRussG: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [02:04:38] K [02:04:38] AndyRussG: I just found this line in another file [02:04:38] GRANT ALL PRIVILEGES ON $CIVI_DB_NAME.* TO $CMS_DB_USER@'%'; [02:04:47] it might be those params are there [02:04:59] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [02:05:06] I guess later I'll see later what happened to fr-dev [02:05:11] So your one won't pass without this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/684537 [02:05:11] eileen: ah interesting! [02:05:19] so I'm doing re-check on that now [02:06:48] K I'm understanding that even less [02:07:13] those are files in the main crm repo that are only used by ci? [02:08:54] yep [02:09:16] & they are what causes the circular dependency when you tried to create fredge the other way [02:12:30] weird [02:12:56] so we're now un-creating fredge for CI and will use the drupal db like the buildkit patch I blindly merged says, I guess? [02:13:12] I see a bunch more stuff in that patch other than just not creating fredge? [02:13:38] eileen: if you think it'd be more efficient to fix this we could also talk it out on a call [02:14:18] hmm I'm still kinda digging in - but we can remove that create I guess because it's failed again so we are not quite there on it [02:15:39] K hmmm so that patch not ready for review? [02:17:24] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [02:17:38] yeah so it's not picked up that last merged change [02:17:40] I guess [02:18:54] hmmm [02:27:13] eileen: K I'm here for whatever is needed for this, not going anywhere for the next 3-4 hours... I do think we should do whatever needed to merge the queue consumer this evening, if I may say so? does that sound reasonable? [02:28:21] AndyRussG: OK - I'm testing that it might be that if mysql -uroot picked up the hostname from my.cnf locally it might work on both local & jenkins [02:28:49] actually maybe we don't even need that [02:30:21] eileen: k no worries, also no rush, again I'm here all evening [02:31:02] today is one of the week's kidless days for me, so no need to get up early, and I'm not as tired as on the kidful days [02:31:11] also I ordered food, so I didn't even cook! [02:33:28] nice [02:33:47] I'm just testing that syntax I pasted above to see if it works [02:34:21] okok cool [02:40:05] (just gonna be afk for about 10 minutes now, so brb) [03:02:08] (back) [03:23:17] (PS1) Eileen: Revert "Use drupal db for fredge" and set permissions more better [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684635 (https://phabricator.wikimedia.org/T281609) [03:23:26] AndyRussG: ^ is working for me locally [03:24:07] (PS1) Eileen: Update boilerplate code [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684636 (https://phabricator.wikimedia.org/T281647) [03:24:52] Also ^^ is the immediate issue on the monolog one - ie the reason for the log effort I think - but I also have another exception catching patch on that that is more robust [03:24:58] (to do as well) [03:27:20] (CR) jerkins-bot: [V: -1] Update boilerplate code [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684636 (https://phabricator.wikimedia.org/T281647) (owner: Eileen) [03:29:10] eileen: okok whee trying it out now eh [03:30:40] I'm confused at the eval and also why it's not the amp sql thing but I think my understanding of that is secondary heheh [03:34:09] AndyRussG: because copy & paste worked ..... [03:34:33] tbh the amp thing might work but I got copy & paste right & the amp thing wrong [03:35:32] eileen: works great eh! [03:35:37] fredge db created [03:35:46] and I didn't need to nuke the vendor directory [03:35:59] Yeah that vendor I still have more fixes on [03:36:11] it's not vendor I don't think - it's templates_c [03:36:19] monolog table is there [03:36:46] AndyRussG: I just attempted to log into this https://github.vmix.cc/eileenmcnaughton/au.org.greens.extendedmailingstats - do you think I got password harvested? [03:36:51] eileen: well this civi-drupal-buildkit machine is not one for simple solutions [03:36:59] :-) [03:37:39] eileen: yes, change your password asap [03:37:47] ok [03:38:42] and change the password on any other sites where you may use the same or very similar passwords [03:38:53] wow crazy stuff [03:41:17] yeah that's a crazy github clone phishing site [03:41:44] yeah creepy [03:42:10] if I had just gotten an ok-ish thing after entering it I wouldn't have realised [03:42:38] I think they actually briefly managed to reset my password! [03:42:46] eileen: oh no! [03:43:01] eileen: but you have it back now? [03:43:18] are you in the wmf group on GitHub? [03:43:18] yes I reset it [03:43:27] maybe - not sure [03:43:30] and the new one still works? [03:43:46] yeah so I think I failed to log in and then I reset it and the new one works [03:43:56] & there was 1 minute between events [03:44:05] actually I also got an email saying it had been reset [03:44:24] but the ip for the first reset is my own [03:44:32] so they did something browser-cunning [03:44:36] huh [03:44:51] oh right from your own browser! [03:44:53] wow [03:45:17] what is the email for security - I should advise [03:47:08] eileen: I think security-team@wikimedia.org [03:48:13] maybe also trustandsafety@wikimedia.org [03:59:22] ok emailed - that was a bit freaky tbh [04:00:04] yeah crazy stuff! [04:12:25] eileen: shall I merge the "Revert "Use drupal db for fredge" and set permissions more better" one? [04:12:39] AndyRussG: yep cos it's more better [04:12:52] you mean moar better? [04:12:58] okok yeah looks great [04:13:28] Oh I'm going to add some bug tags to the commit message first k? [04:13:55] oh yeah - please change to moar too [04:14:04] that's def moar better [04:14:12] ah okok [04:15:40] (PS2) AndyRussG: Revert "Use drupal db for fredge" and set permissions moar better [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684635 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [04:15:58] (CR) AndyRussG: [C: +2] Revert "Use drupal db for fredge" and set permissions moar better [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684635 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [04:16:02] (CR) AndyRussG: [V: +2 C: +2] Revert "Use drupal db for fredge" and set permissions moar better [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684635 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [04:19:07] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [04:19:17] ok let's see what that one does now [04:30:25] (PS1) Eileen: Catch and handle exceptions in Monolog [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684664 (https://phabricator.wikimedia.org/T281647) [04:31:51] AndyRussG: so that one I just added ^^ - I think would cause monolog to degrade gracefully - the issue I believe is that the deprecated functions fixed in 684636 were triggering the log() function & it was loading the outdated cached container - since monolog wasn't registered as yet [04:32:33] all of which says we can handle more errors in monolog and remove the deprecated functions and I logged https://github.com/civicrm/civicrm-buildkit/issues/615 about the caching [04:33:48] (CR) jerkins-bot: [V: -1] Catch and handle exceptions in Monolog [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684664 (https://phabricator.wikimedia.org/T281647) (owner: Eileen) [04:35:12] Fundraising-Backlog, FR-Docker, Patch-For-Review: [Bug] Docker dev setup: civibuild create wmff fails when wmff/vendor iexists - https://phabricator.wikimedia.org/T281647 (Eileenmcnaughton) Parts to this 1) the cause of the early attempt to do logging I believe is a deprecated function in old boiler... [04:35:37] AndyRussG: woot - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/684537 v+2 [04:42:04] eileen: cool! will the api one pass ci now? or does it need to be rebased on that one? ^ [04:42:10] congrats btw! [04:45:13] AndyRussG: I think we should merge that one & then all the rest should start passing [04:45:41] the changes only affect ci - ie not prod or our buildkit [04:45:47] or our local I mean [04:45:51] ok [04:46:33] I'm assuming that the deleted lines with ${CIVICRM_SCHEMA_PREFIX}${i} and such are currently unused stuff or something? [04:47:40] (CR) AndyRussG: [C: +2] "aaaaaahhhhh yay!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [04:48:04] I may not fully understand the patch, but at least I can cheer for it [04:49:27] lol [04:49:37] it gets rid of complexity :-) [04:49:44] and it's moar better [04:51:02] it was an unexpected snaffu but I think we are ahead now [05:01:48] (Merged) jenkins-bot: Attempt to unravel pre-create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684537 (https://phabricator.wikimedia.org/T281609) (owner: Eileen) [05:02:13] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [05:02:28] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684664 (https://phabricator.wikimedia.org/T281647) (owner: Eileen) [05:03:09] (CR) Eileen: [C: +2] "recheck" [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510 (owner: Eileen) [05:03:52] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/684636 (https://phabricator.wikimedia.org/T281647) (owner: Eileen) [05:06:41] (PS14) AndyRussG: E-mail pref ctr queue consumer using api [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/682349 (https://phabricator.wikimedia.org/T268511) (owner: Eileen) [05:07:49] ^ just rebased the queue consumer one [05:08:27] yep cool - I did a recheck on the others - not sure we want to deploy anything tonight now but can maybe get close [05:08:40] I'm just gonna get outside for a bit - back soon [05:17:20] (Merged) jenkins-bot: Vendor update, phpmailer, monolog handler [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/684510 (owner: Eileen) [05:21:30] eileen: k.... yeah it passed! https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/682349 [05:47:47] eileen: if you're about, would you like to kick this one again? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/682349 [05:48:14] (I guess add your own +2? maybe you'd have to remove cstone's +2? I can also just click "submit", I guess) [05:49:02] hmmm ok sorry to be such a bureaucrat... just gonna submit heheh [05:49:58] (Abandoned) AndyRussG: WMFF: Drop fredge db before creating it [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684138 (https://phabricator.wikimedia.org/T281609) (owner: AndyRussG) [05:50:12] (PS5) AndyRussG: WMFF: sync drupal users and set API key for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684003 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [05:50:41] (PS4) AndyRussG: WMFF: Static CIVI_SITE_KEY for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684121 (https://phabricator.wikimedia.org/T281651) [06:11:46] (Abandoned) Eileen: phpmailer update to 6 [wikimedia/fundraising/crm/vendor] - https://gerrit.wikimedia.org/r/683482 (owner: Eileen) [06:13:58] It's merged! [06:14:22] eileen: yes I pushed the "submit" button [06:14:50] eileen: thanks so much for all your work on this! [06:14:56] K it's sleepy sleepy time time [06:14:59] I think we need to get our buildkit in sync with upstream - there are changes related to civi-sites upstream that we don't have [06:15:02] night [06:15:12] eileen: yes agreed [06:15:24] cya! :) [06:17:53] night [06:21:59] (PS1) Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/684699 [07:35:40] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1253 [07:37:40] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1287 [07:40:40] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1486 [07:42:40] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 396709 Threads: 11 Questions: 9004996 Slow queries: 175 Opens: 80941703 Flush tables: 1 Open tables: 273 Queries per second avg: 22.699 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 547 [07:46:57] Fundraising-Backlog, Analytics-Radar, BetaFeatures, BlueSpice, and 46 others: Prepare User group methods for hard deprecation - https://phabricator.wikimedia.org/T275148 (Aklapper) [07:47:35] Fundraising-Backlog, Analytics-Radar, BetaFeatures, BlueSpice, and 46 others: Prepare User group methods for hard deprecation - https://phabricator.wikimedia.org/T275148 (Aklapper) Re-added codebase tags, so people interested in tickets about a codebase can find these tickets about that codebase. [07:55:40] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1233 [08:00:40] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 3333847 Threads: 17 Questions: 87094897 Slow queries: 2727188 Opens: 678205254 Flush tables: 1 Open tables: 200 Queries per second avg: 26.124 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [13:56:28] Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: follow-on Civiproxy setup - https://phabricator.wikimedia.org/T281651 (jgleeson) Hi @AndyRussG. Appreciate you writing some of this up for me and giving this your attention over the weekend. The first two tasks do feel like good impr... [14:03:13] (CR) Jgleeson: [C: -1] "Thanks for looking at this! After pulling in the latest patch on this chain and the linked latest on fundraising-dev, my local admin user " [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684006 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [14:04:53] (CR) AndyRussG: "> Patch Set 3: Code-Review-1" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684006 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [14:05:13] (CR) Jgleeson: [C: +1] "Thanks for looking at this! This worked for me as expected. Nice work!" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684122 (https://phabricator.wikimedia.org/T281651) (owner: AndyRussG) [14:14:43] Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: follow-on Civiproxy setup - https://phabricator.wikimedia.org/T281651 (AndyRussG) >>! In T281651#7057497, @jgleeson wrote: > Hi @AndyRussG. Appreciate you writing some of this up for me and giving this your attention over the weekend... [14:16:07] (CR) Jgleeson: [C: -1] "Yep. wmff is on 47a2ffeefeb64ccc179d1a7531a07b6c620afbb8" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684006 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [14:21:36] Fundraising-Backlog, FR-Docker, Patch-For-Review: Docker dev setup: follow-on Civiproxy setup - https://phabricator.wikimedia.org/T281651 (jgleeson) Yeah, agreed that it made sense to merge it as it was but I feel like the other follow-up tasks would have been better discussed and agreed on the patch... [14:41:31] AndyRussG: did the api key stuff work at your end? [14:47:41] jgleeson: yes currently working for me on the latest of stuffs... [14:48:06] note that buildkit source should also be at the latest patchsets of thos changes [14:48:09] 86d088e8852f8 for buildkit [14:48:27] also I removed the vendor directory under wmff [14:48:34] and reset persistent storage [14:49:08] ahhh though if it's not working for you it might be prudent to just set it manually again this one time [14:49:38] if I understand correctly, getting a version of e-mail pref center deployed is quite high priority today.... [15:04:19] oh man [15:04:23] I think I know what it was [15:05:10] looks like you changed docker-api-key to docker_api_key ha! [15:05:24] jgleeson: ahhh yes correct that was me heheheh sorry! [15:05:58] no worries. I just tried calling the api cli update and setting it to test and it worked so I was scratching my head [15:06:09] I also ran into an error where in one place it was underscore and the other place it was dashes [15:06:25] and set it to one, but I guess I didn't check which it was originally, sorry about that! [15:06:35] ahhh glad it worked then! [15:07:02] no worries. just thinking of a quick way to retest without rebuilding lol, as I've overwritten it to test [15:07:26] I'll update the patch and put it down for now based on the email-prefs deployment priority [15:08:25] :) thanks much! [15:10:26] (CR) Jgleeson: [C: +1] "removing -1 as I was looking for the wrong value, in the wrong database!!! When I realised I was looking at another install of civicrm and" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684006 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [15:25:01] AndyRussG: seems to be working for me too [15:31:04] ejegg: cool! :) :) [15:34:12] oh wait, last night it was, but now I see an error.. hmm [15:34:27] lemme get debugging running on the civiproxy bit [15:35:55] ejegg: ^ update the latest crm (wmff) master, and also the latest patch set of last buildkit Gerrit change on the chain , and the same for fr-dev, and reset persistent storage, and remove wmff/vendor [15:36:10] sorry it's a bit involed [15:37:30] did they change since last night? [15:37:46] I rebuilt around 7 local time last night with the then-latest of everything [15:37:56] ejegg yes they changed since then :) [15:37:59] ok [15:38:00] all of the stuff [15:38:55] ejegg also just to bring u up to speed, apologies don't mean to tell u what to work on, yesterday it was emphasized that we might all pile onto getting e-p-c deployed [15:39:03] though I'm not sure what more we should do at this point [15:39:06] yep yep, i figured as much [15:39:15] I guess maybe deploying civi since the consumer patch is now merged? [15:39:18] well, all of us having it working locally seems good [15:39:29] heheh yes also important indeed ;p [15:39:33] right, the civi deploy [15:39:44] was the manual steps on civiproxy blocking you both? curious why that got reprioritised after the standup on friday? [15:39:47] also in about 10 min I should run to the bank, ahh apologies, won't take long [15:39:48] and then the localsettings for the new wiki [15:39:52] AndyRussG: ejegg ^ [15:40:31] jgleeson: it wasn't officially reprioritized at all, there was just... uh.... a rabbit.... [15:40:43] jgleeson: oh, on friday i just had a bolt from the blue occur to me that the auto syncing of contact records should be possible with the uf_match [15:40:44] (though also there wasn't much that could be done on e-p-c over the weekend) [15:41:19] and once I found that, I figured i might as well finish the job with setting the api key in a scripted way too [15:42:02] ejegg: yeah hugely appreciate u finding that eh! [15:42:29] cool [15:46:16] ok wrt the email preferences deployment... are we waiting on something to serve the code? https://phabricator.wikimedia.org/T281321 [15:46:52] I think that may still be a missing piece yeah [15:47:01] Jeff_Green: dwisehaupt any update on the server for civiproxy? [15:47:08] https://etherpad.wikimedia.org/p/emailpreference [15:47:24] jgleeson: working on it, there are a lot of moving parts [15:47:33] gotcha [15:47:43] sorry i was out yesterday so just wanted to check in [15:48:39] where we are right now--I think we've got clarity on what we need as far as redis service and roughly what the smashpig config should look like [15:48:57] we confirmed that civiproxy can hit a non-ssl version of civi and I think the frpm1001 job is done [15:49:08] ah thanks Jeff_Green [15:49:32] so right now I'm working on all the peripheral stuff to configure another redis instance and all the peripheral stuff around that, monitoring etc [15:49:54] jgleeson: great [15:51:02] are we separating out the redis instances for security ? [15:51:05] fr-tech^ [15:51:07] yes [15:51:15] fundraising-tech-ops, fr-email-preference-center: Add Apache/Nginx config to serve Civiproxy - https://phabricator.wikimedia.org/T281321 (Ejegg) [15:52:30] I'm trying to imagine the attack [15:52:39] from the email pref centre wiki [15:53:11] fwiw there's also privatebin running on that server [15:53:33] jgleeson: also maybe to limit PCI scope? [15:53:44] since the other redis instance talks to the payments servers [15:53:55] ah was that raised? [15:54:02] just speculating [15:54:10] I wasn't aware of the separate redis instance [15:54:20] i've only been popping in momentarily since thursday [15:54:31] got it [15:55:56] jgleeson: the theoretical attack issue is that if there were an issue with email prefs that led to someone to being able to query redis, we don't want them to be able to query the keys that contain payments information. user ACL separation doesn't come until redis 6 which we aren't currently running. [16:04:42] fr-tech K heading out to the bank, apologies, back in about 1hr [16:06:33] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Unsubscribe CIDs with 'zero net' donations - https://phabricator.wikimedia.org/T273843 (krobinson) Closing this out as @KHaggard did some investigating and found out that: All refunds in Civi are handled correctly in Acoust... [16:06:56] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fr-donorservices: Civi: Unsubscribe CIDs with 'zero net' donations - https://phabricator.wikimedia.org/T273843 (krobinson) Open→Resolved [16:20:30] Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, FR-Ingenico, Recurring-Donations: Failed recurring donation attempts making it to status 600 in Ingenico - https://phabricator.wikimedia.org/T281091 (EMart... [16:22:09] Fundraising Sprint Humongous bacteria petting zoo, Fundraising Sprint Interstitial ads halfway down the coaster hill, Fundraising-Backlog, FR-Ingenico, Recurring-Donations: Failed recurring donation attempts making it to status 600 in Ingenico - https://phabricator.wikimedia.org/T281091 (DStri... [16:22:16] Fundraising Sprint Humongous bacteria petting zoo, Fundraising-Backlog, FR-Ingenico, Recurring-Donations: Failed recurring donation attempts making it to status 600 in Ingenico - https://phabricator.wikimedia.org/T281091 (DStrine) [16:49:58] fr-tech I posted the work in progress smashpig config template and result for each role to frpm1001:/tmp/smashpig_puppet_new [16:52:16] as far as I can tell, apart from renaming the section "redis-servers" to more general "services" (which afaik doesn't matter except for readability) the new looks to me like it's producing the same yaml data, there's a silly python test script in the top level that dumps the data structures for comparison [17:02:25] looking Jeff_Green [17:03:35] fundraising-tech-ops, fr-email-preference-center: Add Apache/Nginx config to serve Civiproxy - https://phabricator.wikimedia.org/T281321 (Dwisehaupt) [17:06:17] Jeff_Green: only the ruby script is readable [17:06:26] oh whoops, fixing [17:07:12] fixed [17:07:50] fundraising-tech-ops, fr-email-preference-center: Add Apache/Nginx config to serve Civiproxy - https://phabricator.wikimedia.org/T281321 (Dwisehaupt) I have pushed out the apache and nginx configs. nginx is serving on port 442 with the puppet ssl certs and forwarding to a low proxy port for apache. Still... [17:09:25] jgleeson: fwiw it's easy to deploy the new-format conf to frdata* without touching the template/format for the other servers, it just crufts up puppet a little [17:10:50] yeah that looks good to me [17:11:11] cool [17:11:44] (back) [17:12:19] AndyRussG: see ^^^ backscroll re. work in progress for puppetized smashpig config [17:14:40] Jeff_Green: ahh cool thx! looking... [17:17:45] (CR) Ejegg: [C: +2] "Works for me!" [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684003 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [17:18:20] (CR) Ejegg: [V: +2 C: +2] "Looks good, works fine!" [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684121 (https://phabricator.wikimedia.org/T281651) (owner: AndyRussG) [17:18:24] (CR) Ejegg: [V: +2 C: +2] WMFF: sync drupal users and set API key for Docker dev setup [wikimedia/fundraising/crm/civicrm-buildkit] - https://gerrit.wikimedia.org/r/684003 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [17:20:35] (CR) Ejegg: [V: +2 C: +2] "Looks good, works for me locally!" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684006 (https://phabricator.wikimedia.org/T281651) (owner: Ejegg) [17:21:14] (CR) Ejegg: [V: +2 C: +2] "Looks good, matches other key, works for me locally!" [wikimedia/fundraising/dev] - https://gerrit.wikimedia.org/r/684122 (https://phabricator.wikimedia.org/T281651) (owner: AndyRussG) [17:28:33] AndyRussG: when loading trying to load up email-prefs wiki I see this https://phabricator.wikimedia.org/F34440480 am I missing some config maybe? [17:28:51] jgleeson: the link needs more params [17:28:56] here's a better one: [17:29:05] https://localhost:9002/index.php/Special:EmailPreferences/emailPreferences?contact_hash=1027858961&contact_id=188 [17:29:29] thanks! [17:29:31] I /think/ that sample data hash and id are consistent across reinstalls [17:30:21] then you can change the contact's preferred language in civi manually if you want: https://wmff.localhost:32353/civicrm/contact/add?reset=1&action=update&cid=188 [17:30:37] (need to expand the 'Communication Preferences' group 3rd from bottom) [17:31:05] core data, not to be confused with the 'Communication' custom data group [17:31:47] oh hey, we should set opt-in source in the prefs queue consumer when it's not already set [17:36:50] Jeff_Green: I can't recall seeing the email-preferences smashpig queue. Was there a queue called that in one of those files? [17:37:04] in the new/old config [17:37:07] jgleeson I think we just added that [17:37:17] oh, lemme look at the new version [17:37:32] yeah ejegg, Jeff_Green was asking if the config he put up with the new dynamically generated additions looked good [17:38:24] oh hey it's in frdata [17:38:30] ok yeah, the old config has email-preferences with just the <<: *REDIS backref [17:38:31] in the new dir [17:39:20] jgleeson: ah good point, it shows up only for frdata, b/c haven't added it the civicrm role manifest yet [17:39:34] oh yah, civi will need that too [17:39:50] yeah, easy to add, I just spaced it [17:40:00] coolio [17:40:06] does this format look ok otherwise? [17:40:12] does to me [17:40:46] yep, looks fine to me [17:41:15] any concern about rolling it out? or should I just go for it [17:41:15] seems to have a couple stray newlines, but nothing that affects the actual interpretation of the yaml [17:41:38] maybe we stop the queue consumers first? [17:41:43] and do a slow start [17:41:50] after deploy [17:41:52] where re. the newlines? [17:42:00] sounds good ejegg [17:42:38] Jeff_Green: never mind about the newlines [17:42:56] just the one right after the data-store: key caught my eye [17:43:16] but srsly no need to change [17:43:43] ok [17:44:08] Jeff_Green: so the frdata one has no provider-configuration-directory key [17:44:27] which means it will look for e.g. logging config in /etc/smashpig/provider-defaults.yaml [17:44:30] (CR) DannyS712: [C: +2] build: Updating composer dependencies [extensions/FundraisingTranslateWorkflow] - https://gerrit.wikimedia.org/r/685004 (owner: Libraryupgrader) [17:44:55] oh, right, there was confusion about that yesterday [17:45:00] should it have that? [17:45:14] and if that file's not there it will use these defaults: https://phabricator.wikimedia.org/diffusion/WFSP/browse/master/config/provider-defaults.yaml [17:45:19] it would be better to have that [17:45:24] ok, fixing [17:45:41] so it can send failmail if necessary [17:45:53] yep! [17:47:01] so there are two possibilities - either just deploy provider-defaults.yaml to /etc/smashpig on the frdata host [17:48:34] or add that provider-configuration-directory key to main.yaml pointing to the /srv/ subdir and then make some more changes in the SmashPig f_c_u project so that frdata only gets provider-defaults from that folder, and none of the processor subdirs [17:48:58] maybe putting provider-defaults.yaml in /etc/smashpig on frdata is easier? [17:49:47]