[00:03:36] (PS1) Eileen: Remove KAM, update civitutorial status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501454 (https://phabricator.wikimedia.org/T218754) [00:04:10] OK this one spun out too - it should be straight forward https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501454/ [00:05:28] (PS1) Eileen: Enotice fixes to support updates to contact editor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501455 (https://phabricator.wikimedia.org/T218754) [00:05:47] (Abandoned) Eileen: Enotice fixes to support updates to contact editor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500870 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [00:07:56] (PS1) Eileen: Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) [00:08:16] (Abandoned) Eileen: Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500872 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [00:08:58] ok that should be better now - the ones that are not intertwined should be the ones that are passing now [00:12:15] (CR) jerkins-bot: [V: -1] Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [00:17:27] (CR) Ejegg: [C: +1] "Looks like a fine change, just a couple questions about potential cleanup" (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499687 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:18:09] ok ejegg I got the tests for specialpages to pass on devwki with the fundraisingunsubscribe extention ! [00:19:47] (PS3) Ejegg: Update query to not treat blank location types as duplicate [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499688 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:19:49] (CR) Jgleeson: [C: +2] "tests passing on mediawiki 1.32 with extension installed" [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/501433 (owner: Ejegg) [00:20:09] (CR) Ejegg: [C: +2] "Nice trick!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499688 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:21:19] jgleeson: awesome! [00:21:36] heh, for those core tests I've always just relied on CI [00:22:19] jgleeson: it'll need the parent patch +2ed as well to merge [00:22:39] oh sorry, that's the one you just +2ed [00:22:43] ah I thought I'd gotten them all? [00:22:44] thanks again! [00:22:47] np! [00:25:08] (CR) Eileen: Update primary location fix to also fix when more than one option (2 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499687 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:25:12] (Merged) jenkins-bot: Fix deprecated functions [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/501433 (owner: Ejegg) [00:25:14] (Merged) jenkins-bot: Fancy variant for opt-in page [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/498287 (https://phabricator.wikimedia.org/T218240) (owner: Ejegg) [00:25:23] (Merged) jenkins-bot: Update query to not treat blank location types as duplicate [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499688 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:27:36] (CR) jenkins-bot: Fix deprecated functions [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/501433 (owner: Ejegg) [00:28:07] (PS3) Eileen: Update primary location fix to also fix when more than one option [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499687 (https://phabricator.wikimedia.org/T218616) [00:28:27] (CR) jenkins-bot: Fancy variant for opt-in page [extensions/FundraisingEmailUnsubscribe] - https://gerrit.wikimedia.org/r/498287 (https://phabricator.wikimedia.org/T218240) (owner: Ejegg) [00:31:27] (PS4) Eileen: Update primary location fix to also fix when more than one option [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499687 (https://phabricator.wikimedia.org/T218616) [00:31:28] (PS3) Eileen: Add a check to fix blank location types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499689 (https://phabricator.wikimedia.org/T218616) [00:37:25] (CR) Ejegg: [C: +2] "Thanks! Just found the place that BaseLocation was used - sorry for the nitpick!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499687 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:41:44] (Merged) jenkins-bot: Update primary location fix to also fix when more than one option [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499687 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [00:43:29] I think I’ve figured out the test fails on the upgrade & why they don’t hurt locally https://github.com/civicrm/civicrm-core/pull/13962/files [00:45:30] see you all tomorrow fr-tech! [00:51:34] eileen: why do we prefer assigning 'Other' instead of 'Main' if there's no 'Main' phone number? [00:52:11] ejegg: it was a bit random since we don’t have a clear standard for ‘weird shit' [00:52:29] I thought ‘Other’ was closer to ‘weird who knows’ than main [00:52:37] hmm, ok [00:52:48] I can flip it if you think I should [00:53:23] When we create a contact from scratch with a single address, it's tagged as 'Main', right? [00:53:39] Yeah I guess so [00:53:54] ehh, any old value is better than blank [00:54:40] but it seems nice to assign 'Main' if there is no 'Main' [00:55:40] ejegg: I’ll flip it [00:55:52] thanks! [00:56:52] done [00:56:57] (PS4) Eileen: Add a check to fix blank location types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499689 (https://phabricator.wikimedia.org/T218616) [00:58:00] (sorry) the fn comment still has the previous order... [00:59:05] ooh, just looking at the github pull request. So core's now statically caching the processor list? [00:59:13] I guess that's usually safe [01:16:47] ejegg: yeah - I’m gonna get that change included in our code [01:17:14] as in the ‘stock’ part is still being finalised based on what I get merged before the branching happens [01:18:47] ejegg: comment fixed [01:18:47] (PS5) Eileen: Add a check to fix blank location types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499689 (https://phabricator.wikimedia.org/T218616) [01:19:19] ejegg: re that static caching I *think* we should be pushing more use of the Civi::cache() construct in core now but need to still clarify a bit with Tim [01:21:47] :) [01:22:12] ok, right, with the multiple tiers all accessed via the same interface [01:22:22] yeah, that'd be great to see [01:23:59] (CR) Ejegg: [C: +2] "Nice! If that SQL gets more complicated in the future, it might be better to do the type assignment in a PHP loop, but this I'm comfortabl" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499689 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [01:24:48] (CR) Eileen: "Ah yeah - I'm a bit too keen on array functions for re-ordering perhaps ...." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499689 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [01:26:01] ejegg: so there is a page here https://docs.civicrm.org/dev/en/latest/framework/cache/ [01:26:17] but this stuff is a bit more of a conversation https://github.com/civicrm/civicrm-dev-docs/issues/549 [01:26:57] if you DO come to the sprint in NYC we can corner Tim & discuss over appropriate drinks [01:28:05] eileen: shoot, there's an Argentinian village my wife really wants to visit just then [01:28:14] (Merged) jenkins-bot: Add a check to fix blank location types [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/499689 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [01:28:22] ejegg: I can see the appeal [01:28:22] Are you going to the sprint in Barcelona in October? [01:28:27] I hope to! [01:28:36] I'd like to check that one onut [01:28:39] *out [01:28:47] When we get asked for travel requests for budget that will be the one I ask for [01:30:28] I think Jack wants to go too [01:38:35] closer for him! [01:44:32] eileen: for that patch adding the test processors, would there be any need to append or prepend 'Test' to the name? [01:44:46] Or do they never appear in the same dropdowns? [01:44:59] ejegg: no - the civi convention is matching names for test & non test [01:45:02] I want to CHANGE that [01:45:10] & have added a title field to the table in this update [01:45:12] ah, ok [01:45:18] but we would be ahead of the game to change it now [01:45:20] hmm, name and title? [01:45:29] (PS2) Ejegg: Add test processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501453 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [01:45:30] yeah that’s consistent with other places [01:45:43] (CR) Ejegg: [C: +2] Add test processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501453 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [01:45:54] But also the name should be name_test IMHO [01:46:02] but some work to do to get there [01:47:43] ok, looking at the enotice fixes bit now [01:47:57] I guess _tabset is only ever called in the context of a single contact_id? [01:48:45] tbh I can’t fully remember that one - it was already merged into extended report & seemed relevant [01:48:50] oh, or... extended reports only should add tabs [01:48:54] I think it is called on event pages [01:48:56] in the context of a single contact_id [01:49:03] right the latter [01:49:20] so when we are looking at an event page I think I was getting an enotice [01:49:56] I’m just trying to convince Tim to accept a core change that will reduce our upgrade pain [01:50:10] (Merged) jenkins-bot: Add test processors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501453 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [01:50:12] 3 [01:50:13] https://github.com/civicrm/civicrm-core/pull/13551 [01:50:47] that would stop it from always failing to upgrade because it’s not listening to us [01:51:00] ooh [01:51:04] about not addin triggers on the mailing_provider table [01:51:18] Tim’s last comment “ok, gonna walk the dog while i try to find a way to articulate my reaction and a way to (in)validate a theory about it" [01:51:40] His dog is a chihuahua & that was over an hour ago so I suspect she’s either very tired now [01:52:08] or is now riding in his backpack [01:52:51] haha [01:54:59] (PS2) Ejegg: Enotice fixes to support updates to contact editor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501455 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [01:55:14] (CR) Ejegg: [C: +2] Enotice fixes to support updates to contact editor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501455 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [01:56:40] eileen: ok, shall we deploy that bundle? [01:56:54] Or I guess tack on the APIv4 update, since those were mostly test fixes? [01:57:05] ejegg: yeah the only other one in the ‘not much change really’ set is this https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501454/ [01:57:23] & then maybe we deploy that [01:57:49] & then if you get a chance tomorrow we can maybe merge the apiv4+ CLE with the goal that I deploy that on my Monday [01:57:49] hmm, donations queue fail [01:58:01] CLE ? [01:58:31] ooh, it's another one of those 'value already exists in database' things [01:58:31] Contact Layout editor [01:58:43] oh right [01:58:44] These are the fails from 19 minutes ago? [01:59:09] (we haven’t made any changes yet have we - is it the language again?) [01:59:39] (Merged) jenkins-bot: Enotice fixes to support updates to contact editor [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501455 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [01:59:40] yeah, that one [01:59:45] same language code, I think [01:59:51] I think we don’t have a Phab yet? [01:59:58] wonder if it's got some weird chars prepended or appended? [02:00:07] So here is what I think happens [02:00:22] we get some weird ‘let’s just create this’ thing & create it [02:00:23] so that 'remove KAM' patch - won't it mess with people's menus till we upgrade? [02:00:39] oh, it's currently disabled? [02:00:44] 2 seconds later we get another & it’s not in our statics so we try to create it … again [02:00:49] yep - disabled [02:01:01] it is in core now - no extension anymore [02:01:07] ah drat, I need to upgrade my cert [02:01:25] but I comfort myself that the core version has fixes to the things we found while it WAS an extension [02:01:29] oh, it's in the version of core we have deployed, i thought you meant it came in with the upgrade! [02:01:42] no - it’s in the core to deploy [02:01:50] but we disabled in on our live [02:01:57] ok, got it [02:02:15] yep - just re-checking - definitely disabled [02:03:26] I won’t run any of those data fixes until Monday - just so my focus is fully there when I do it [02:03:34] but we can deploy those now [02:05:55] cool [02:06:12] argh, and we don't even have that message in the damaged table :( [02:06:36] darn atomic queue operation isn't doing its job [02:08:47] wait, has 'Extensions' always been under 'Customize data and screens' ? [02:08:58] I really want a 'search all menu items' extension [02:09:10] Guess I should make one! [02:11:41] (PS2) Ejegg: Remove KAM, update civitutorial status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501454 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [02:11:49] (CR) Ejegg: [C: +2] Remove KAM, update civitutorial status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501454 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [02:12:58] Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising-Backlog, Patch-For-Review: Create variants for fancy new opt in pages on payments wiki - https://phabricator.wikimedia.org/T218240 (Ejegg) This is ready to deploy as soon as we get MediaWiki 1.31 back on paym... [02:13:02] lol [02:13:21] I think some things are in different places on live than local for legacy reasons [02:13:44] fun :P [02:15:56] ejegg: is php version now updated on live? [02:16:18] (Merged) jenkins-bot: Remove KAM, update civitutorial status [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501454 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [02:18:48] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501463 [02:19:04] eileen: not on the civi box [02:19:05] wow that looks like a lot of change [02:19:21] dang when do we update? [02:19:32] https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/501463/-1..1 [02:20:02] still not so much change in actual fact [02:20:35] no, mostly tests and KAM [02:20:49] yeah only real change is the processors in the install [02:21:00] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501463 (owner: Eileen) [02:21:04] actually, mind zapping those datachecks tests on the deploy branch just to keep consistent? [02:21:39] oh - I have a feeling there are quite a few extesnsions in that boat [02:22:30] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501463 (owner: Eileen) [02:22:35] ok done [02:22:36] (PS1) Eileen: Remove tests from deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501464 [02:22:55] (CR) Ejegg: [C: +2] Remove tests from deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501464 (owner: Eileen) [02:23:37] (Merged) jenkins-bot: Remove tests from deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501464 (owner: Eileen) [02:23:46] ejegg: so this is where I’m not sure about that deployment conversation - are people wanting to change deploys on things like this [02:23:55] or just bigger things [02:25:34] looks like omnimail repair finished again but I think that I need to change time & re-run [02:25:59] i'm not so sure we need a change either [02:26:08] or at least not regarding the schedule [02:26:18] I feel like 'no friday deploys' is good enough [02:26:29] and we already do try to put the big stuff early in the week [02:26:34] I guess we should have done a debrief on the civi probs because I think we don’t all have the same perception on it [02:27:23] !log update civicrm revision changed from 7560af93df to 3c55850631, config revision is 9ad5ef3e15 [02:27:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:28:30] Where I don’t want to be is where I hear other orgs being - really scared to update civi because they have ‘really complex extensions & customisation’ [02:29:04] I want to be the opposite - find ways to feel really good about updating because we do things really cleanly & have lots of tests [02:29:41] ok roll out looks good - pages load [02:31:00] Doing some taxi-service now but will take laptop & check in from kids swimming [02:31:08] thanks! [02:36:10] actually not leaving yet now [02:36:26] decided to flag luke’s class & see if the teacher will let him join Jack’s [02:37:33] ejegg: so when you next get a chance just review the apiv4 & contact layout editor ones & I’ll try to deploy those my monday & get that potential for mis-matches out of the way [02:37:56] sure thing eileen, I'll take a look at those tomorrow [02:38:11] they are just from upstream [02:38:25] & the are both only used on the contact summary page [02:38:48] it’s just the versions need to be in sync [03:01:29] cool cool. [03:01:43] Have a good weekend eileen! [12:39:03] fundraising-tech-ops: EPIC: migrate fundraising hosts to Debian Stretch - https://phabricator.wikimedia.org/T185013 (Jgreen) [12:39:05] fundraising-tech-ops: upgrade frauth servers to debian stretch - https://phabricator.wikimedia.org/T220132 (Jgreen) Open→Resolved This is done. For the record the standard dist-upgrade did not go smoothly in testing or on the production machine, apt gets stuck on upgrading freeradius. I was able to w... [13:36:30] hi, something broke in puppet on fr hosts [13:36:51] frdev1001, frpig1001, frqueue1001/1002, payments1001, payments1004 [13:37:15] PROBLEM - check_puppetrun on frqueue1001 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 5 minutes ago with 12 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_frtech_mail],File[/usr/lib/nagios/plugins/check_fundraising_jobs],File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_puppetrun] [13:38:15] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:38:17] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 6 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/usr/local/bin/aide_new_db],File[/usr/local/bin/risky_package_updater],File[/etc/apparmor.d/local/usr.bin.freshclam],File[/usr/lib/nagios/plugins/check_frtech_mail] [13:38:19] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 6 minutes ago with 15 failures. Failed resources (up to 3 shown) [13:38:25] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 6 minutes ago with 15 failures. Failed resources (up to 3 shown): File[/usr/local/bin/aide_new_db],File[/usr/local/bin/risky_package_updater],File[/etc/apparmor.d/local/usr.bin.freshclam],File[/etc/init.d/mysql] [13:40:15] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 8 minutes ago with 3 failures. Failed resources (up to 3 shown) [13:41:25] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 9 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:42:09] PROBLEM - check_puppetrun on frqueue1001 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 10 minutes ago with 12 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_frtech_mail],File[/usr/lib/nagios/plugins/check_fundraising_jobs],File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_puppetrun] [13:43:09] PROBLEM - check_puppetrun on frqueue1002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 11 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:43:15] PROBLEM - check_puppetrun on frpig1001 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 11 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/usr/local/bin/aide_new_db],File[/usr/local/bin/risky_package_updater],File[/etc/apparmor.d/local/usr.bin.freshclam],File[/usr/lib/nagios/plugins/check_frtech_mail] [13:43:19] PROBLEM - check_puppetrun on payments1001 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 11 minutes ago with 15 failures. Failed resources (up to 3 shown) [13:43:20] mutante: yeah that was me. I switched their puppetmaster in config, so they all restarted, which for some reason tends to come with a alertstorm [13:43:25] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 11 minutes ago with 15 failures. Failed resources (up to 3 shown): File[/usr/local/bin/aide_new_db],File[/usr/local/bin/risky_package_updater],File[/etc/apparmor.d/local/usr.bin.freshclam],File[/etc/init.d/mysql] [13:43:59] Jeff_Green: ack !:) [13:44:24] it would be nice to have a group-ack feature :-( [13:45:15] PROBLEM - check_puppetrun on frdev1001 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 13 minutes ago with 3 failures. Failed resources (up to 3 shown) [13:46:19] PROBLEM - check_puppetrun on payments1003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 14 minutes ago with 6 failures. Failed resources (up to 3 shown) [13:47:09] PROBLEM - check_puppetrun on frqueue1001 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 15 minutes ago with 12 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_frtech_mail],File[/usr/lib/nagios/plugins/check_fundraising_jobs],File[/usr/lib/nagios/plugins/check_impression_logs],File[/usr/lib/nagios/plugins/check_puppetrun] [13:48:09] RECOVERY - check_puppetrun on frqueue1002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:48:15] RECOVERY - check_puppetrun on frpig1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:48:19] RECOVERY - check_puppetrun on payments1001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:48:25] RECOVERY - check_puppetrun on payments1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:50:15] RECOVERY - check_puppetrun on frdev1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:51:19] RECOVERY - check_puppetrun on payments1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:52:15] RECOVERY - check_puppetrun on frqueue1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [13:55:20] if you have a search term that finds them all .. you do [13:55:28] icinga offers a checkbox that selects them all [13:55:53] or by putting them in a hostgroup [14:55:06] (PS1) Ejegg: Filter out test processor with same name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501588 [14:55:15] fr-tech I think this should do it ^^^^ [14:56:31] turning off recurring QC for now [14:58:49] cool will check it out [15:00:32] (CR) Jgleeson: [C: +2] "Looks good to me!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501588 (owner: Ejegg) [15:00:51] thanks jgleeson ! [15:01:28] !log disabled recurring donation queue consumer [15:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:05:04] (Merged) jenkins-bot: Filter out test processor with same name [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501588 (owner: Ejegg) [15:06:29] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501592 [15:06:45] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501592 (owner: Ejegg) [15:07:26] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/501592 (owner: Ejegg) [15:08:55] !log upgraded fundraising CiviCRM from 3c55850631 to 83478013a8 [15:08:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:09:44] OK fr-tech, who knows how to recover from this situation? [15:10:50] guessing with switch the recurring queue consumer back on in the drupal system settings? [15:11:15] s/with/we/ [15:11:34] jgleeson: ah, I turned off the drush job via process-control. [15:11:39] but there are other things to do! [15:11:51] So, what happened to all the recurring donations that came in overnight? [15:12:22] got stuck because the api call kept finding two payment processors? [15:12:38] and ended up where? [15:12:56] damaged queue I'm guessing [15:13:01] let me jump on the box [15:13:01] yep! [15:13:13] so how to get them back into the recurring queue? [15:14:22] I think there's somewhere in drupal we can retry them but I can't remember exactly where. If left to my own devices I'd probably write something to insert the messages from damaged to recurring in the cli [15:15:04] actually I have it bookmarked I think [15:15:17] https://civicrm.wikimedia.org/damaged [15:15:39] yeah I'm guessing we can resend from there [15:15:48] once the job is back on [15:16:13] we can, but unfortunately we never made the batch functions work :( [15:16:18] how did you stop the job in process-control? comment out the lines that make the drush call? [15:16:23] so in that interface we'd be doing it one by one [15:16:24] curious about that [15:16:34] jgleeson: no, I commented out the schedule line [15:16:39] ah [15:18:33] ejegg, is the answer to do with one of our maintenance scripts? [15:19:04] * jgleeson clutching at straws at this point [15:19:07] (PS1) Umherirrender: Require MediaWiki 1.31 in extension.json for AutoloadNamespaces [extensions/FundraisingTranslateWorkflow] - https://gerrit.wikimedia.org/r/501598 [15:19:21] jgleeson: that would be nice to have! [15:19:38] sadly right now all we have is SQL [15:21:33] ok I wouldn't of guess that [15:21:36] guessed [15:23:18] I'm curious how we get the donations from the damaged tbl to the recurring redis queue ejegg [15:23:23] nicely [15:23:39] jgleeson: there's a field 'retry_date' on that table [15:23:47] ahhhhhhh [15:23:51] which we usually set when we first add stuff [15:24:29] so then this job called requeue_delayed_messages looks at all the messages with a retry date [15:24:34] and requeues all the ones due up. [15:24:52] So the trick is to figure out a SQL update statement just for the messages that we wnat to requeue [15:24:57] give them all a retry_date of now [15:25:02] and then run that job [15:27:17] ejegg, it looks like all the affected donations are ingenico [15:27:40] yep, that's the only integration we're storing tokens for now [15:28:30] so something like `UPDATE damaged SET retry_date=NOW() WHERE gateway='ingenico'` [15:28:49] might want a date filter on that too [15:28:56] oh good point [15:36:25] ah man, trying to get civitests working on vagrant is proving a challenge [15:36:28] oops, those were mostly the donations queue, not the recurring queue [15:36:33] keep going down blind alleys [15:58:28] !log re-enabled recurring donations queue consumer [15:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:08:22] well I just discovered `drush watchdog-show --tail` and it's pretty handy [16:14:23] actually it's not that useful because it doesn't seem to display all watchdog calls [16:31:53] coming fr-tech! [16:53:54] AndyRussG: to answer your question, I don't think we have a good reset button for civi DBs [16:54:25] besides the build scripts we use for CI, to start all over from scratch [16:54:52] At one point I had some scripts to restore a db I was using just for unit tests [16:54:58] but that backup is way out of date now [17:04:11] (PS26) Jgleeson: Create queue consumer for contribution tracking messages and add drush command to call it. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/498413 [17:06:41] ejegg, I've separated out the schema change and push the patch up ready for a next round of review. I think I updated all the items mentioning in previous feedback except the usage the db_merge, which i couldn't work out how to use so I will look at that next week, along with a clean install of vagrant to see if it makes my tests run any better [17:11:09] great, thanks! [17:11:20] will do a thorough review shortly [17:16:49] have a great weekend fr-tech! [17:16:51] bye for now [17:54:11] fundraising-tech-ops, Patch-For-Review: Revoke production prometheus fundraising access - https://phabricator.wikimedia.org/T217355 (cwdent) [17:54:14] fundraising-tech-ops: upgrade pay-lvs servers to stretch - https://phabricator.wikimedia.org/T220100 (cwdent) [18:39:03] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising-Backlog, and 2 others: Send message to opt-in queue on failed payments attempts with opt_in=1 - https://phabricator.wikimedia.org/T216293 (CCogdill_WMF) @DStrine can w... [18:50:02] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising-Backlog, and 2 others: Send message to opt-in queue on failed payments attempts with opt_in=1 - https://phabricator.wikimedia.org/T216293 (Ejegg) Hi @CCogdill_WMF. Whe... [18:52:15] Fundraising Sprint Ewoks Take Manhattan, Fundraising Sprint Fistful of $variables, Fundraising Sprint Greps of Wrath, Fundraising-Backlog, and 2 others: Send message to opt-in queue on failed payments attempts with opt_in=1 - https://phabricator.wikimedia.org/T216293 (CCogdill_WMF) Ahh awesome, t... [18:59:29] (CR) Ejegg: [C: -1] "Looking better, but unfortunately we're not getting good order_ids when the switch is on. The order_id is in most gateways derived from th" (2 comments) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/496351 (https://phabricator.wikimedia.org/T215462) (owner: XenoRyet) [19:09:31] XenoRyet: if you can disassemble that mess of side-effects while you're at it, or in a preliminary patch, I'd love to help! [19:20:05] fr-tech argh, looks like removing auto_increment from contribution_tracking.id might take a long time after all: https://jira.mariadb.org/browse/MDEV-12836 [19:20:39] https://www.percona.com/blog/2007/10/29/hacking-to-make-alter-table-online-for-certain-changes/ [19:21:23] horrible hack to do it quick, probably breaks replication ^^^ [19:29:26] cwd / Jeff_Green how long do you guys think an alter table on contribution_tracking will take? [19:29:51] no idea, we can test it [19:30:27] on the dev_ database? [19:30:47] hmm, probably should wait till we've got code that can handle it tho [19:30:58] yeah i'd rather make a throwaway copy [19:31:06] that one's in use for staging right? [19:31:51] yep [19:33:24] Jeff_Green and cwd: so, it sounds like 3DS redirects are working on payments.frdev [19:33:45] The experience of having them in an iframe is crappy, but it's just the same as current payments. [19:34:15] We should be able to swap back over to 1.31, then work on a solid, well-tested way to do full redirects for 3DS [19:34:57] sounds good [19:35:23] Jeff_Green: want to do that monday? [19:36:05] i think the 10 minutes downtime was pretty good [19:43:41] cwd: sure [19:44:40] :) [20:09:14] i would not be surprised if this is not the last thing of this type we see [20:10:08] actually scratch that, with the permissive header i guess i would be surprised [20:11:53] (PS1) Umherirrender: Add phan [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/501708 [20:13:01] (CR) Umherirrender: "check experimental" [extensions/FundraiserLandingPage] - https://gerrit.wikimedia.org/r/501708 (owner: Umherirrender) [20:50:49] (PS1) Umherirrender: Add missing use statement for Database [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/501788 [20:53:12] (CR) Umherirrender: Add missing use statement for Database (1 comment) [extensions/ContributionTracking] - https://gerrit.wikimedia.org/r/501788 (owner: Umherirrender) [22:09:16] (CR) Ejegg: [C: +2] "Looks good, seems to work well with new contact layout editor" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/500871 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:14:23] (Merged) jenkins-bot: Update apiv4 to v 4.3.0 as ships with latest civi-core [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/500871 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:15:13] (PS2) Ejegg: Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:20:27] (CR) jerkins-bot: [V: -1] Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:26:53] (PS3) Ejegg: Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:47:25] (CR) Ejegg: [C: +2] "Looks good, works with the new API. I just had to update the pointer to the submodule to match the merged patch." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:52:26] (Merged) jenkins-bot: Update Contact Layout Editor. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501456 (https://phabricator.wikimedia.org/T218754) (owner: Eileen) [22:55:28] (PS4) Ejegg: Start recording the payment processor id for recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501128 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [22:59:50] (CR) jerkins-bot: [V: -1] Start recording the payment processor id for recurring contributions [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/501128 (https://phabricator.wikimedia.org/T218616) (owner: Eileen) [23:17:48] (CR) Ejegg: [C: -1] "This is looking really good! Just need to restore at least one field to the filter list." (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/498413 (owner: Jgleeson) [23:54:36] (PS5) Ejegg: Add extension to cleanup sort name for orgs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) (owner: Eileen) [23:58:38] (CR) jerkins-bot: [V: -1] Add extension to cleanup sort name for orgs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/500851 (https://phabricator.wikimedia.org/T115536) (owner: Eileen)