[00:23:16] Fundraising Sprint Dirt Farming, Fundraising Sprint Elevator Maintenance 2016, Fundraising Sprint Freshmaking, Fundraising-Backlog, and 3 others: Make dedupe reversible step 1 of 5 test turning on logging on Staging to see how it performs. - https://phabricator.wikimedia.org/T127133#3987839 (Eilee... [00:33:10] (PS4) Eileen: Rude hack to make system checks STFU [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410681 [00:40:11] (PS3) Eileen: [NFC] pass arrays rather than strings to construct urls in dedupe code [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410634 [00:41:05] hi eileen [00:41:14] ejegg: hey [00:41:29] I messed up the chain removing that change :-( [00:43:12] spatton: have you already taken all campaigns down? [00:44:15] i can shut the site down whenever [00:44:21] i do not know how to disable the catchpoint alerts [00:44:47] eileen & cwd - have not. Can do now? [00:45:18] spatton: well it's technically 15 mins before outage starts but was starting to see if we could do some steps already [00:45:28] cwd - we need to check that with Jeff_Green [00:45:40] yeah, or we can just ignore them [00:45:45] :-) [00:46:36] (PS2) Ejegg: wmf: CRM-17158 block empty searches unless deliberate [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410376 (owner: Eileen) [00:46:46] (CR) Ejegg: [C: 2] wmf: CRM-17158 block empty searches unless deliberate [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410376 (owner: Eileen) [00:47:24] Word eileen, just disabled the live banner campaigns, so all clear to start working from our end [00:47:26] ejegg: we can roll with what is merged when outage starts & then update - the main think is the upgrade script starts [00:47:27] (PS2) Ejegg: wmf-sunset patch add extra columns to contribution recur tab [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410377 (owner: Eileen) [00:47:31] thanks spatton [00:47:41] eileen: OK, sounds good [00:48:09] ejegg: also, I'm trying to pull the ones mepps reviewed out of the chain as they can merge now [00:48:36] they're mostly familiar patches, I shouldn't be too long reviewing the rest [00:48:41] :-) [00:48:52] the chain is a bit messed up tho I think [00:49:13] s'okay, i'll rebase as I go [00:49:22] :-) [00:49:22] (PS3) Eileen: CRM-21766 quash reload warning [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410628 [00:50:24] (Merged) jenkins-bot: wmf: CRM-17158 block empty searches unless deliberate [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410376 (owner: Eileen) [00:50:52] we actually last did a civi upgrade in June last year [00:51:04] hence it's a bit big this time [00:51:11] yeah, it sure has been a while! [00:51:47] partly because I failed to crack the test issue to do with dao->free() last year [00:51:55] (PS2) Ejegg: CRM-21659 Add hook to CRM_Utils_System::redirect [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410637 (owner: Eileen) [00:52:27] ejegg: I'm happy to omit that hook on redirect for now [00:52:45] I can't use it yet & am pushing for an upstream merge [00:52:49] ok, i'll focus on the rest [00:54:18] (PS3) Ejegg: wmf-sunset patch add extra columns to contribution recur tab [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410377 (owner: Eileen) [00:55:37] took icinga down for 3hrs [00:56:01] (PS2) Ejegg: Return more rows per query and use unbuffered query to manage the memory [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413065 (https://phabricator.wikimedia.org/T120892) (owner: Eileen) [00:56:04] (PS4) Eileen: CRM-21753 add pass-through support for criteria in urls on dedupe pages [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410635 [00:56:18] cwd thanks [00:56:36] (CR) Ejegg: [C: 2] Return more rows per query and use unbuffered query to manage the memory [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413065 (https://phabricator.wikimedia.org/T120892) (owner: Eileen) [00:56:44] cwd - we can cross off 4 then https://etherpad.wikimedia.org/p/21-02-18CiviCRM4.7.31_upgrade [00:56:59] (PS3) Ejegg: Increase time out while doing exports [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410380 (https://phabricator.wikimedia.org/T120892) (owner: Eileen) [00:57:31] (PS2) Eileen: Deny web access to /vendor [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413003 [00:57:46] (CR) Ejegg: [C: 2] Increase time out while doing exports [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410380 (https://phabricator.wikimedia.org/T120892) (owner: Eileen) [00:59:04] cwd - where am I looking to do this from Begin watching logs, tail -f payments.error payments-initial fundraising-misc fundraising-drupal.   [00:59:29] eileen: payments.error and payments-initial won't show much [00:59:35] but that's on frlog1001 [01:00:02] what he said [01:00:32] thanks [01:01:31] (Merged) jenkins-bot: wmf-sunset patch add extra columns to contribution recur tab [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410377 (owner: Eileen) [01:02:37] eileen: shall i take down payments then? [01:02:42] ejegg: I'm going to do a submodule commit now & then another one later since it takes a while to get to deploy & we can start to kick off [01:02:45] cwd - yes [01:02:47] (Merged) jenkins-bot: Return more rows per query and use unbuffered query to manage the memory [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413065 (https://phabricator.wikimedia.org/T120892) (owner: Eileen) [01:02:51] here goes [01:04:26] eileen: sure, sounds good [01:04:42] maybe that 'move FindDuplicates' patch will pass if we rebase it over the submodule update [01:04:50] probably [01:05:01] (PS1) Eileen: CiviCRM submodule update (4.7.31 upgrade) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413076 [01:05:03] I'll work on it once the script has started [01:05:24] (Merged) jenkins-bot: Increase time out while doing exports [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410380 (https://phabricator.wikimedia.org/T120892) (owner: Eileen) [01:05:27] (CR) Eileen: [C: 2] CiviCRM submodule update (4.7.31 upgrade) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413076 (owner: Eileen) [01:07:26] ok things should be stopped [01:07:29] i'll stop replication [01:07:31] cwd - have we stopped replication (step 5) & disabled p-c (11) [01:07:37] ah cool - cross over timing [01:07:48] :) [01:07:51] i'll get p-c too [01:08:05] !log start outage to upgrade civicrm to 4.7.31 [01:08:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:09:19] (Merged) jenkins-bot: CiviCRM submodule update (4.7.31 upgrade) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413076 (owner: Eileen) [01:09:29] (PS2) Ejegg: Move Find Duplicates action to our extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/412828 (https://phabricator.wikimedia.org/T184955) (owner: Eileen) [01:10:07] !log disabled process-control [01:10:17] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/413077 [01:10:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:10:27] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/413077 (owner: Eileen) [01:10:46] (PS3) Ejegg: Hack out version & extension checks. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410381 (https://phabricator.wikimedia.org/T138334) (owner: Eileen) [01:10:57] (CR) Ejegg: [C: 2] Hack out version & extension checks. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410381 (https://phabricator.wikimedia.org/T138334) (owner: Eileen) [01:12:22] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/413077 (owner: Eileen) [01:12:47] (CR) Eileen: [C: -2] "put on hold for now - let's do the others first" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410637 (owner: Eileen) [01:13:47] ok - we are merged - cwd - ejegg I can push code out now but cwd needs to remove the triggers [01:13:57] ok [01:14:22] ok one sec [01:14:48] 'Find Duplicates' passes now :) [01:15:09] ejegg: yay [01:15:48] and it looks like a clean move, so... [01:15:49] (CR) Ejegg: [C: 2] Move Find Duplicates action to our extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/412828 (https://phabricator.wikimedia.org/T184955) (owner: Eileen) [01:16:09] !log update civicrm revision changed from efba904b06 to 43a7641597, config revision is ef884a2c5d [01:16:14] (Merged) jenkins-bot: Hack out version & extension checks. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410381 (https://phabricator.wikimedia.org/T138334) (owner: Eileen) [01:16:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:16:50] fantastic [01:16:50] "Enter the following 3-digit key to update. [ meh / n ]:" [01:17:25] cwd - I will kick the upgrade as soon as you confirm the triggers are gone [01:17:36] lol [01:17:54] sounds like either answer would no-op [01:18:15] haha nice [01:18:23] ok i'm going to run the file now [01:18:55] cool [01:19:29] (PS3) Ejegg: CRM-19752 Fix slow query on contribution dash. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410382 (https://phabricator.wikimedia.org/T152936) (owner: Eileen) [01:19:40] (CR) Ejegg: [C: 2] CRM-19752 Fix slow query on contribution dash. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410382 (https://phabricator.wikimedia.org/T152936) (owner: Eileen) [01:20:06] (Merged) jenkins-bot: Move Find Duplicates action to our extension [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/412828 (https://phabricator.wikimedia.org/T184955) (owner: Eileen) [01:20:12] eileen: ok they are gone [01:20:47] (PS3) Ejegg: Do not give validation error on address with no location_type. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410383 (https://phabricator.wikimedia.org/T152426) (owner: Eileen) [01:21:01] cwd cool so show triggers is clean? [01:21:22] (CR) Ejegg: [C: 2] Do not give validation error on address with no location_type. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410383 (https://phabricator.wikimedia.org/T152426) (owner: Eileen) [01:21:41] eileen: yep [01:22:00] great - it's running [01:22:38] confirmed the drop replicated [01:22:44] there is not much we can do now for over an hour [01:22:46] but not to 2001 [01:22:49] in case we need to roll back [01:23:05] ok sounds good, i'll get some dinner then! [01:23:50] cool [01:23:51] (Merged) jenkins-bot: CRM-19752 Fix slow query on contribution dash. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410382 (https://phabricator.wikimedia.org/T152936) (owner: Eileen) [01:25:39] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [01:26:20] (CR) Ejegg: "Looks like the core change merged: https://github.com/civicrm/civicrm-core/pull/10705/files" [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410384 (https://phabricator.wikimedia.org/T170900) (owner: Eileen) [01:26:46] (Merged) jenkins-bot: Do not give validation error on address with no location_type. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410383 (https://phabricator.wikimedia.org/T152426) (owner: Eileen) [01:27:18] (PS3) Ejegg: When undelete is true, add new contact as assignee for merged contact activity record [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410385 (https://phabricator.wikimedia.org/T178003) (owner: Eileen) [01:29:51] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [01:34:34] oops, I'll quit poking those patches to let you get the deploy merge through zuul [01:35:08] ejegg: nope that's fine [01:35:18] I'm now just poking around myself [01:35:25] the scripts are running [01:35:32] ah, cool [01:35:42] I'll do a sceond deploy once they have finished [01:35:58] I'm cursing my failure to use nohup [01:36:05] eek [01:36:23] "ALTER TABLE `civicrm_contribution` ADD COLUMN `invoice_number` varchar(255) COMMENT 'Human readable….. still running & will be for a bit [01:37:20] ejegg: oh - good point on the mail switch [01:37:31] we can drop that & add to settings [01:37:45] wait, don't we already have invoice number in there? [01:38:03] or I guess that was invoice ID [01:38:31] long story [01:38:35] yes - it was invoice_id [01:38:44] huh [01:38:47] & in 4.5 code was added that altered behaviour [01:39:10] so after some people throwing their weight around & pissing me off Coleman added invoice_numer [01:39:15] enforcing uniqueness? [01:39:39] kinda [01:39:54] more readable numbers = only unique within numbering scheme [01:40:30] plus the change made it so that only 'invoiced' contributions got numbers I think [01:40:32] (CR) Ejegg: [C: 2] When undelete is true, add new contact as assignee for merged contact activity record [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410385 (https://phabricator.wikimedia.org/T178003) (owner: Eileen) [01:41:03] it only showed up in an obscure combo & people who didn't notice the change for 2 years suddenly started using that combo & it was urgent & someone else's fault [01:41:17] ah, so should we be storing our non-civi-generated merchant references in invoice_number now? [01:41:56] hmm - no - merchant refs are trxn_id I think [01:42:09] the invoice number & id are both for civi generated [01:42:45] number is readable & id is an md5 for reasons that seemed less compelling to me than to others [01:42:48] ooh, shoot. we started using invoice_id for our payments-wiki-generated ctid.sequenceno identifier [01:43:02] it seemed to have been empty before [01:43:14] just got a call that there is a mountain lion down the street [01:43:23] and trxn_id is the payment-processor-generated ID [01:43:27] cwd oh dear [01:43:33] dog inside? [01:43:41] heh yeah [01:43:49] but she has been barking more than normal lately [01:43:50] shotgun loaded? [01:43:58] not yet [01:44:36] (Merged) jenkins-bot: When undelete is true, add new contact as assignee for merged contact activity record [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410385 (https://phabricator.wikimedia.org/T178003) (owner: Eileen) [01:44:37] cwd wow [01:44:40] (PS4) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [01:45:04] invoice_id is for a number we generate as is invoice_number [01:45:12] i caught a bobcat on my trail came recently: https://sadbastard.org/photo/cat.avi [01:45:13] (PS3) Ejegg: Packages updates: retry on deadlock, log details for queries [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410386 (owner: Eileen) [01:45:58] we've gone from altering table to committing alter [01:46:03] nice [01:46:56] ok - that's the first of the slow updates done [01:46:59] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [01:50:27] (PS5) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [01:52:38] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [01:56:03] (CR) Ejegg: [C: 2] Packages updates: retry on deadlock, log details for queries [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410386 (owner: Eileen) [01:59:13] (PS3) Ejegg: Fix drush performance command. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413067 (owner: Eileen) [01:59:53] (CR) Ejegg: [C: 2] "Thanks! looks like this has been busted since Sept 2016 (Iae39859d0a00a22b8b removed fn queue2civicrm_import)." [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413067 (owner: Eileen) [02:00:17] (Merged) jenkins-bot: Packages updates: retry on deadlock, log details for queries [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410386 (owner: Eileen) [02:01:34] ejegg: I'm going to do another submodule commit now since it seems everything that matters is committed [02:01:41] great! [02:03:49] (Merged) jenkins-bot: Fix drush performance command. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413067 (owner: Eileen) [02:05:35] (PS1) Eileen: CiviCRM Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413082 [02:06:24] (CR) Eileen: [C: 2] CiviCRM Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413082 (owner: Eileen) [02:10:38] (Merged) jenkins-bot: CiviCRM Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413082 (owner: Eileen) [02:11:33] ejegg: there is a change in localsettings if you could eyeball it - I tested locally & seems OK [02:12:22] looking [02:12:51] ccccccddfrjhljudbdludvjbjfrbjrcehlcidkuuukli [02:12:54] ack [02:13:46] lol [02:15:37] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/413083 [02:16:36] just got a bounced sms sending a civicrm cert password [02:16:57] eileen: should that first array element be 'Mailing Preferences'? [02:17:02] https://github.com/civicrm/civicrm-core/blob/master/settings/Mailing.setting.php#L319 [02:17:57] got to be a better way to keep that list up to date [02:18:08] ejegg: I wonder if it matters? We kinda deprecated those groups [02:18:16] ah, ok [02:18:18] & it worked locally as was [02:19:44] in that case, looks good! [02:19:45] (CR) Eileen: [C: 2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/413083 (owner: Eileen) [02:20:10] (Abandoned) Eileen: Temporary WMF fix on CRM-20905 [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/410384 (https://phabricator.wikimedia.org/T170900) (owner: Eileen) [02:20:17] thanks ejegg [02:20:24] one more patch gone [02:20:30] :) [02:20:43] (Merged) jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/413083 (owner: Eileen) [02:37:40] ejegg: so not the update is done we can schedule this command [02:37:41] echo '{"group": 268, "addressee_id": {"IS NOT NULL": 1},"contact_type": "Individual", "api.contact.create": {"addressee_id": 1} }'| drush cvapi --in=json Contact.get [02:37:59] & it will find individuals in group 268 with null as addresse_id & set it to 268 [02:38:02] set it to 1 [02:38:08] (individual default) [02:39:06] what's that group? [02:39:13] that seems to be enough to get the email & postal greeting too [02:39:17] mg? [02:39:20] I'ts one of the major gifts ones [02:39:26] ok [02:39:31] not sure which but 268 & 270 are etched in my braing [02:40:01] heh, oh man [02:40:08] you're in deep [02:40:10] I'm thinking to schedule every 10 mins or so rather than handle in the import code [02:40:18] (can I be cured?) [02:43:45] just checked - it's $500+ [02:55:53] ejegg: what are your thoughts about just dropping https://phabricator.wikimedia.org/T184823 from the sprint until such time as it occurs again since the job is back on & running [02:56:15] eileen: sure, if it's not a problem now, let's not worry about it [02:56:20] :-) [02:56:36] Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Omnimail recipient load tripping over non-download... - https://phabricator.wikimedia.org/T184823#3988094 [02:57:06] Fundraising Sprint Bermuda Rhombus (where things disappear then reappear), Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email: Omnimail recipient load tripping over non-download... - https://phabricator.wikimedia.org/T184823#3897611 [02:58:30] ejegg: also I added that change for T186162 to process-control (I figure if any syntax issues when I put in the slashes we can easily back up) - the command worked on staging [02:58:30] T186162: Process greetings for Major Gifts contacts - https://phabricator.wikimedia.org/T186162 [02:58:48] great [02:59:55] eileen: looks like the db updates are done? [03:00:32] cwd not quite [03:00:42] the columns get added to the log tables at the end :-( [03:01:45] eileen: i am not showing that in processlist [03:01:51] i don't think? [03:02:14] cwd nope - I think it has a delay of 30 sec between some steps [03:02:20] probably for the UI :-( [03:02:25] aah ok [03:02:30] no problem! [03:02:40] select version from civicrm_domain; now shows 4.7.31 [03:02:49] so it is on the last one [03:02:59] but I think then it will fix the log tables = slow [03:03:15] yep - it has started [03:03:59] but the good thing is the log tables are a recoverable step - ie. if it crashed part way through it's easy to retrigger that step (easier - it's not that tough with the upgrade itself) [03:05:48] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Process greetings for Major Gifts contacts - https://phabricator.wikimedia.org/T186162#3988098 (Eileenmcnaughton) I... [03:06:05] cool, yeah worst case we still have a copy from before it started [03:06:13] cwd the log tables are bigger but have less indexes [03:06:26] do you think they will take more time or less? [03:06:26] so my guess is quicker but not sure how those 2 balance out [03:06:31] yeah [03:06:51] well no worries i'll be around if increasingly drowsy :) [03:06:56] lol [03:21:39] (PS6) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [03:23:25] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [03:40:10] eileen: i should probably extend the icinga downtime huh? [03:42:08] cwd - yeah it has 1.5 of those long commands to go [03:42:32] probably at least 30 mins :-( [03:43:19] no prob, just submitted another 2hrs [03:43:28] we can cancel it if desired [03:43:40] ok cool [03:43:48] also will be some slave catch up at the end [03:44:40] oh good point [03:46:21] can't get good slaves anymore [03:47:01] X| [03:48:57] ug I might have just been cut off [03:50:08] from what? [03:50:44] the server [03:50:44] only irc seems to be working - is that alter still running? [03:51:31] yep i still see it [03:52:04] good - I have lost my connection but hopefully that will still complete (forgot nohup) [03:53:07] (trying to get back on) [03:53:58] eileen: i may have never mentioned it but i put tmux everywhere on frack [03:54:07] ah ok [03:54:27] there is a way to nohup a running command... [03:54:39] ooh [03:54:52] well hopefully that query will finish [03:55:09] to get the last one to run we can turn logging off & on - ie [03:55:16] drush cvapi Setting.create logging=0 [03:55:28] drush cvapi Setting.create logging=1 [03:57:05] hrm maybe not if the shell has gone away [03:57:24] is the sql still running? [03:57:45] yep [03:57:48] i won't rattle it [03:58:16] eileen: was it a drush command? [03:58:22] yep [03:59:06] interesting i don't see that running [03:59:12] ok so the php process may have exited [03:59:18] but left the query running [03:59:33] so if it was going to do anything else that will probably be missing [04:00:22] I;m back in - I think the problem was between the modem & the x-box controller.... [04:00:28] :P [04:00:50] anyway the process can be restarted in this part - just hope we can finish the running sql command [04:01:09] cool yeah i don't see why it wouldn't finish *knock on wood* [04:01:24] yep looks like it did [04:01:55] ah yes! [04:02:14] oh I think I've rekicked it [04:02:45] yep - it's doing the last big mysql command [04:03:05] I got it going faster than a speeding snail, stopping only to forget to add tmux :-) [04:04:33] \o/ [04:05:06] i set up a weather station and it is now the default for our village: https://www.wunderground.com/weather/us/nm/glorieta [04:05:15] running open source weewx software [04:05:21] nice! [04:05:43] cool [04:05:54] :) [04:06:13] the previous closest one was quite far away in a canyon and not indicative of the weather here [04:20:19] (PS7) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [04:21:57] (Abandoned) Eileen: Updated vendor submodule [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/407161 (owner: Mepps) [04:22:39] that long one is finished - into the short ones now [04:22:53] great [04:23:51] finished! [04:23:58] whew! [04:24:37] cool, should i re create the triggers? [04:25:34] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [04:26:31] cwd - yep - I sent the filename [04:26:58] thanks! [04:29:23] eileen: ok i show 449 triggers [04:29:50] cwd sounds good [04:31:19] eileen: want me to bring civi back before the others? [04:31:26] or do you have another way to test it works? [04:31:52] cwd I did one test [04:32:06] I edited a contact (addressee_id) via the ui [04:32:10] via drush [04:32:19] & I can see changes in change log - select addressee_id, addressee_display, log_date, postal_greeting_id from log_civicrm_contact WHERE id = 513869; [04:32:20] ah nice [04:32:23] so that's good [04:32:34] oh hey, we can use the --slow-start option to run-job [04:32:47] oh wait, that actually doesn't apply to the dq consumer [04:32:51] so yeah step 1 of bringing it back I think [04:33:05] ok here goes [04:34:35] just reading this note - NOTES [04:34:35] there are two triggers on drupal.contribution_tracking [04:34:49] I'm not sure if anything needs doing there [04:35:18] vaguely familiar [04:35:29] hope it doesn't matter cause i flipped the switch [04:36:16] everything looks ok so far [04:38:42] hmm trying to load the website [04:39:57] it's still showing http drops due to firewall :-( I though I might have killed those [04:41:15] !log update civicrm revision changed from 43a7641597 to b27e6a5019, config revision is ef884a2c5d [04:41:24] my process_list change was a dud [04:41:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:42:01] spatton: if you are still about we are into bringing back mode [04:42:52] eileen: want me to bring back process control? [04:43:05] cool! TY for the heads up eileen. Ready for me to reenable our 4 Swedish campaigns? [04:43:07] cwd let's do a short run of the qc first [04:43:21] I'll change the setting in the ui [04:43:59] spatton: in about 5 mins I think - after ejegg does his check [04:44:29] ok cool [04:44:36] Sounds good, no rush from my end. I'll wait for your email [04:45:21] cwd want to enable the jobs? I've got the batch size set to 1 [04:46:09] ejegg: sure, all of them? [04:46:37] ah, one sec, I'll run it directly for a test [04:46:42] ok [04:46:46] i'm going to re-enable icinga [04:47:06] ejegg: can you see what is wrong with the job I just added [04:47:10] to process_control [04:47:19] I'll take a look [04:47:52] I think I got it - trailing " [04:47:59] well, the donations queue consumer looks good [04:49:11] i think icinga is workng again [04:49:14] nope I didnt 6:51 error syntax error: mapping values are not allowed here [04:49:24] i have hacked the interface with an html file on my laptop [04:49:25] cwd want to turn on the other jobs? [04:49:30] sure [04:49:36] eileen: erk? [04:49:50] shall we punt that one till tomorrow? [04:50:14] ejegg: sure [04:50:18] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) [04:51:15] whoops, restarted replication [04:52:26] ACKNOWLEDGEMENT - check_mysql on frdb2001 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) Casey Dentinger catching up [04:54:35] ejegg: I found the yaml error [04:54:43] nice [04:55:10] it's spaces [04:55:14] restarted them jobs [04:55:19] great [04:55:42] spaces in the json? [04:57:19] yep [04:57:30] (shudder) [04:59:17] I ok - it's passing now aside from the same errors the others have [05:00:37] !log enable major gifts address job [05:00:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:01:16] ok cwd - ejegg can we tell people it's back up - have we done the last steps ? https://etherpad.wikimedia.org/p/21-02-18CiviCRM4.7.31_upgrade [05:02:04] well, things are running again [05:02:30] yep an alerts are on again [05:03:04] ok will send an email [05:03:21] quality outage everyone [05:04:08] and i will go to bed... [05:04:26] thanks for your help! [05:04:28] gnight all! [05:04:32] buenas noches hombre [05:05:00] :-) [05:05:49] eileen: anything i have forgotten? [05:06:46] cwd - is everything crossed out? [05:07:43] I think it all is - I don't know what to make of the contribution_tracking triggers tho & I think you might make a note to get jeff to tell you how to take down catchpoint [05:07:45] yep except maybe the last git thing? [05:07:55] I'll do the git thing [05:08:15] cool yeah i will bring those up tomorrow [05:09:10] (PS1) Eileen: Update trigger script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413094 [05:09:29] cwd - here is the git thing https://gerrit.wikimedia.org/r/#/c/413094/ [05:14:25] (CR) Cdentinger: [C: 2] Update trigger script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413094 (owner: Eileen) [05:14:27] eileen: works for me! [05:15:25] cwd - cool - all done - I'll be around for a bit if there are any fail mals [05:16:10] good deal, i'm going to wind down but will have phone on, have a good one! [05:17:58] cwd - did you see email from Pat [05:18:27] (Merged) jenkins-bot: Update trigger script. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413094 (owner: Eileen) [05:19:59] yeah i'll reply, some automated test [05:20:29] it was the outage that caused it? [05:21:34] yeah i'm sure [05:22:01] i assume they have some bot that hits the site? [05:22:30] cwd - even if we miss some we get them back on audit don't we? [05:22:59] oh yeah there are a few layers of defense [05:23:41] and since campaigns were down i bet it wasn't many [05:27:36] ok bed time, good night! [05:28:25] night [07:07:43] (PS8) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [07:13:33] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [08:07:38] (PS1) Eileen: Temporary fix to prevent geocoding on dedupe [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413110 [08:09:57] (PS2) Eileen: Temporary fix to prevent geocoding on dedupe [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413110 [09:20:10] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1018896 Threads: 2 Questions: 33868262 Slow queries: 684567 Opens: 6883 Flush tables: 1 Open tables: 824 Queries per second avg: 33.240 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5356 [09:46:13] (PS9) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [09:52:05] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [09:55:20] (PS10) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [10:01:03] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [10:05:27] (PS11) Eileen: [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 [10:10:59] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [15:36:32] PROBLEM - Host alnitak is DOWN: PING CRITICAL - Packet loss = 100% [15:36:44] morningz! [15:40:22] RECOVERY - Host alnitak is UP: PING OK - Packet loss = 0%, RTA = 36.20 ms [15:41:23] hey AndyRussG :) [15:49:39] jgleeson: :) [15:55:18] (PS1) Zfilipin: WIP Selenium test example [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/413181 (https://phabricator.wikimedia.org/T99955) [16:03:34] Wikimedia-Fundraising: Create translated versions of new Thank You page - https://phabricator.wikimedia.org/T185333#3989571 (schoenbaechler) Continued working on the multilingual Thank You page today. The page’s now completely designed in modular way. It allows to easily swap primary/secondary call to action... [16:13:51] whoa, cool! [16:14:01] selenium tests for DonationInterface [16:14:09] we neeeds to jump on that [16:14:38] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Browser-Tests, Patch-For-Review, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989614 (Ejegg) [16:16:05] (CR) Zfilipin: "check experimental" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/413181 (https://phabricator.wikimedia.org/T99955) (owner: Zfilipin) [16:19:35] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: saved field mapping Fraud Detection not exporting properly - https://phabricator.wikimedia.org/T187903#3989637 (MBeat33) [16:24:17] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Browser-Tests, Patch-For-Review, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989655 (zeljkofilipin) I have set up Zuul so mediawiki-core-qunit-selenium-jessie job runs if you leave... [16:26:09] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Browser-Tests, Patch-For-Review, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989662 (zeljkofilipin) >>! In T99955#3989612, @Ejegg wrote: > If you haven't found it yet, the vagrant... [16:30:24] (PS2) Zfilipin: WIP Selenium test example [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/413181 (https://phabricator.wikimedia.org/T99955) [16:35:34] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Browser-Tests, Patch-For-Review, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989718 (Ejegg) @zeljkofilipin the vagrant role should install a Main Page with a bunch of links to paym... [16:52:26] oh drat, we didn't do the CN/translation review [16:52:34] sorry fr-tech, [16:52:55] i totally meant to set that up before the meething in an hour [16:53:37] ejegg: 'sOK... :) [16:55:50] maybe we can do a quick look after standup [17:09:13] Wikimedia-Fundraising: Create translated versions of new Thank You page - https://phabricator.wikimedia.org/T185333#3989887 (Pcoombe) it should be possible to include messages in HTML attributes, but you need to be careful with URLs as they will get automatically turned into links by MediaWiki if they are ou... [17:15:48] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: saved field mapping Fraud Detection not exporting properly - https://phabricator.wikimedia.org/T187903#3989928 (mepps) a:mepps @MBeat33 I'm looking into this. [17:16:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-WMF-Audit, Recurring-Donations: wmf_audit: support recurring donations - https://phabricator.wikimedia.org/T187505#3977074 (DStrine) [17:20:25] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: saved field mapping Fraud Detection not exporting properly - https://phabricator.wikimedia.org/T187903#3989984 (mepps) This impacts all exports, not just contributions, regardless of field mapping. [17:37:35] Fundraising-Backlog, Fr-backlog-cleanup-Q3_2017-18, Zero: Disable fundraising banners for zero rated devices - https://phabricator.wikimedia.org/T105816#3990062 (Pcoombe) Open>declined Closing this since Wikipedia Zero is [no longer adding more countries](https://blog.wikimedia.org/2018/02/16... [18:10:51] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Adding translatable message to banner FR2015_translations takes a long time and shows error - https://phabricator.wikimedia.org/T167442#3990183 (DStrine) per a CN/translatio... [18:13:44] Fundraising-Backlog, Fr-CentralNotice-Translation-Bugs, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: "Internal error" on testwiki's Special:LanguageStats due to non-existing CentralNotice banner - https://phabricator.wikimedia.org/T157997#3022975 (DStrine) @Sa... [18:36:32] fr-tech any scrum of scrums news? [18:36:44] None here, thanks. [18:37:17] ejegg, I'd be interested to know if any of the mediawiki core guys are tracking test success on vagrant [18:37:41] and if it's a priority worth looking at or would result in diminishing returns [18:37:56] sure, I'll ask [18:38:03] thanks [18:38:21] jgleeson do you know if the tests pass on the non-fr checkout of mediawiki in vagrant? [18:38:54] Tests: 18076, Assertions: 102904, Errors: 11, Failures: 81, Skipped: 40, Risky: 5. [18:39:05] non-fr mediawiki output ^ [18:39:15] thanks [18:39:25] paymentswiki is higher but doesn't currently complete on my machine so need to dig into that [18:45:49] jgleeson: nobody from the releng team was present to answer [18:46:00] np, thanks [18:47:30] I'm gonna try a few things out myself and chip away at the failures over the next few weeks in my spare time. I figure it's a good learning exercise to get more familiar with the stack. [18:48:17] actually, I might check out the mediawiki-vagrant phab project to see if others are already aware and working on it [18:48:28] probably the best place to start [19:13:26] just logged on to check & no sign of any fallout from the upgrade [19:15:19] mepps are you able to do a quick review on this - https://gerrit.wikimedia.org/r/#/c/413110/ - v straight forward & will help me figure out the remaining test issues [19:16:00] (CR) Mepps: [C: 2] Temporary fix to prevent geocoding on dedupe [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413110 (owner: Eileen) [19:16:10] thanks mepps! [19:16:17] looks good eileen! i'm looking at one seemingly upgrade related issue [19:16:37] mepps ah - what's that? [19:16:45] eileen: https://phabricator.wikimedia.org/T187903 [19:17:05] see my note too--it's affecting all exports [19:17:16] ohhh [19:17:19] yuck [19:17:33] any flashes on inspiration? i'm digging around a lot of corners right now [19:17:41] it's something to do with group by [19:17:45] ahhh [19:17:47] there have been changes around that [19:17:47] that makes sense [19:18:10] I was discussing with Monish last night on chat - but in a different context [19:19:37] = it's kinda why I'm so leary of changing that 'search primary details only' setting - I don't trust some of that stuff [19:19:46] (Merged) jenkins-bot: Temporary fix to prevent geocoding on dedupe [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413110 (owner: Eileen) [19:20:04] https://www.reddit.com/r/sanfrancisco/comments/7yof6h/head_of_the_wikimedia_foundation_on_the_fact_that/ [19:20:43] (PS1) Eileen: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413222 [19:22:52] hmm interesting eileen [19:24:25] So to back up a bit - in 5.7 mysql it enforces 'FULL GROUP BY' mode [19:24:38] which is basically the mysql equivalent of e-notice IMHO [19:24:50] ie. it blocks queries that could give dodgey results [19:25:13] which is a query where a field is neither grouped by or one of the aggregate columns [19:25:59] to take the example of the primary search by email - if we neither group by or aggregate the email & there is more than one then you will get one of the emails that exists & it is technically an inaccurate result [19:26:34] In order to address this they originally tried to add ANY_VALUE to the queries if mysql full group by mode is enabled [19:27:12] but that didn't work on MariaDB & I kept saying 'you're hiding not fixing here' [19:27:23] any way I'm not sure anyone listened to me :-( [19:27:39] but Monish seems to agree with me now :-) [19:27:46] haha [19:28:00] then it turned out that ANY_VALUE didn't work with Maria DB! [19:28:27] argh! that is problematic [19:28:43] so I'm not quite sure how it is now but I think the issue is that when it is calling searchQuery with $count=TRUE it's grouping differently to when calling without $count [19:29:07] hmm [19:29:10] & it exports $count number of rows [19:29:28] well - that is an issue Monish was discussing last night - but it feels related [19:29:37] and / or the same [19:30:21] yeah so currently it's only organizing two rows at a time [19:30:33] i'm curious about looking again keeping group by in mind though [19:30:44] because i saw one place it was just an individual and an org [19:30:56] in contact export that is [19:31:03] brb, check in with K4 [19:33:58] (CR) Krinkle: Impression events via EventLogging (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/408488 (https://phabricator.wikimedia.org/T185932) (owner: AndyRussG) [19:36:07] (CR) Krinkle: "Code LGTM, haven't tested myself with a local install though. I do have one question, but that's likely just because I'm not yet familiar " [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/408488 (https://phabricator.wikimedia.org/T185932) (owner: AndyRussG) [19:36:41] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: " Could we also update the tokens for the labels? Currently addressee doesn't show up (nor contact first and last/org name). Thank you kindly!" - https://phabricator.wikimedia.org/T187248#3990555 (LeanneS) Sounds good. Let me know when works for you! [19:36:51] mepps - I'll try reverting this on staging once I've replicated there https://github.com/eileenmcnaughton/civicrm-core/commit/b07c960a351d992e19bc734f667a69a2bbc28f95 [19:37:39] (CR) Eileen: [C: 2] Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413222 (owner: Eileen) [19:41:44] (Merged) jenkins-bot: Submodule update [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/413222 (owner: Eileen) [19:48:12] Fundraising Sprint Cottage Cheese isn't Made of Cottages, Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Process greetings for Major Gifts contacts - https://phabricator.wikimedia.org/T186162#3990612 (Eileenmcnaughton) I... [19:51:26] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: saved field mapping Fraud Detection not exporting properly - https://phabricator.wikimedia.org/T187903#3989637 (Eileenmcnaughton) I think we should test reverting this commit https://github.com/eileenmcnaughton/civicrm-core/commit/b07c960a351d992... [20:04:58] (CR) XenoRyet: Script for Paypal refunds (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/411386 (https://phabricator.wikimedia.org/T186883) (owner: Mepps) [20:06:41] (CR) Eileen: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [20:11:47] (CR) jerkins-bot: [V: -1] [WIP] add geocoder [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/410552 (owner: Eileen) [20:34:00] Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: " Could we also update the tokens for the labels? Currently addressee doesn't show up (nor contact first and last/org name). Than... - https://phabricator.wikimedia.org/T187248#3990780 [20:34:12] Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: Investigate getting output to word option to work better - https://phabricator.wikimedia.org/T187251#3990781 (DStrine) [20:34:39] Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: saved field mapping Fraud Detection not exporting properly - https://phabricator.wikimedia.org/T187903#3990784 (DStrine) [21:07:56] (PS10) Mepps: Script for Paypal refunds [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/411386 (https://phabricator.wikimedia.org/T186883) [21:25:16] fundraising-tech-ops, Operations, ops-eqiad: rack frbast1001 - https://phabricator.wikimedia.org/T187363#3990946 (cwdent) [21:25:54] fundraising-tech-ops, Operations, ops-eqiad: rack frdata1001 - https://phabricator.wikimedia.org/T187364#3990951 (cwdent) [21:29:30] fundraising-tech-ops, Operations, ops-eqiad, Patch-For-Review: Rack/setup frmon1001 - https://phabricator.wikimedia.org/T186073#3990975 (cwdent) [21:30:01] fundraising-tech-ops, Operations, ops-eqiad: rack frpig1001 - https://phabricator.wikimedia.org/T187365#3990976 (cwdent) [21:30:48] fundraising-tech-ops, Operations, ops-eqiad: rack frdata1001 - https://phabricator.wikimedia.org/T187364#3990984 (cwdent) [21:59:57] mepps I have found the cause of that bug [22:00:28] nice eileen! i was just staring at that [22:00:31] what is it? [22:01:13] mepps I'll put up my - in-progress fix [22:01:33] it only affects us because it relates to the DAO clearing that was breaking out tests [22:03:38] (PS1) Eileen: Stop weird cloning [wip] [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413263 [22:04:59] mepps: ^^ I haven't tested that yet - will switch to a unit test to tidy up now I see the issue [22:53:45] i'm off a little early, but will be checking in [23:30:59] (PS2) Eileen: Stop weird cloning to fix export. [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/413263 (https://phabricator.wikimedia.org/T187903) [23:35:01] Fundraising Sprint Dinosaur Cookies co-existed with Gingerbread People, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Civi: saved field mapping Fraud Detection not exporting properly - https://phabricator.wikimedia.org/T187903#3991394 (Eileenmcnaughton) I have found the cause of this - due to s... [23:35:58] mepps I cleaned up the patch - I'm going to keep working on a couple of unit tests against master [23:36:12] but I'm feeling pretty confident we can just strip that rubbish out [23:45:18] Gotta run an errand. Be back in a bit.