[00:05:48] wileen I'm having trouble finding instances of bad data [00:06:08] *eileen [00:06:29] definitely seeing some 4+ digit numbers without commas [00:06:34] ejegg: this is what I'm going off https://phabricator.wikimedia.org/T229684 [00:07:17] is DOES seem to reference the new fields - I can't see the diff though [00:07:45] I'm looking at the latest update file in /var/lib/silverpop_export and the dates and 4+ digit numbers look good to me from a spot check at the end of the file [00:07:56] let's see if I can regex out some of the bad ones [00:12:37] cool [00:16:46] huh, all the fields I can find with more than 29 commas have a quoted field like address or name with the extra comma [00:23:05] hmm - any date field issues? [00:23:39] I'm guessing mnoor is not about - because we should also confirm the new fields WERE added at the silverpop end.... [00:24:50] haven't checked on the date fields yet, but I'll try to think of a good pattern [00:29:32] (PS1) Eileen: Exclude Refund contributions from calculations. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/527706 (https://phabricator.wikimedia.org/T229273) [00:29:33] nothing with the day field first (2-9 in first digit) [00:30:22] hmm we never did get it confirmed that the new fields were added at silverpop end [00:30:47] As an aside I just pushed up https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/527706/ [00:31:59] i think mnoor said something about adding those field in her wednesday advancement email standup [00:32:19] 'working with Brian from Trilogy today on getting our new shiny IBM fields mapped up' [00:33:35] eileen: ejegg my understanding was that this could be tackled early next week [00:34:40] (CR) jerkins-bot: [V: -1] Exclude Refund contributions from calculations. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/527706 (https://phabricator.wikimedia.org/T229273) (owner: Eileen) [00:34:48] (CR) Ejegg: [C: +2] "Seems correct!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/527706 (https://phabricator.wikimedia.org/T229273) (owner: Eileen) [00:36:24] mepps cool. Since nothing's jumping out, I think I'll just roll back the pymysql change again [00:36:39] and that'll at least give tell us whether it's that or the new fields [00:38:19] eileen I'm going to be lazy and just do the rollback via fundraising_tools_update without changing the contents of the repo [00:39:20] err, that's fundraising_code_update -p tools=rollback [00:40:06] (CR) jerkins-bot: [V: -1] Exclude Refund contributions from calculations. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/527706 (https://phabricator.wikimedia.org/T229273) (owner: Eileen) [00:40:13] !log rolled back fundraising python tools from 493a38f9e0 to 2a56e5e283 [00:40:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:44:57] Fundraising Sprint Office  , Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Bug: "Total Bad Data" rows in IBM import should be in the correct format when exported out of Civi - https://phabricator.wikimedia.org/T229684 (Eileenmcnaughton) @MNoorWMF @CCogdill_WMF all the referenced fields are... [00:45:46] Fundraising Sprint Office  , Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Bug: "Total Bad Data" rows in IBM import should be in the correct format when exported out of Civi - https://phabricator.wikimedia.org/T229684 (Eileenmcnaughton) Also note that @ejegg has rolled back a different pat... [00:45:52] eileen: OK, that's rolled back [00:46:21] ejegg: ok - I'm going to deploy that patch you just +2d & regen triggers for Jeff [00:46:59] sounds good. Have a good (rest of your) weekend! [00:47:08] :-)