[11:38:11] Fundraising-Backlog, Language-Engineering, Phabricator: Mingle migration script - https://phabricator.wikimedia.org/T822#2532453 (Danny_B) [13:27:19] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: PHPMailer dying on multiple email addresses - https://phabricator.wikimedia.org/T142371#2532998 (Ejegg) [14:13:13] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: PHPMailer dying on multiple email addresses - https://phabricator.wikimedia.org/T142371#2532998 (Danny_B) The error mentions "Invalid address", so it's more likely issue of an address rather than being them multiple (unless it considers all addresses as... [14:15:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: WMF CiviCRM failmail - PHPMailer dying on multiple email addresses - https://phabricator.wikimedia.org/T142371#2533100 (Ejegg) [15:50:35] Fundraising Sprint Octopus Untangling, Fundraising-Backlog, FR-Ingenico: Ingenico: AmEx card showing up on forms where it shouldn't - https://phabricator.wikimedia.org/T142050#2533553 (cwdent) Open>Resolved a:cwdent This was fixed in https://gerrit.wikimedia.org/r/#/q/Ifea1eaba3a6153fc4d5... [16:16:53] (CR) Cdentinger: [C: 2] Move phpunit tests to phpunit folder [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303282 (owner: Reedy) [16:18:45] (Merged) jenkins-bot: Move phpunit tests to phpunit folder [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303282 (owner: Reedy) [17:26:33] awight: looking at these mutating subscriptions [17:26:46] That sounds fun! [17:27:13] it's a lot more interesting now that we have these log gables [17:27:15] *tables [17:27:56] it was the butler in the garage with a frying pan [17:28:16] holler if I can help [17:28:25] so, for all the earlier charges, there are two updates in the logs, one 2 secons after the other [17:28:44] bu tfor the latest charge, there's a third immediately after the first [17:28:49] wow, so something about the actual db records changed? [17:29:22] that changes the currency to USD and sets the next_sched_contribution_date back one day from the previous one [17:29:36] and also sets the modified_date [17:30:02] omg [17:30:07] but the really odd thing about this rogue update is that the log_conn_id is the ( i think ) old-style 16 hex chars [17:30:19] O_O [17:30:33] while the rest are the new c_\d{6}.\d{8} [17:31:01] I can't recreate it locally, so I'm thinking there's something trigger-ish going on [17:31:14] Fundraising-Backlog, fundraising-tech-ops, Operations, Patch-For-Review: Allow Fundraising to A/B test wikipedia.org as send domain - https://phabricator.wikimedia.org/T135410#2534111 (Jgreen) Open>Resolved [17:31:16] good thinking [17:31:34] sounds nasty [17:32:10] Jeff can dump the triggers... [17:32:55] ejegg: is USD the column default? [17:33:13] ooh I hope not [17:33:39] nope, default null in code and in schema [17:34:54] seems weird that the db would choose that value [17:36:06] hmm, if it was a trigger, it would have the same log_conn_id though, right? [17:36:32] so strange that it's in the exact same second as the legit update [17:38:31] It might make sense to put this on pause until eileen's around... [17:39:10] yeah log_conn_id comes from a mysql session-scoped variable [17:39:19] aha, it's the same log_conn_id as the associated insert into civicrm_contribution [17:39:21] absolutely freakish that it can change to something unexpected [17:39:40] which is also a 16-char hex? [17:39:44] ooh, we store all the contributions as 'USD' [17:39:46] yep [17:40:10] anyway, eileen will definitely want to hear about the log_conn_id part of this [17:40:12] I'm guessing this is a 4.7 'improvement' to keep the recur row in sync with the latest child contribution [17:41:18] for background, upstream Civi used connection_id() for log_conn_id, but it was too small to prevent collisions [17:41:54] eileen did something fancy involving concatting a mt_rand stored as a PHP static variable to get around this [17:42:05] However, 16 chars of hex doesn't sound like anything I recognize [17:42:33] Almost sounds like upstream did the same "fix" but in a different way? [17:44:56] ejegg: ah very good point--we've been subverting the Civi currency thing all along [17:45:40] Dang. I guess the correct fix is to set civicrm_contribution_recur to match the contribution, and have recurring_globalcollect pull from wmf_contribution_extra.original* fields. [17:45:43] older log lines had the same buncha hex [17:46:56] err, you mean set the civicrm_contribution_recur to be in USD for the converted amount? [17:48:04] yah [17:48:12] that unfortunately would be consistent [17:48:14] ugly [17:48:24] well, it's what we do for the contribution records [17:48:52] have you tried using Civi once contributions are actually multi currency? [17:49:20] it's pedantically correct, but super obscure. All the reports are suddenly grouped and totaled by currency [17:49:34] ugh [17:49:56] (CR) Awight: Damaged datastore (4 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302758 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [17:49:59] (PS6) Awight: Damaged datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302758 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [17:50:04] (CR) Awight: [C: 2] Damaged datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302758 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [17:50:05] do we want to charge them in USD? like that's the number that gets sent to the processor? [17:50:14] maybe we need a new wmf_recur_extra table [17:50:24] cwd nope, we want to use the original currency [17:50:25] no, we want to charge them the original amount and currency [17:50:32] ejegg: why would we need that? [17:50:55] the current detente is that we look up the first contribution record in the series, and pull w_extra fields from there [17:50:56] awight do we always make the first charge from the frontend? [17:51:03] yeah [17:51:07] k [17:51:52] the real nasty thing is it changing the recur currency but not amount [17:52:00] if it's doing one it should do the other [17:52:04] eff-word [17:52:30] man, btw nice work knocking out the damaged/delay stuff. [17:53:07] thanks! Mostly just comming a lot of discussions to code [17:53:08] you did that in about the time it would've taken me to write a bunch of 4-point tasks :p [17:53:12] *committing [17:53:15] eesh [17:53:47] * ejegg takes off useless fingersplint [17:53:54] harr [17:54:23] I can't make these keys mash 'cause my tags ain't right [17:59:22] yep, 4.8. BAO/Contribution.php, line 217 [17:59:44] (Merged) jenkins-bot: Damaged datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302758 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [18:03:02] https://github.com/civicrm/civicrm-core/commit/912594079224e305e4723e6538eaed1e9e081448 [18:05:07] (CR) Awight: [C: -1] "Trivial incompatibility with previous patch" (2 comments) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [18:07:00] ooh, so updateOnNewPayment is not actually trying to change the currency, it's just not setting it in the create params [18:07:24] and then BAO/ContributionRecur.php is using the default on line 90 [18:09:24] i gotta run a few errands [18:10:09] awight: that code is also changing the next_sched_contribution_date in a way that doesn't do our snazzy day-of-month preservation [18:10:20] looks like it always adds 30 days [18:11:57] Fundraising-Backlog, FR-Paypal, FR-WMF-Audit: PayPal audit date format changed? - https://phabricator.wikimedia.org/T142417#2534257 (awight) [18:12:03] dear god [18:12:32] I guess that earns an extra 10% on each subscription per year :p [18:12:49] so, i'm going to have it leave the 'next_sched' date alone if it's already in the future [18:13:00] hehe 365/30 = 12 1/6 [18:13:05] 17% bonus [18:13:28] and make it leave the currency alone, because it's just resetting to default now, not even matching the latest contribution [18:13:48] wow. [18:14:06] thanks. We should eventually have a unit test for this [18:15:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Failmail code no longer recognizes comma-separated addresses - https://phabricator.wikimedia.org/T142420#2534300 (awight) [18:16:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Failmail code no longer recognizes comma-separated addresses - https://phabricator.wikimedia.org/T142420#2534324 (awight) p:Triage>High [18:17:00] Fundraising Sprint Octopus Untangling, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Failmail code no longer recognizes comma-separated addresses - https://phabricator.wikimedia.org/T142420#2534300 (awight) [18:17:24] ejegg: Can I make a task for the recurring thing, or you already have one? [18:17:59] awight T142312 [18:18:07] ty [18:19:08] Fundraising Sprint Octopus Untangling, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Unplanned-Sprint-Work: Failmail code no longer recognizes comma-separated addresses - https://phabricator.wikimedia.org/T142420#2534337 (Ejegg) [18:19:14] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: WMF CiviCRM failmail - PHPMailer dying on multiple email addresses - https://phabricator.wikimedia.org/T142371#2534339 (Ejegg) [18:29:12] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Merging Records manually in Civi - WMF Donor information doesn't automatically merge over - https://phabricator.wikimedia.org/T142008#2534382 (RLewis) @Eileenmcnaughton sorry I'm confused on this one. Wouldn't we always want the activities and contribut... [18:34:26] (PS1) Ejegg: Fix recurring contribution update bugs [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/303598 (https://phabricator.wikimedia.org/T142312) [18:34:48] awight: I think that'll do it, not yet tested tho ^^ [18:35:26] i mean fr-tech ^^ [18:38:12] I'll take a peek after testing this failmail patch... [18:43:07] (PS1) Awight: Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) [18:44:39] (CR) jenkins-bot: [V: -1] Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [18:45:37] (PS2) Awight: Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) [18:47:49] ejegg: Any idea why we're not sending a currency param? [18:48:31] awight pretty sure that updateOnInstallment function isn't meant to change the currency [18:48:31] (CR) jenkins-bot: [V: -1] Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [18:48:41] so it just never sets it in 'params' [18:49:04] it should be kosher to do that though [18:49:26] leaving a param out of an edit api call should never mean overwrite with default' [18:49:37] for real [18:49:45] this is a really smelly hook [18:52:21] (CR) Ejegg: Fix recurring contribution update bugs (1 comment) [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/303598 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [18:52:51] ok, i'll prep some sql to fix the busted records [18:54:03] (Merged) jenkins-bot: Fix recurring contribution update bugs [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/303598 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [18:54:45] Ahh I see what you mean now--the add() function gets called even during update on an existing row, where it should be 110% kosher to omit columns [18:54:54] right [18:56:26] I'm gonna put another coat of varnish on these shelves, back in 30 min... [19:09:00] (PS1) Ejegg: Fix incorrect contribution_recur rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) [19:11:15] fr-tech got some fix-up SQL for review ^^ [19:11:50] I'll take a look [19:12:07] (CR) Ejegg: [C: 2] "I already ran this on prod, self-merging for posterity" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/302290 (owner: Ejegg) [19:12:24] (that was some older fir-up sql, not the current script) [19:12:29] *fix-up [19:12:47] ejegg: I assume there'd be a db backup before that's run? [19:13:07] err, that's an idea [19:13:13] :) [19:13:33] * AndyRussG runs for the hills [19:15:09] (Merged) jenkins-bot: Backfill wmf_contribution_extra [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/302290 (owner: Ejegg) [19:15:48] whoa, di tests all moved [19:15:54] Huh [19:15:58] cwd|afk might need to update frig [19:17:38] Not used to seeing all those ANDs in an inner join... Looks like it's the same as WHERE tho... http://stackoverflow.com/questions/18153665/inner-join-where-clause [19:18:09] yeah... the curency<> clause is the only real join-y thing besides the id [19:18:30] but I could rewrite as 'WHERE' if that's clearer [19:19:02] I think if it works it's fine :) [19:19:13] Seems totally acceptable syntax [19:19:22] I'm more used to seeing it in the WHERE too, but nothing wrong with this way, I don't think. [19:19:28] I don't see how the <> is join-i-er [19:19:35] * AndyRussG snarfs a nit [19:20:05] ah, good point [19:21:34] ejegg: what about someone whose currency was intentionally changed? Does that happen? [19:21:43] And might not it seem the same in the log? [19:21:54] or also be detected as a case of this bug by that SQL? [19:21:57] aww sad, I tried to link to my favorite literary rendition of roasting and eating lice, but the internet thinks I'm only interested in cliff notes [19:22:03] (dunno really how the log table works) [19:22:47] AndyRussG: I don't think we've changed the currency intentionally before [19:22:55] ooh, but I could limit on the log_user too [19:22:56] say, can you see... any bed bugs on me? [19:23:44] K [19:23:49] From "all quiet on the western front", anyway. I guess it's only used in classrooms and all anyone wants to do is read the summary. sad [19:25:17] awww [19:25:35] * AndyRussG thinks of more things to add to his kids' internet filter in a few years [19:26:18] hehe -cliffnotes nice [19:26:56] Always looking for more ways things to repress and censor [19:27:17] (PS2) Ejegg: Fix incorrect contribution_recur rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) [19:28:44] ejegg: was there a problem with the tests moving? [19:28:59] nope, i just noticed when i pulled [19:29:04] oh, but i don [19:29:21] 't know what frig does to delete tests on deploy prep [19:29:47] Seems to find the same number of rows, so apparently we never did intentionally change currency. Good to check though. [19:30:21] ejegg: http://stackoverflow.com/questions/11011384/how-to-test-an-sql-update-statement-before-running-it [19:30:32] i think it should be fine since it just regexes the path [19:30:36] and it still starts with tests [19:30:44] but i haven't used it in awhile :P [19:30:57] AndyRussG: yeah, i often do the txn thing [19:31:21] ah cool [19:31:22] I ran the select and it looks sane. [19:32:25] Fundraising Sprint Nitpicking, Fundraising Sprint Octopus Untangling, Fundraising-Backlog, Unplanned-Sprint-Work: Help setting up a Top Prospects report in Civi - https://phabricator.wikimedia.org/T113904#2534671 (RLewis) @Eileenmcnaughton and @DStrine thanks for the new prototype. Just a few t... [19:36:21] (PS1) Ejegg: Optional limit of number of retry messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 [19:38:49] (PS2) Ejegg: Optional limit of number of retry messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 (https://phabricator.wikimedia.org/T142028) [19:38:54] (CR) Awight: [C: 2] Optional limit of number of retry messages (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [19:39:40] (PS2) Ejegg: Requeue delayed messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) [19:40:11] AndyRussG: That SQL is looking pretty good to me. Did you have any more concerns, or should one of us press the ol' +2? [19:40:38] (PS7) Awight: QueueConsumer always shunts to damaged datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302633 (owner: Ejegg) [19:42:01] * awight ponders whether pointing the power washer at the barbeque would cause some type of suburban singularity... [19:43:27] haha [19:43:40] (PS3) Ejegg: Requeue delayed messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) [19:46:33] (PS3) Ejegg: Required limit of number of retry messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 (https://phabricator.wikimedia.org/T142028) [19:47:07] (PS4) Ejegg: Requeue delayed messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) [19:47:55] (PS5) Ejegg: Requeue delayed messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) [19:48:37] XenoRyet: go for it! [19:48:42] thanks for looking [19:49:06] No worries. I kind of like SQL and I haven't had an excuse to look at it for a while. [19:49:16] (CR) XenoRyet: [C: 2] Fix incorrect contribution_recur rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [19:49:27] I'd like to deploy the code fix before I run the sql, just in case any recurring contribution records get saved in the meantime [19:49:38] Makes sense. [19:49:42] (CR) Awight: [C: 2] QueueConsumer always shunts to damaged datastore (1 comment) [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302633 (owner: Ejegg) [19:51:12] (PS3) Awight: Use a yaml reference for Redis queue params [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 (owner: Ejegg) [19:51:19] (CR) Awight: [C: 2] Use a yaml reference for Redis queue params [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 (owner: Ejegg) [19:51:53] (CR) jenkins-bot: [V: -1] Required limit of number of retry messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [19:52:14] (Merged) jenkins-bot: Fix incorrect contribution_recur rows [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [19:52:16] (Merged) jenkins-bot: QueueConsumer always shunts to damaged datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302633 (owner: Ejegg) [19:52:18] (CR) jenkins-bot: [V: -1] Use a yaml reference for Redis queue params [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 (owner: Ejegg) [19:52:42] ejegg: sorry to be so merge-happy. I'll hold off when I have suggestions... [19:52:48] (PS4) Ejegg: Required limit of number of retry messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 (https://phabricator.wikimedia.org/T142028) [19:53:12] hey, no worries! I don't mind making follow-on patches [19:53:39] (CR) jenkins-bot: [V: -1] Requeue delayed messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [19:54:02] (PS6) Ejegg: Requeue delayed messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302839 (https://phabricator.wikimedia.org/T142028) [19:55:16] (PS5) Awight: Required limit of number of retry messages [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303611 (https://phabricator.wikimedia.org/T142028) (owner: Ejegg) [19:55:44] (PS1) Ejegg: Update CiviCRM submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303618 [19:55:46] (PS8) Awight: Subclass BaseQueueConsumer instead of callbacks [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303081 (owner: Ejegg) [19:57:38] oh hey, we need this one to actually deploy the last two compare/update pending patches: [19:57:45] https://gerrit.wikimedia.org/r/303232 [19:59:51] awight / XenoRyet - are all the CiviCRM updates in https://gerrit.wikimedia.org/r/303618 kosher for deploy? [20:00:38] (CR) Cdentinger: [C: 2] Listener shouldn't talk to pending datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303232 (owner: Ejegg) [20:00:40] (CR) Awight: [C: 2] Subclass BaseQueueConsumer instead of callbacks [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303081 (owner: Ejegg) [20:00:53] As far as I know they are. [20:01:26] ejegg: looks mostly safe [20:02:00] Anyone know how the payments wiki on vagrant is configured wrt server paths? My vagrant debugger somehow isn't recognizing break points.... [20:02:23] I think it could be a server-side symlink ishiew [20:02:50] AndyRussG: yikes, are you familiar with multiversion? [20:03:07] (Merged) jenkins-bot: Listener shouldn't talk to pending datastore [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303232 (owner: Ejegg) [20:03:09] (Merged) jenkins-bot: Subclass BaseQueueConsumer instead of callbacks [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303081 (owner: Ejegg) [20:04:27] AndyRussG: in phpstorm, i have to set up path mapping to debug vagrant [20:05:21] Speaking of, my phpstorm settings got wiped out somehow. I need to be reminded which things need mapping to get that to work. [20:14:14] Fundraising Sprint Octopus Untangling, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Regression, Unplanned-Sprint-Work: Failmail code no longer recognizes comma-separated addresses - https://phabricator.wikimedia.org/T142420#2534830 (Danny_B) [20:26:16] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Merging Records manually in Civi - WMF Donor information doesn't automatically merge over - https://phabricator.wikimedia.org/T142008#2534847 (Eileenmcnaughton) @rlewis - sorry - yes that was what I meant [20:41:08] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Merging Records manually in Civi - WMF Donor information doesn't automatically merge over - https://phabricator.wikimedia.org/T142008#2534887 (RLewis) @Eileenmcnaughton I just did another manual merge and it doesn't seem right to me, especially the Last... [20:46:47] Fundraising Sprint Nitpicking, Fundraising Sprint Octopus Untangling, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-ActiveMQ: Migrate fredge to new queue - https://phabricator.wikimedia.org/T131273#2534905 (awight) I see that the remaining patches can't be merged yet, due to dependen... [20:47:31] (PS4) Ejegg: Use a yaml reference for Redis queue params [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 [20:47:55] (CR) Ejegg: [C: 2] Update CiviCRM submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303618 (owner: Ejegg) [20:48:55] fr-tech any objections to a CN deploy this week? it'd be the extension registration patch and also fixing a thing that's causing ucky DB warnings [20:49:19] AndyRussG: sounds good! [20:49:54] (PS6) Awight: WmfQueueConsumer to replace dequeue_loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/302975 (https://phabricator.wikimedia.org/T131277) (owner: Ejegg) [20:50:14] Fundraising Sprint Nitpicking, Fundraising Sprint Octopus Untangling, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Queue mirroring needs to copy source_* fields into the message body - https://phabricator.wikimedia.org/T141948#2534937 (cwdent) a:cwdent [20:50:41] awight: ah, that'll need a vendor update now that te smashpi stuff is merged [20:50:54] awight: ejegg so in theory if we merge this https://gerrit.wikimedia.org/r/#/c/290828/ & deploy all merged code [20:51:19] then we can run a batch of dedupes via drush - would we just run 20 & then look at the merged contacts? [20:51:30] or would we configure via jenkins first? [20:51:53] ah right, let's do it via jenkins for the logs [20:52:01] eileen: I've been doing experimental stuff in the "one off" jenkins job [20:52:06] easy enough to set that up though [20:52:13] ok - how does that work ? [20:52:22] ssh tunnel & just configure in the UI? [20:52:32] yep [20:52:49] (Merged) jenkins-bot: Update CiviCRM submodule [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303618 (owner: Ejegg) [20:52:54] ok [20:54:11] (CR) Awight: [C: 2] Drush merge command [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/290828 (owner: Eileen) [20:54:30] cool, I'll deploy that as soon as it merges in [20:56:06] awight: is this one also good to merge? https://gerrit.wikimedia.org/r/#/c/301325/ [20:56:22] (it's not important that it's merged now but just a tidy up) [20:56:56] (CR) Awight: "I really like the new handleError, it's self-documenting!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/302975 (https://phabricator.wikimedia.org/T131277) (owner: Ejegg) [20:57:01] oh, looks good [20:57:15] (CR) Awight: [C: 2] WmfQueueConsumer to replace dequeue_loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/302975 (https://phabricator.wikimedia.org/T131277) (owner: Ejegg) [20:58:12] (CR) Awight: [C: 2] Fix ProcessMessageTest to not fail on exchange rates. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/301325 (https://phabricator.wikimedia.org/T138542) (owner: Eileen) [20:58:53] (Merged) jenkins-bot: Drush merge command [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/290828 (owner: Eileen) [21:01:31] (PS2) Ejegg: Error checking [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300766 (owner: Awight) [21:01:50] (CR) Ejegg: [C: 2] Error checking [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300766 (owner: Awight) [21:01:57] I'm sitting in a cloud of yellowjackets... OSHA would have a fit [21:03:18] (CR) Awight: [C: 2] Use a yaml reference for Redis queue params [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 (owner: Ejegg) [21:04:02] woot - I got into jenkins 2nd go [21:04:09] :) [21:04:20] such a maddening bug [21:04:44] (Merged) jenkins-bot: WmfQueueConsumer to replace dequeue_loop [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/302975 (https://phabricator.wikimedia.org/T131277) (owner: Ejegg) [21:04:48] It has some kind of heuristics to make it more difficult to log in the more urgent it is to do so [21:04:54] it makes you try... [21:04:56] yeah [21:05:15] maybe it's measuring how hard you're typing [21:05:36] :-) [21:06:37] (Merged) jenkins-bot: Error checking [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/300766 (owner: Awight) [21:06:39] (Merged) jenkins-bot: Use a yaml reference for Redis queue params [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 (owner: Ejegg) [21:08:08] (CR) Ejegg: [C: -1] Failmail can unpack multiple "to" addresses (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [21:09:03] ejegg: so do we also need to run an update to fix incorrect currencies on recurring records? [21:09:32] (CR) Awight: Failmail can unpack multiple "to" addresses (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [21:09:44] eileen yep, I think this does it: https://gerrit.wikimedia.org/r/303608 [21:11:47] (PS3) Awight: Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) [21:12:25] (PS8) Ejegg: Fix ProcessMessageTest to not fail on exchange rates. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/301325 (https://phabricator.wikimedia.org/T138542) (owner: Eileen) [21:12:35] good to see those log tables coming in handy! [21:12:47] they were a life saver! [21:13:35] the connection ID matching the _recur update to the contribution insert pointed me in the right direction [21:13:57] So it looks like there are 1394 rows involved [21:14:20] I think there are a lot more [21:14:35] oh really - that's what I got on dev just now [21:14:43] it's a bit out of date I guess [21:14:44] it's been corrupting rows for a month [21:15:43] There is a test on core I thought might have caught it - but probably because the field is currency & that is dicey in the tests (due to set up issues) it missed it [21:20:23] (PS4) Ejegg: Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [21:20:25] (CR) Eileen: "This looks right to me - but we should just run it on staging first to check it isn't really slow to run" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [21:20:46] (CR) Ejegg: [C: 2] Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [21:21:38] (CR) Eileen: Fix incorrect contribution_recur rows (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [21:24:35] (Merged) jenkins-bot: Failmail can unpack multiple "to" addresses [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303601 (https://phabricator.wikimedia.org/T142420) (owner: Awight) [21:24:50] ejegg: you're still sitting on a CRM deployment? [21:25:06] just waiting for the failmail patch to merge [21:25:15] oh, there it is! [21:25:21] ty! [21:26:53] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/303695 [21:27:18] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/303695 (owner: Ejegg) [21:27:50] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/303695 (owner: Ejegg) [21:30:02] !log updated civicrm from 2d68638471aded73d05a796b05cab11809e31c56 to d9a765903ac682c0fbd329ced59f5ba3953970c9 [21:30:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:32:48] (CR) Ejegg: "checked on staging, insert got 19180 rows in less than 2 sec. About to run on prod" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303608 (https://phabricator.wikimedia.org/T142312) (owner: Ejegg) [21:35:12] I just saved a drowning yellowjacket... can I take it back? [21:35:24] is this correct ? http://localhost:9000/job/Dedupe%20CiviCRM%20contacts/configure [21:35:56] awight: drowning yellowjacket? [21:36:43] It was probably trying to fly away with my dog's kibble, found it taking an unsuccessful swim in the water dish [21:36:51] hmm I see that may not refer to a piece of clothing but possibly a comic strip character or a wasp... [21:37:11] (the latter is sounding more likely all the time) [21:37:15] ah some yankees call wasps "yellowjackets" [21:37:45] ok, subscriptions are fixed, and editing a child contribution does not mess up the currency [21:38:01] let's try running one recurring charge... [21:38:48] pick mine pick mine ;) [21:40:33] eileen: are you using the on-off job right now? [21:40:39] *one-off [21:40:53] I see what you did there [21:41:03] I'm not sure - I didn;t schedule it per se [21:41:19] looks like no [21:41:50] I just chose new job from the left & guessed at what to fill in.... [21:42:00] ah, cool [21:43:11] I think I just made it run - I clicked on the clock thingee [21:43:34] eileen: when you're ready, schedule for immediate execution with the "build now" button [21:43:52] ah yeah mebbe a clock also? [21:43:54] ok, JPY charged as JPY, let's see the recur record [21:44:07] hmm well I wasn;t ready but I seem to have set it going [21:44:35] still JPY! [21:44:52] It merged about 70 records [21:45:20] (CR) Awight: [C: 2] Fix documentation for CustomFiltersFunctions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301429 (owner: Ejegg) [21:45:52] and respecting our own next_sched_date [21:46:49] so for example civicrm/contact/view?reset=1&cid=225 is now one contact from 4 [21:46:59] (Merged) jenkins-bot: Fix documentation for CustomFiltersFunctions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/301429 (owner: Ejegg) [21:48:20] ok, looking good [21:48:37] just did 10 more, and haven't seen corruption yet [21:49:41] Take tomorrow off! [21:50:09] (PS2) Awight: Config shortcut for queue testing. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303299 (owner: Ejegg) [21:50:18] heh [21:50:40] i just might... [21:51:16] ok, turning the job back on! [21:53:44] !log enabled globalcollect recurring job [21:53:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:53:49] cool [21:54:30] cwd: no I don't know multiversion really... ejegg what server-side path do you use for path mapping for DI? [21:55:28] from /vagrant/mediawiki-fr/extensions/DonationInterface to where my code lives locally [21:55:32] (CR) Awight: "On second thought, there's got to be an opportunity for reusing the store server configuration at the logic level, for example: Each queue" [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/302636 (owner: Ejegg) [21:56:41] So, any thoughts on what checks we should do on the merged contacts? [21:57:43] if I search for 'Merge Contacts' activities from yesterday (because you guys are behind) I find around 96 [21:58:04] about 25 of those were manual ones [21:58:39] That's odd--I thought the Civi UI was in the local server time, not the admin's browser time [21:59:13] (PS1) Cdentinger: Add ActiveMQ headers to Redis messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303699 (https://phabricator.wikimedia.org/T141948) [21:59:42] no - I think you can change that through drupal config somewhat [21:59:49] eileen: I'll read the log tables for those contacts [22:01:18] cool [22:04:00] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Added new Tag to Production Civi - https://phabricator.wikimedia.org/T142439#2535128 (RLewis) [22:04:09] awight I've found something I don't like. civicrm/contact/view?reset=1&cid=172 had a home address on the original record & then on later records the home address was just her country [22:04:25] but because they were more recent contributions they got priorities [22:04:45] ejegg|brb: thx... right, I have that, basically... Or the same, but just up to /vagrant/mediawiki-fr, since I have all of mediawiki-fr as my Eclipse project [22:05:05] Hmmm maybe server mapping isn't the problem=? [22:05:09] dang [22:05:43] well, you could sniff the xdebug port with wireshark and see what paths it's passing back and forth [22:06:02] eileen: should that be marked as a conflict for now, then? [22:06:41] awight: yeah or else we say 'if there is just a country then it is not a better address' [22:07:48] at this stage our rules prioritise a 'new' home address - but they are not filtering country-only home addresses [22:08:13] fwiw we have a similar problem with say IPN vs audit vs frontend donationinterface messages--some data is better than others, and what we usually want is to refine the contact information, not really to "update" it. [22:08:37] awight: and eileen I vote for marking things and a conflict for early runs and slowly adding complexity [22:08:58] awight: ooh, if you were about to merge this I'll get some tests going in the crm consumers: https://gerrit.wikimedia.org/r/303299 [22:09:22] eileen: So remind me, it's intentional that log_conn_id is different for every contact, right? That was part of the undo work? [22:09:39] awight: yeah - now you want to know if you can undo all at once :-) [22:09:46] it's in our hook that we set that [22:10:19] I'll look at the log table but I doubt there is more than 10 contacts affected by the address change [22:10:40] cool--so 16-char hex is the new norm. ejegg said there are still processes logging with the 7.6 \d though? [22:11:02] (CR) Ejegg: [C: -1] "Yeah, we definitely don't want all the headers, just basically push https://github.com/wikimedia/wikimedia-fundraising-crm/blob/d1e6167bde" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303699 (https://phabricator.wikimedia.org/T141948) (owner: Cdentinger) [22:16:49] (PS1) Ejegg: Update SmashPig and PHP-Queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303701 [22:19:18] ejegg: heh that's an idea [22:19:23] eileen: log rows contain the data after an update, not before it, right? [22:19:36] yes [22:19:51] heh, i really resorted to that when i was perplexed [22:20:19] (PS2) Cdentinger: Add ActiveMQ headers to Redis messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303699 (https://phabricator.wikimedia.org/T141948) [22:20:32] That's unfortunately though, cos it means I can't see the history of e.g. civicrm_email id=5898918 [22:20:37] ^ly^ [22:21:04] * awight checks staging backup [22:21:25] well you can see what it was before as that should be there as a separate transaction [22:23:01] it happened to not be, for the thing I was looking at is all [22:23:14] Probably because the (duplicate) contact dates to 2013 [22:24:34] hmm - but when the logging was turned on there should have been an entry created for every row with the log_action = 'Initialize' [22:26:03] eileen: ah thank you--my query was filtering those out because the email contact_id was changed [22:27:01] ah right [22:27:25] So, if you look at contact 247 that is an example of where we DID purposely select the later home address [22:27:37] (from his 2012 donation) [22:28:12] - so far I've only found the one merge where the address decision was wrong - & that was the country thing [22:28:14] eileen: civi-merge could use some more logging... [22:28:27] or maybe it's going into a log stream that Jenkins doesn't capture? [22:28:56] (Also, the undefined indexes are a bit unnerving) [22:29:15] no probably not - the main 'logging' is the activities created I guess [22:30:03] I'll look at those indexes [22:30:22] probably not going into aa log stream that Jenkins doesn't capture I mean [22:30:23] that works. I wonder if we could add a hook to copy new activities to stdout... one day [22:30:36] (CR) Ejegg: "Maybe have buildBody and buildHeaders call the same function to set the source_ stuff? It'll make things nicer when we get rid of the sto" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303699 (https://phabricator.wikimedia.org/T141948) (owner: Cdentinger) [22:36:21] eileen: It's interesting that the merge is updating rows before deleting them... not sure if that makes sense [22:37:04] I guess it must do 2 saves [22:38:58] it looks like 44 contacts exist now who 'took part in' merges from the script [22:39:14] I'm going through & checking the addresses & making notes here https://docs.google.com/document/d/1nHweqxONiolc-TLpd7jqx1w_gY1pFUUvY51Uzl7jCWw/edit [22:40:21] * awight tugs at collar as geolocations scroll by [22:40:31] I should mention that to Aeryn... [22:43:18] eileen: We're always taking the newer address, for non-major-donors, right? [22:43:42] awight: yeah [22:43:58] cool [22:46:29] eileen: okay, the update->delete seems sort of reasonable--we're setting the contact_id to the new contact before deleting the rejected, linked entity [22:46:46] Too bad, cos I was thinking it would be an easy 50% db load improvement. [22:47:27] Are we also deduping deleted contacts? If not, maybe it's more consistent to eliminate the update [22:48:21] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM merge should not update before deleting record - https://phabricator.wikimedia.org/T142441#2535229 (awight) [22:48:28] dstrine: ooh ty for disabling the fr-tech-backlog tag [22:48:52] I just had my first pleasant "fu ba" experience in months [22:50:57] (PS34) Ejegg: Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [22:51:33] (CR) Ejegg: "PS34: rebased around test folder move" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [22:51:49] eileen: What's wrong with 150? The addresses seem correct to me, newest one preferred [22:52:31] (CR) jenkins-bot: [V: -1] Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [22:52:34] awight: yeah that might actually be the right outcome [22:52:47] originally he had 2 addresses Mailing + Home [22:52:58] & I thought one would be kept [22:53:43] in merge 1 the mailing address was dropped (as an exact duplicate of the home) [22:54:09] the is_deleted column downright hurts my brain now [22:54:25] & on merge 2 the home address was replaced by the new home address [22:54:34] ooh I see, tricky [22:54:42] which is almost certainly correct in this case [22:54:54] Didn't realize merging is always pairwise [22:54:57] that's a solid way to do it [22:54:57] ie. he moved & eventually wound up with only one address at the new place [22:55:05] (PS35) Ejegg: Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [22:55:08] yep it is [22:58:05] ah so schema updates are applied to the log_* tables now? [22:58:13] (CR) Ejegg: [C: 2] "This looks ready for an IRL test! Love that it has unit tests now." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [22:58:38] * awight jumps at alarms [22:59:18] awight: yeah fr-tech backlog is no more :) [22:59:41] (Merged) jenkins-bot: Convert orphan rectifier to use the PendingDatabase [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/300173 (https://phabricator.wikimedia.org/T131275) (owner: Awight) [22:59:54] yeah whenever you turn logging on & off (like an upgrade script does) it reconciles the log_ tables [23:00:28] eileen: This looks really promising--and I'm suddenly optimistic that our data is pretty decent quality to begin with [23:00:35] & will only get better [23:01:13] eileen: oh wow, that sounds scary. Would it be possible to lose columns or see destructive casting? [23:02:10] log tables only add columns on reconcile - not delete [23:02:24] thank you for the reassurance [23:02:39] I think it would throw a wobbly if you tried to change the column type to something that did not work on existing data [23:03:42] donno--mysql will only emit warnings, which are ignored by default [23:03:54] I think CiviCRM errors on those [23:04:30] ! That would be great [23:04:31] I'm feeling OK about the cumulative merge I think - I found another example where the end result seemed fine - [23:05:09] Also, I do tend to check the sql commands in any civicrm upgrade [23:05:12] aww cute--I just noticed the "note" column on early donations [23:05:46] eileen: and awight I'm a bit scattered today. You ran at least one dedupe test right? Is that what you are discussing right now? [23:05:56] y [23:06:10] how was it? [23:06:14] It's looking sturdy! [23:06:19] yay! [23:06:21] cwd / XenoRyet: got a sec to review a convenience patch for SmashPig? [23:06:34] Sure. Whatcha got? [23:06:36] https://gerrit.wikimedia.org/r/303299 [23:06:36] dstrine: yeah - going through the merged contacts - there is only one thing that is bothering me which is that it chose a later address where there was only a country in the address [23:07:29] (CR) Ejegg: [C: -1] "waiting on Ie82fd9c50afbde3d8ee7c8a1a5fdfaa285d246d4" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303701 (owner: Ejegg) [23:07:40] eileen: I wonder what the quick fix is for that, though. Medium-term solution is something like a health score for each address, and conflict if the score is going to decrease [23:08:01] yeah - there seems to be a pattern of 'only the country; [23:08:07] (CR) XenoRyet: [C: 2] Config shortcut for queue testing. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303299 (owner: Ejegg) [23:08:09] which will be the most common one [23:08:10] But I don't see an obvious workaround for going forward with the pilot run, other than conflicting on a ton of nearly identical addresses. [23:08:24] k I fully support special-casing the edge cases [23:08:29] eileen: ok yeah that is the part I saw earlier. I'm all for calling that a conflict and not merging ... unless you have a quick fix. I feel like we should get this running and then tweak it. The longer this runs the better off we will be [23:09:04] (Merged) jenkins-bot: Config shortcut for queue testing. [wikimedia/fundraising/SmashPig] - https://gerrit.wikimedia.org/r/303299 (owner: Ejegg) [23:09:09] I'm not sure calling it a conflict is a quicker solution - the issue is defining the edge case where 'take most recent' does not apply [23:09:20] ok [23:09:58] It's a difficult triage, in theory I would fall on the dstrine side, where we should just skip those cases for the first run [23:11:18] eileen: what are you thinking about the other cases where the address is inferior? I don't think you can hardwire the country case and then do nothing for the other cases, cos the default is to overwrite [23:11:57] awight: an obvious inferior check is 'no street address' [23:12:24] if we have a street address then it shows some level of data entry [23:12:31] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops, FR-ActiveMQ, and 3 others: [Epic] SPOF: Replace ActiveMQ donation queues with a more robust software stack - https://phabricator.wikimedia.org/T108229#2535316 (Ejegg) [23:12:33] Fundraising Sprint Muggle Baiting, Fundraising Sprint Nitpicking, Fundraising Sprint Octopus Untangling, Fundraising-Backlog, and 3 others: Provision Redis cluster for Fundraising - https://phabricator.wikimedia.org/T130283#2131633 (Ejegg) Open>Resolved We are using this! [23:12:58] but for the damaging updates that aren't caught by whatever address quality heuristics you write--those get overwritten with no recourse, which seems wrong for the pilot run [23:14:01] right - so we can back out of doing the whole 'take the latest' [23:14:31] which from a code point of view is basically replace 'take the latest when not major gifts' to 'take the latest when 1=2' [23:14:44] hehe. k and those would be a conflict? [23:15:12] yeah - I think we would reduce out good catch rate by over 50% [23:16:20] Maybe we should talk about the strategy for making improvements to the dedupe script [23:16:30] like, are we going to let the first run complete before deploying v2? [23:16:56] I think we should steer away from a situation where we're invested in each iteration being "complete" [23:16:59] (PS2) Ejegg: Update SmashPig and PHP-Queue [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303701 [23:17:44] yeah I was imagining iterative updates would go out which it was going through & when we got the end we'd let it have a whole nother run-through [23:19:21] (PS14) Ejegg: Move banner history off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/301660 (https://phabricator.wikimedia.org/T141555) [23:19:57] eileen: and awight This script will run all the time in the background eventually right? [23:20:56] (CR) jenkins-bot: [V: -1] Move banner history off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/301660 (https://phabricator.wikimedia.org/T141555) (owner: Ejegg) [23:21:16] eileen: That sounds cool. Are there any assumptions about lower IDs being processed first, which would make it bad to start deduping with new algorithms later in the sequence? [23:23:05] yeah - it takes the first x number of contacts & looks for contacts that match them - so it is starting from a lower number [23:23:16] & yeah dstrine eventually this would always run [23:24:05] eileen: And it only searches through contact_id > mine? [23:24:49] ? [23:25:05] it grabs say contacts 1-20 & looks for any contact that matches them [23:25:31] but once you iterate on the script and criteria, it'll be at contact_id=1M, for example [23:25:49] is it going to be a problem that we're starting a more liberal dedupe from the higher contact number? [23:27:09] I guess not, cos dedupe is meant to be run on an arbitrary contact [23:28:35] yeah & also if contact 5 & 5555555 match there will be 2 attempts to dedupe them [23:29:22] ok perfect [23:30:15] I've gone through most of the contacts merged & I found one contact where the address was inferior because there was only a country [23:30:32] and one where the address was inferior because there was only country & state [23:31:12] if what we are saying is 'lets switch to the major gifts behaviour' - many many more conflicts - for all dedupes then maybe I should run a test on the major gifts group only? [23:33:44] That makes sense anyway, to give us a chance to focus on their special issues [23:37:12] (PS1) Ejegg: Use SmashPig config shortcuts, reset Context [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303717 [23:38:39] Either way, I guess plan "A" was to run with this possibly damaging address update logic, so we should circle back with ccogdill and MBeat to see what they think [23:40:20] It seems like the options are to either * make it conflict on almost everything, because people move and misspell, or * write an address score thing and conflict if the newer address might be damaging [23:41:13] My preference is towards a general score rather than just missing country, cos I expect that there will be waves of bad data over the history of donations, and the form of these glitches will be impossible to predict head of time. [23:41:38] (PS15) Ejegg: Move banner history off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/301660 (https://phabricator.wikimedia.org/T141555) [23:41:48] We can special-case "only country" and give it a score of 0.01 or whatever, but yeah I think it's important to generalize [23:43:04] I gotta run for the day but I will be talking to MBeat and Cogdill tomorrow morning. Now that we have data I think e can work out some of this logic with them. [23:43:25] OK cool [23:44:00] There are 3 contact that I don't think it made the right choice on wrt inferior addresses - should I fix now or wait in case people want to look [23:44:22] +1 leave them for now and undo later [23:44:34] lol I was going to say the opposite [23:44:40] haha [23:44:46] (PS3) Ejegg: Move antifraud queue off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303316 (https://phabricator.wikimedia.org/T131273) [23:44:52] (PS3) Cdentinger: Add ActiveMQ headers to Redis messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303699 (https://phabricator.wikimedia.org/T141948) [23:45:01] I'm not sure whether those votes add up in different directions or cancel [23:45:07] I say undo them so we don't forget. [23:45:12] ok [23:45:25] wfm, screenshots or whatever will be fine. [23:45:28] Just not in Phabricator :p [23:45:36] lol yeah [23:45:47] (PS2) Ejegg: Move payments-init queue off ActiveMQ [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303318 (https://phabricator.wikimedia.org/T131273) [23:46:36] (CR) jenkins-bot: [V: -1] Add ActiveMQ headers to Redis messages [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/303699 (https://phabricator.wikimedia.org/T141948) (owner: Cdentinger) [23:48:40] I need to tidy up before the family comes home, but can be online later if it's helpful to have a rubber duck... [23:50:15] :-) [23:53:18] (PS1) Ejegg: Don't crash Mailer on missing locale [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/303718 [23:54:09] so long, all! [23:54:17] Catch ya later [23:54:23] Fundraising-Backlog, Operations, Ops-Access-Requests: Access request: AWight access to iridium - https://phabricator.wikimedia.org/T142446#2535387 (awight) [23:55:20] woot, civi core patch merged!