[00:47:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: "The" issue: Contacts with dup records (with and without "The") - https://phabricator.wikimedia.org/T122264#1899850 (CaitVirtue) NEW [00:52:41] oh hey [00:57:12] * cwd waves [01:04:17] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Translation tool showing zero translations in MessageGroupStats - https://phabricator.wikimedia.org/T122251#1899889 (awight) [01:04:38] Anything fun happen today? [01:06:31] silly question. [01:07:56] some last minute xmas shopping! [01:07:58] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM bounce import job is failing - https://phabricator.wikimedia.org/T122265#1899895 (awight) NEW [01:12:18] I'm jealous. Christmas presents are like kryptonite to me [01:12:26] ... with this job, at least [01:13:35] i wish i could opt out of presents [01:14:52] It's totally an arms race. [01:19:02] (CR) Awight: [C: 2] "Thanks! We should keep an eye on the total execution time, and whether we see a jump in deadlocks from other jobs." [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/260622 (https://phabricator.wikimedia.org/T120899) (owner: Ejegg) [01:21:15] (Merged) jenkins-bot: Persist intermediate Silverpop export tables [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/260622 (https://phabricator.wikimedia.org/T120899) (owner: Ejegg) [01:29:55] (CR) Cdentinger: [C: 2] Fix filter summary for text functions [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/258718 (owner: Ejegg) [01:30:11] (CR) Cdentinger: [C: 2] Update gulpfile, hush up jslint [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/258717 (owner: Ejegg) [01:32:25] (Merged) jenkins-bot: Fix filter summary for text functions [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/258718 (owner: Ejegg) [01:32:29] (Merged) jenkins-bot: Update gulpfile, hush up jslint [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/258717 (owner: Ejegg) [01:55:01] augh. Moderated lists. [04:38:46] Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: Globalcollect Status 25/404 errors - https://phabricator.wikimedia.org/T120030#1900152 (awight) Yes, MBeat33 has given us a lot of information and it would probably b... [05:02:42] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Translation tool showing zero translations in MessageGroupStats - https://phabricator.wikimedia.org/T122251#1900164 (awight) I can reproduce the bug locally. * Install CentralNotice and Translate extensions. * Up... [05:51:20] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Translation tool showing zero translations in MessageGroupStats - https://phabricator.wikimedia.org/T122251#1900299 (awight) So far, it looks like the thing that's failing is the TranslateEditAddons::onSave handl... [06:00:08] Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate: Translation tool showing zero translations in MessageGroupStats - https://phabricator.wikimedia.org/T122251#1900313 (Jalexander) >>! In T122251#1900299, @awight wrote: > So far, it looks like the thing that's fai... [06:14:29] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, Unplanned-Sprint-Work: Translation tool showing zero translations in MessageGroupStats - https://phabricator.wikimedia.org/T122251#1900324 (awight) [06:14:43] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, Unplanned-Sprint-Work: Translation tool showing zero translations in MessageGroupStats - https://phabricator.wikimedia.org/T122251#1900325 (awight) p:Triage>Unbreak! [06:38:53] atgomez: don't watch, I just pulled in a new unbreak now! bug... [10:26:22] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, Unplanned-Sprint-Work: Translation tool showing zero translations in MessageGroupStats for CentralNotice banner - https://phabricator.wikimedia.org/T122251#1900502 (awight) [10:56:58] (PS1) Awight: Fix Translate integration [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/260738 (https://phabricator.wikimedia.org/T122251) [10:58:19] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: Translation tool showing zero translations in MessageGroupStats for CentralNotice banner - https://phabricator.wikimedia.org/T122251#1900554 (awight) a:awight [11:08:58] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: Translation tool showing zero translations in MessageGroupStats for CentralNotice banner - https://phabricator.wikimedia.org/T122251#1900569 (awight) My patch should fix... [12:02:46] (CR) Nikerabbit: [C: -1] Fix Translate integration (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/260738 (https://phabricator.wikimedia.org/T122251) (owner: Awight) [15:01:19] Fundraising-Backlog: Add Discover card type to Canada iframe options - https://phabricator.wikimedia.org/T122218#1900927 (Ppena) [15:35:03] For the record "Fundraising Sprint Zapp" is an amazing name especially at the end of the year [16:26:32] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, MediaWiki-extensions-Translate, and 2 others: Translation tool showing zero translations in MessageGroupStats for CentralNotice banner - https://phabricator.wikimedia.org/T122251#1901047 (Jalexander) Dangerously I ju... [17:08:06] (CR) Awight: Fix Translate integration (1 comment) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/260738 (https://phabricator.wikimedia.org/T122251) (owner: Awight) [17:08:34] (PS2) Awight: Fix Translate integration [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/260738 (https://phabricator.wikimedia.org/T122251) [17:08:38] ejegg: is this needed for adyen "campaign ready"? https://phabricator.wikimedia.org/T122244 [17:10:36] dstrine: IMO it's a "strong soft spot for", we would be taking a larger than usual fraud risk without AVS enabled. [17:11:37] ejegg: n.b. adding the NDA project doesn't do anything [17:12:09] I don't quite understand "strong soft spot" I just want to know if I should add this to the epic or make a "post release" epic for adyen. atgomez any opinions^^ [17:12:28] dstrine: PPena thinks it's essential [17:12:46] +1 [17:13:30] also, this is a thing we usually don't discuss publicly [17:13:43] Fundraising Sprint Zapp, Fundraising-Backlog, Unplanned-Sprint-Work, FR-Smashpig: Don't delete Adyen audit files - https://phabricator.wikimedia.org/T121695#1901134 (DStrine) [17:13:44] Fundraising-Backlog, Epic: [epic] Adyen campaign ready - https://phabricator.wikimedia.org/T118202#1901132 (DStrine) [17:13:49] ejegg awight dstrine IMO yes, you should add dstrine. If affects fraud rates and processing costs [17:14:28] * awight checks which side of the bed I woke up on [17:14:41] my cat just typed ^S -- this is gonna be a long day [17:15:54] It's on the epic. Thanks all [17:16:25] atgomez: any opinion on where this should go in the backlog? https://phabricator.wikimedia.org/T122218 [17:16:38] * awight ponders what uses we might have for a cat that has mastered flow control [17:17:52] I'm so stoked that we got the analysis column to zero! we can really monitor incoming tasks now [17:18:21] dstrine: atgomez: fwiw, I kept myself entertained all night by making this an unbreak now: https://phabricator.wikimedia.org/T122251 [17:20:33] Fundraising-Backlog, fundraising-tech-ops: fr-log-announcer should not hang out in our private channel - https://phabricator.wikimedia.org/T122104#1896770 (awight) Putting this back on the calendar, cos it's a simple and necessary security thing. Currently, a naughty person could impersonate our bot and... [17:21:50] Fundraising Tech Backlog, Fundraising-Backlog, § Fundraising Sprint Abba: Make DI forms visually match what's on donate wiki - https://phabricator.wikimedia.org/T86086#1901187 (awight) [17:24:30] AndyRussG: Hi! [17:24:38] awight: hey :) [17:24:40] AndyRussG: If you have a minute, this should be fun, https://gerrit.wikimedia.org/r/#/c/260738/ [17:25:41] awight: ah great! Yeah I'll check it out :) [17:26:12] I rushed the patch out cos they're trying to prepare for a Jan 15 thing. [17:26:19] ... in every language [17:27:58] ext:Translate kicks all of my butt. I spent 3 hours printf'ing through that code before I got back to CN to find that the bug was on our side [17:52:32] I think this fix is a win, but I'm pretty nervous about: how did this ever work [17:53:52] hey dstrine [17:54:09] * dstrine waves [17:54:24] Fundraising-Backlog: Add Discover card type to Canada iframe options - https://phabricator.wikimedia.org/T122218#1901334 (atgo) @ppena are you saying that this isn't actionable until GC responds? [17:54:36] hey awight - you have weird hobbies :P [17:54:49] awight: do you know how exactly refunds are supposed to look in the new Civi schema? [17:55:05] is there supposed to be a single contribution now? [17:59:06] atgomez: hehe that was billable [17:59:40] ejegg: Yeah, they will be two financial items, under a single contribution [18:00:00] awight: ok, so we need to fix wmf_mark_refund [18:00:11] ejegg: pls wait for eileen, she's actively doing things in there [18:00:14] but yeah, I agree [18:00:30] I'm still getting a second negative contribution, along with the extra financial trxns under the original [18:00:36] If you want to create a new-style refund, edit a contribution and just change the status to "refunded" [18:00:47] that sounds bad. Is this deployed? [18:01:07] awight: it's the way the code's been all along [18:07:06] awight: damn! Sounds like that was rough [18:09:57] awight / AndyRussG / XenoRyet : anything for Scrum of Scrums? [18:10:08] Nope, I'm good [18:10:11] ejegg: nothing here, thanks! [18:12:29] cool [18:22:58] Fundraising-Backlog: [BUG] GC100 Auth Fail Errors 10/18 very high fail rate - https://phabricator.wikimedia.org/T115818#1901382 (MBeat33) @atgo, GC confirmed this was a Wells Fargo issue. [18:29:21] Fundraising-Backlog: [BUG] GC100 Auth Fail Errors 10/18 very high fail rate - https://phabricator.wikimedia.org/T115818#1901414 (atgo) Open>Invalid a:atgo [18:32:43] K4-713: 1:1? [18:33:07] atgomez: Yep. BRT. [18:34:18] Fundraising-Backlog: Add Discover card type to Canada iframe options - https://phabricator.wikimedia.org/T122218#1901424 (Ppena) @atgo yes! :) [18:38:15] AndyRussG: They really need some pictures explaining how the pieces fit together... [18:39:17] awight: wrt translation stuff? [18:39:29] awight: also, BTW did you install translation ext locally? Or with vagrant? [18:40:24] AndyRussG: yeah--I installed locally, the only trick is to set that wgNoticeUseTranslateExtension global [18:40:38] Ah, and UniversalLanguageSelector is a dependency [18:42:28] awight: cool thx! [18:45:14] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review: DonationInterface should log client-side errors - https://phabricator.wikimedia.org/T121800#1901482 (Ejegg) look into this: https://www.mediawiki.org/wiki/Extension:Sentry [18:46:35] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/260778 [18:47:02] (CR) Ejegg: [C: 2 V: 2] Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/260778 (owner: Ejegg) [18:51:45] !log updated fundraising dashboard from 59e51c4ff74c3c584daf6c5de3bb66daa764cd28 to af8a493ab9ac5431e0d294e5019ac4e426ac6e08 [18:51:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:53:30] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/260780 [18:53:43] (CR) Ejegg: [C: 2 V: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/260780 (owner: Ejegg) [18:55:10] !log updated fundraising tools from ebed29c0eccf38c812b20a957b3487a15bfa9cbc to 1bc23cb4bfaf2a9d4d215aad79dd67d891b5d973 [18:55:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:05:38] ok, I'm going to try running the silverpop export with persistent tables and make sure it doesn't lock all the tables [19:06:51] wheee! [19:09:41] cool, donation queue consumer seems to be doing its thing just fine [19:18:51] (CR) Ejegg: "In kvStoreMaintenance, how about replacing the check at the start of removeExpiredItemsWhenIdle with a call to isAvailable? You shouldn't " [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/259986 (https://phabricator.wikimedia.org/T121738) (owner: AndyRussG) [19:45:11] (CR) AndyRussG: "> In kvStoreMaintenance, how about replacing the check at the start of removeExpiredItemsWhenIdle with a call to isAvailable?" [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/259986 (https://phabricator.wikimedia.org/T121738) (owner: AndyRussG) [19:53:18] (PS4) Ejegg: KVStore: Wrap accesses to LocalStorage in try/catch blocks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/259986 (https://phabricator.wikimedia.org/T121738) (owner: AndyRussG) [19:54:15] (CR) Ejegg: [C: 2] "Ahh, very thoroughly thought through." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/259986 (https://phabricator.wikimedia.org/T121738) (owner: AndyRussG) [19:54:33] ejegg: thx! [19:55:26] my pleasure [19:56:26] hi, just watned to let you know that on "barium", the FR civicrm server [19:56:40] there is a warning that there are 12 zombie processes [19:56:47] (Merged) jenkins-bot: KVStore: Wrap accesses to LocalStorage in try/catch blocks [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/259986 (https://phabricator.wikimedia.org/T121738) (owner: AndyRussG) [19:56:49] probably not urgent but wanted to mention it [20:02:46] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, and 2 others: Track email clickthroughs on donate wiki - https://phabricator.wikimedia.org/T114010#1901741 (Ejegg) Hi @CCogdill_WMF, Are the numbers looking closer to Silver... [20:07:24] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, and 2 others: Track email clickthroughs on donate wiki - https://phabricator.wikimedia.org/T114010#1901758 (CCogdill_WMF) @Ejegg there was a rogue link with no contact_id in... [20:08:25] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, and 2 others: Track email clickthroughs on donate wiki - https://phabricator.wikimedia.org/T114010#1901762 (CCogdill_WMF) But to answer your question - yes, the numbers are l... [20:09:19] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, and 2 others: Track email clickthroughs on donate wiki - https://phabricator.wikimedia.org/T114010#1901763 (Ejegg) Ahh, sorry the setup is more complicated than the Silverpop... [20:14:32] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, and 2 others: Track email clickthroughs on donate wiki - https://phabricator.wikimedia.org/T114010#1901779 (CCogdill_WMF) Not a problem, this is a temporary hassle for a long... [20:16:06] Hi eileen ! [20:16:46] hey ejegg [20:16:51] I think we need to fix our wmf_civicrm_mark_refund. I'm trying to set up test data for your lybunt patch, and getting funny-looking results [20:17:17] I imported some refunds via our offline2civicrm drush import-refunds, which uses that wmf_civicrm fn [20:17:34] and ended up with a second negative contribution for each [20:17:59] as well as the extra financial_trxn items associated with the original contribution [20:18:03] ejegg: this is the gateway report? [20:18:13] Did you read my essay on the phab ticket? [20:18:14] oh right, gateway not lybunt [20:18:22] nope, let me look! [20:18:26] https://phabricator.wikimedia.org/T116317 [20:18:31] thanks [20:18:47] It's in the description field & you should get coffee before you start [20:19:29] in other news I just replied to someone one another channel & mistyped g for f when typing buffered [20:21:40] hehe [20:26:51] double :D [20:31:04] here is the quote… [20:31:08] routinet: that broadly makes sense to me but you really want to run it past Tim - I'm not actually familiar with why you would prefer unbuggered [20:32:00] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, and 2 others: CiviCRM upgrade: Adapt refund processing & reporting to reflect changes since the upgrade. - https://phabricator.wikimedia.org/T116317#1901839 (Ejegg) One more thing: Some pro... [20:32:10] buggered is always butter [20:34:41] ... in the end [20:39:26] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, and 2 others: CiviCRM upgrade: Adapt refund processing & reporting to reflect changes since the upgrade. - https://phabricator.wikimedia.org/T116317#1901866 (Eileenmcnaughton) We are curren... [20:41:43] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, and 2 others: CiviCRM upgrade: Adapt refund processing & reporting to reflect changes since the upgrade. - https://phabricator.wikimedia.org/T116317#1901873 (Ejegg) Yep, that's currently in... [20:42:43] ejegg: what does "(also gateway -> payment processor) " mean [20:43:11] eileen: we're storing the processor name in that custom field to [20:43:13] *too [20:43:36] and it looks like there's a standard payment_processor_id field in the financial transaction table [20:44:23] I'll make another ticket to investigate using more standard fields [20:44:39] ejegg: yes & a payment_processor table…. if we want to do refunds direct from CIvi at some point we would need to store the processors better [20:44:43] (CR) Awight: [C: 2] "Now, to translate it to Australian ;-)" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/260621 (https://phabricator.wikimedia.org/T119912) (owner: Ejegg) [20:44:54] heh, thanks awight! [20:48:25] Fundraising-Backlog, Epic: [epic] Adyen campaign ready - https://phabricator.wikimedia.org/T118202#1901906 (awight) p:Triage>High If GC goes down, we still lose... millions of dollars. Increasing priority of the backup work. [20:49:40] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM upgrade: migrate custom fields to standard fields - https://phabricator.wikimedia.org/T122342#1901910 (Ejegg) NEW [20:53:28] (Merged) jenkins-bot: Warn BPay donors not to recycle reference numbers [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/260621 (https://phabricator.wikimedia.org/T119912) (owner: Ejegg) [20:54:33] ejegg: did you have any thoughts about whether it would otherwise matter that if we only have one contribution we only have one copy of fields like source_type' 'source_name' 'source_host' 'source_version' 'source_run_id' 'source_enqueued_time' [20:55:01] that's a problem [20:55:29] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, and 2 others: CiviCRM upgrade: Adapt refund processing & reporting to reflect changes since the upgrade. - https://phabricator.wikimedia.org/T116317#1901941 (Eileenmcnaughton) [20:55:39] awight: what is that data used for? [20:55:44] ooh, yeah, that needs to be recorded distinctly for the refunds [20:55:48] OTOH, my long-term wish for that sort of audit data is that we keep a history of events, rather than keeping a single set of properties [20:55:56] eileen: auditing imports, basically [20:55:59] We can do that work any time... [20:56:19] imports in the broadest sense, anything coming into Civi [20:56:43] awight: good call, like an audit table with entity type, id, operation, and maybe old data? [20:56:48] as auditing how well imports are working, or the audit of import files [20:57:22] just so we know where the imported contribution came from [20:58:06] e.g. if we found out the people that import checks had bad data/instructions [20:58:20] we could go back and look at all the things imported by those folks [20:59:02] So, in that case the 'who' is key [20:59:46] (That might potentially be helped by turning on CiviCRM logging ) [21:00:02] ejegg: but more than just imports, another major use is to tell whether contributions came through DI, audits, or a realtime listener [21:00:29] also, to audit which contributions were affected by buggy code revisions [21:01:20] right on [21:01:29] hmm - so the issue here is that CiviCRM has basically moved in theory to the contribution record being separate from the payment record [21:01:47] so the payment records are the civicrm_financial_trxn table now [21:01:48] yeah, I imagine this theoretical audit history could tell us if a contribution came in through multiple sources, and maybe a third event when we merge the data [21:01:56] right [21:02:07] but, the data we want to store is specific to the payment, rather than the contribution [21:02:32] exactly, looks like we need to split up wmf_contribution_extra [21:02:34] I'm okay with eating that, temporarily [21:02:52] As long as we don't lose so much data that we can't search in logs [21:03:31] Seems like civicrm_financial_trxn.trxn_id gets us over that threshold, at least. [21:03:32] hmm - right but that is temporarily … do we have a longer term plan [21:03:46] I do not [21:03:56] nor i [21:03:59] :) [21:04:02] We do still have the 2 options I mentioned on https://phabricator.wikimedia.org/T116317 [21:04:16] which are 'Co-operate with CiviCRM' & 'Tell CiviCRM to stick it' [21:04:51] I feel like #2 sets us up for horrible upgrade problems next time around [21:04:52] The issue with the former appears to be we want to store custom data against the financial transactions themselves [21:05:07] yeah, it's going to take a bunch of adaptation [21:05:21] yeah - I'm instinctively against it & haven't to force myself to give it a fair hearing :-) [21:05:21] 100% option #1, right [21:05:22] ? [21:05:45] awight: yah, totally [21:06:08] So, one option in terms of #2 is that we could actually extend the financial_trxn entity to support custom data [21:06:35] (I think this is possible as a core contribution - I could check that ) [21:07:08] At that scale of effort, I'd rather fix our broken 1:1 audit assumption [21:07:35] looks like parent_contribution_id and no_thank_you are the only wmf_contribution_extra fields that apply to the contribution rather than the payment [21:07:41] eileen: Do you think Civi would want this type of audit trail upstreamed? [21:07:49] I guess the effort involved on adding to core is mostly a) discussing, b) a small tweak to add bao support, c) some form work [21:07:59] ejegg: That's perfect, those aren't necessary for refunds [21:08:19] I'm not suggesting adding OUR fields at this stage - just the capability to add custom fields against that type of entity [21:08:27] sure [21:08:33] sorry I conflated the two things [21:08:37] eileen: definitely should be extendable [21:08:41] extensible? [21:08:54] I don't think custom data on the line items is tremendously valuable [21:09:07] But I believe the audit history will be really great [21:09:14] not line items, financial transactions (line items also has a meaning) [21:09:19] sorry! [21:09:25] Hmm, I still need to wrap my head around those distinctions... [21:09:34] when you say the audit history will be really great - what do you mean by that [21:09:39] will re-read your essay [21:09:50] line-items = the line-items on an order - I didn't go into those too much [21:09:51] so, this is only tangential to the problem at hand [21:09:59] ... looking for task [21:10:02] financial_transactions roughly translates to payments [21:10:12] https://phabricator.wikimedia.org/T116317 [21:10:22] awight: are you thinking of audit tables that will be generic enough to work for the contact merge too? [21:10:47] or distinct tables for auditing different entities? [21:11:46] So, the CiviCRM core logging implementation (which is an option we should turn on on staging but need a mysql perm to do so to create triggers) works like this : [21:12:01] For each table a table called log_tablename is created [21:12:13] by default it is an archive table with no indexing [21:12:31] & triggers are created to add a row to the table on any update, insert or delete [21:12:53] including the logged in user's contact id , timestamp & connection_id [21:13:28] whew, wonder what that would do to performance [21:13:48] well, it's write performance that is affected - not read [21:13:49] probably not too bad, it's just doubling the work [21:14:02] I have read that simply having triggers imposes an overhead [21:14:14] the archive tables are optimised for writing to [21:14:22] k [21:14:37] but I have often had cause to convert some of them to INNODB & add indexes [21:15:18] Sounds like those would capture everything we needed to revert a merge [21:15:22] you could generally break them down into tables you want the UI to be able to access some history from & you want them to be more accessible [21:15:32] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Remove County field from UK addresses? - https://phabricator.wikimedia.org/T122348#1902039 (Pcoombe) NEW [21:15:47] yeah & in fact turning it on does add some screens for reverting edits [21:16:02] oh, nice. [21:16:07] WRT audit history, The problem we have is that both contribution_tracking and the CRM source_* fields currently have cardinality 1 per donation, but I'd rather get the full history of a donor's interactions with banners, donatewiki, paymentswiki, the CRM and any supporting processing streams [21:16:07] although I think it would revert one contact rather than 2 from the screen [21:16:32] yeah, definitely a more complex scenario than single-entity edit [21:16:36] ejegg: Excellent call re: undo merge [21:17:09] awight: so, those other interactions - they are 1 to 1 with payment or not even? [21:18:49] ejegg: please comment on the task if you get a chance! https://phabricator.wikimedia.org/T111704 [21:19:14] eileen: no. The first interaction is the most important, so we've been okay hobbling along like this [21:19:48] but e.g. donors like to change their donation amount, and we should have a record of how difficult that is for UX [21:20:04] I'm sure it takes 15 minutes and 90% of donors drop out... [21:20:28] hmm & what would you record that against? The contribution? [21:20:38] I guess it's a recurring contribution? [21:21:03] The contribution_tracking stuff is nastier because it needs to live outside of the CRM until (if) they actually donate [21:21:23] true [21:21:30] but for source_* fields, it's sort of similar story. We often get 3x copies of a contribution, but with slightly different information. [21:21:54] I'd like to know if and when all those redudant copies are coming through okay, and whether the data matches. [21:22:01] as in one from the initial gateway, one from the audit log… [21:22:08] And would especially like to enhance the original CRM record, as more info comes in. [21:22:10] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Make deduping reversible - https://phabricator.wikimedia.org/T111704#1902100 (Ejegg) @Eileenmcnaughton suggests turning on CiviCRM's logging function, which inserts into audit tables on every insert, update, and delete. This should capture all the data... [21:22:11] Yes [21:24:39] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Make deduping reversible - https://phabricator.wikimedia.org/T111704#1902115 (Eileenmcnaughton) Actually the grouping is not so touch - it's intrinsic in the connection_id which is stored to the logging table. I'm just not clear how it looks in a merge... [21:25:04] OK - so to bring this back in a bit [21:25:36] The main gap here is that we would not be tracking source data for a refund [21:26:08] & the feeling is that we would lose that short-term & solve it in a better but unknown way longer term? [21:26:12] +1 [21:26:27] sounds fine to me [21:26:38] So my giant digression is just to say, when we do that, I'd like to wring some added value out of whatever change we make [21:26:51] heck yeah [21:27:02] gotta run a quick errand, back soon! [21:27:06] bye! [21:27:13] :-) [21:31:58] Wikimedia-Fundraising-CiviCRM, MediaWiki-extensions-ContributionTracking, MediaWiki-extensions-DonationInterface: [Epic] Keep event-based donation history rather than 1:1 properties - https://phabricator.wikimedia.org/T122355#1902149 (awight) NEW [21:55:04] Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, Unplanned-Sprint-Work: banner data and missing alterImpressionData(): Determine exact impact on data and coping mechanisms - https://phabricator.wikimedia.org/T120017#1902231 (awight) Looks like this investigation can be... [21:57:01] ejegg: Please bless https://phabricator.wikimedia.org/T120727 if you have time, I'd like to break everything about payment processing today [21:57:45] ooh, fun. I was going to deploy that bpay patch, but I'll wait to piggyback on something more exciting [21:58:06] Hehe, I have a dinner rush out to so this could be perfect [21:59:49] hmm, outputPageStub is pretty bare. But I guess for this test it's an error to try anything except redirect [22:00:43] atgomez: Did you have a minute for UX discussion of https://phabricator.wikimedia.org/T96047 ? I can't remember where we left things. [22:00:56] ejegg: Yeah it seems unnecessarily fragile [22:01:10] I could inherit from OutputPage, I guess? [22:01:26] Yeah, that sounds like the thing to do [22:01:51] hehe, I seem to have a kill ping [22:02:20] * ejegg ducks [22:06:16] Okay, I extended OutputPage and tests still pass locally [22:07:04] rockin, will try again here [22:36:28] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Remove County field from UK addresses? - https://phabricator.wikimedia.org/T122348#1902432 (atgo) p:Triage>Low [22:37:41] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM upgrade: migrate custom fields to standard fields - https://phabricator.wikimedia.org/T122342#1902437 (atgo) @ejegg @awight what's the motivation for doing this? Just to simplify upgrades, etc. later? [22:38:57] atgomez: Did you have a minute for UX discussion of https://phabricator.wikimedia.org/T96047 ? I can't remember where we left things. [22:39:51] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: "The" issue: Contacts with dup records (with and without "The") - https://phabricator.wikimedia.org/T122264#1902459 (atgo) Hey @caitvirtue what are you hoping for here? A cleanup of old records entered? Some sort of rule for the future? [22:44:42] Fundraising-Backlog: Sprint A Goal (1/11 - 1/20): Big EN cleanup - https://phabricator.wikimedia.org/T119649#1902477 (atgo) [22:44:55] Fundraising Sprint Zapp, Fundraising-Backlog: Sprint Z goal (12/9-12/23): Finish out December! - https://phabricator.wikimedia.org/T116996#1902478 (atgo) [22:45:03] Fundraising-Backlog: Sprint A Goal (1/11 - 1/20): Big EN cleanup, get Adyen to backup ready - https://phabricator.wikimedia.org/T119649#1832111 (atgo) [22:45:23] Fundraising-Backlog: Sprint Goal B (1/20 to 2/3) - https://phabricator.wikimedia.org/T122371#1902481 (DStrine) NEW [22:46:14] Fundraising-Backlog: Sprint Goal C (2/3 to 2/17) - https://phabricator.wikimedia.org/T122372#1902489 (DStrine) NEW [23:05:57] Fundraising Sprint Zapp, Fundraising-Backlog: Sprint Z goal (12/9-1/11): Finish out December! - https://phabricator.wikimedia.org/T116996#1902563 (atgo) [23:09:00] Fundraising-Backlog, FR-Adyen: Adyen payments listener destroys donation info if capture fails - https://phabricator.wikimedia.org/T117816#1902575 (awight) [23:09:07] Fundraising-Backlog: Adyen listener: reschedule capture on missing "pending" message - https://phabricator.wikimedia.org/T120798#1902576 (awight) [23:09:40] Fundraising-Backlog, Tracking: [EPIC] Fix and improve geolocation, and ensure it's working properly for FR campaigns - https://phabricator.wikimedia.org/T121937#1902578 (DStrine) [23:10:05] Fundraising-Backlog, Tracking: [EPIC] Fix and improve geolocation, and ensure it's working properly for FR campaigns - https://phabricator.wikimedia.org/T121937#1902584 (atgo) We removed this from the sprint because it's an epic [23:10:31] Fundraising Sprint Zapp, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Display money columns as money in the Lybunt - https://phabricator.wikimedia.org/T122027#1902586 (Eileenmcnaughton) [23:10:43] Fundraising Sprint Zapp, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Fix static code warning - https://phabricator.wikimedia.org/T122021#1902587 (Eileenmcnaughton) [23:12:57] Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, and 3 others: Fix "Notice: Undefined index: next_time_24h in SpecialHideBanners.php" - https://phabricator.wikimedia.org/T120890#1902592 (atgo) p:Triage>Normal [23:13:07] Fundraising Sprint Yo La Tengo, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Patch-For-Review, Wikimedia-log-errors: Fix "Notice: Undefined index: next_time_24h in SpecialHideBanners.php" - https://phabricator.wikimedia.org/T120890#1902593 (DStrine) [23:14:30] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Investigate possible brief drop in banner history log rate on 2015-12-01 at 20:21 UTC - https://phabricator.wikimedia.org/T120058#1902601 (atgo) p:Triage>Low [23:15:09] Fundraising Sprint Zapp, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Investigate possible brief drop in banner history log rate on 2015-12-01 at 20:21 UTC - https://phabricator.wikimedia.org/T120058#1843809 (atgo) Hey @andyrussg we dropped this one from the sprint because it seems like so... [23:15:40] Fundraising-Backlog: mobile donation form: expiration date error - https://phabricator.wikimedia.org/T120053#1902606 (DStrine) [23:16:01] Fundraising-Backlog: mobile donation form: expiration date error - https://phabricator.wikimedia.org/T120053#1902610 (atgo) a:MBeat33 [23:16:34] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Investigate possible brief drop in banner history log rate on 2015-12-01 at 20:21 UTC - https://phabricator.wikimedia.org/T120058#1902611 (atgo) [23:16:50] Fundraising Sprint Yo La Tengo, Fundraising-Backlog, FR-GlobalCollect: WR1 audit parser emitting multiple bad refund lines - https://phabricator.wikimedia.org/T119927#1902612 (DStrine) [23:18:33] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, Fundraising-Backlog, Unplanned-Sprint-Work: TY email shows two different donation amounts - https://phabricator.wikimedia.org/T118867#1902614 (atgo) @mbeat33 did you hear anyone else experiencing this? [23:18:47] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising-Backlog: TY email shows two different donation amounts - https://phabricator.wikimedia.org/T118867#1902615 (atgo) p:High>Normal [23:19:10] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising-Backlog: TY email shows two different donation amounts - https://phabricator.wikimedia.org/T118867#1902619 (awight) Also noting: "WWikimedia", so something crazy is happening, probably on the receiving end. [23:20:26] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Zapp, Fundraising-Backlog, and 2 others: Don't include refunded donations in Silverpop export - https://phabricator.wikimedia.org/T117931#1902626 (atgo) @ccogdill_wmf is this urgent for you? [23:20:58] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Don't include refunded donations in Silverpop export - https://phabricator.wikimedia.org/T117931#1902629 (DStrine) [23:22:03] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising Sprint Zapp, and 3 others: Failing SmashPig job runner should not act like a mid-1980s photocopy machine - https://phabricator.wikimedia.org/T117447#1902631 (atgo) http://payload.cargocol... [23:22:21] Fundraising Sprint William Shatner, Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising-Backlog, and 2 others: Failing SmashPig job runner should not act like a mid-1980s photocopy machine - https://phabricator.wikimedia.org/T117447#1902632 (DStrine) [23:23:45] Fundraising Sprint X-Ray Spex, Fundraising Sprint Yo La Tengo, Fundraising-Backlog: TY email shows two different donation amounts - https://phabricator.wikimedia.org/T118867#1902634 (MBeat33) @atgo thankfully this was a blip that we have not seen more of since then.