[00:03:42] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Are we overwriting good country data with bad? - https://phabricator.wikimedia.org/T176450#3626028 (Ejegg) [00:06:22] dstrine: hey.. We should close T168673, eh? [00:06:22] T168673: CentralNotice: Add controls to purge banner content in Varnish for a specific language - https://phabricator.wikimedia.org/T168673 [00:09:25] (PS1) Ejegg: Remove old transitional code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379693 [00:22:46] (PS1) Ejegg: Restore good country data from session [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379694 (https://phabricator.wikimedia.org/T176450) [00:45:14] (PS1) Ejegg: PayPal country code mapping [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379697 (https://phabricator.wikimedia.org/T176450) [00:46:28] (PS2) Ejegg: Remove old transitional code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379693 [00:48:34] Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, Patch-For-Review: Are we overwriting good country data with bad? - https://phabricator.wikimedia.org/T176450#3626128 (Ejegg) p:Triage>High a:Ejegg Calling this 'high' as it's a data corrup... [00:51:33] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: Send more parameters to Maxmind minfraud service - https://phabricator.wikimedia.org/T128902#3626137 (Ejegg) @Ppena and @MBeat33 this is deployed! I hav... [02:09:24] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Epic: [EPIC] new round of common dedupe conflicts - https://phabricator.wikimedia.org/T153019#3626191 (Eileenmcnaughton) [02:09:26] Fundraising Sprint Deferential Equations, Fundraising Sprint English Cuisine, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Spike: What should we do to fix data where blank addresses have overwritten valid addresses - https://phabricator.wikimedia.org/T153917#3626188 (Eileenmcnaughton) O... [08:52:02] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), Patch-For-Review: Form chooser short term fix: retry without 'gateway' param - https://phabricator.wikimedia.org/T176356#3626332 (Aklapper) @Zoranzoki21: Please never close o... [10:45:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1260 [10:50:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1368 [10:55:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1494 [11:00:12] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1604 [11:41:41] Fundraising-Backlog, MediaWiki-extensions-DonationInterface, MW-1.31-release-notes (WMF-deploy-2017-09-26 (1.31.0-wmf.1)), Patch-For-Review: Form chooser short term fix: retry without 'gateway' param - https://phabricator.wikimedia.org/T176356#3626731 (Zoranzoki21) Ok @Aklapper [12:10:12] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 354 Threads: 2 Questions: 584865 Slow queries: 24 Opens: 206 Flush tables: 1 Open tables: 220 Queries per second avg: 1652.161 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:11:00] Fundraising-Backlog, fundraising-tech-ops: fundraising database replication lag master thread - https://phabricator.wikimedia.org/T173472#3626832 (Jgreen) [12:21:11] Fundraising-Backlog, fundraising-tech-ops: fundraising database replication lag master thread - https://phabricator.wikimedia.org/T173472#3626846 (Jgreen) Now that we've worked around the BGP/VPN flap issue and improved our prometheus instrumentation, I'm fairly convinced we're either hitting a mariadb b... [14:09:17] AndyRussG: re T168673: if this is totally out on production, you should ping seddon or some other heavy user to confirm. If it can only be verified on your end then maybe get help from another engineer to resolve it. [14:09:17] T168673: CentralNotice: Add controls to purge banner content in Varnish for a specific language - https://phabricator.wikimedia.org/T168673 [14:41:31] (CR) Mepps: [C: 2] Remove old transitional code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379693 (owner: Ejegg) [14:44:01] Fundraising-Backlog, fundraising-tech-ops: start testing mariadb 10.1.23 for fundraising - https://phabricator.wikimedia.org/T176489#3627318 (Jgreen) [14:46:05] (Merged) jenkins-bot: Remove old transitional code [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379693 (owner: Ejegg) [14:49:14] (CR) Mepps: [C: 1] "The tests look good but I think XenoRyet should also look at this. I don't love a class created just for one conditional." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379697 (https://phabricator.wikimedia.org/T176450) (owner: Ejegg) [14:50:16] fundraising-tech-ops, Operations, netops: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627380 (Jgreen) [14:50:33] fundraising-tech-ops, Operations, netops: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627397 (Jgreen) [14:50:37] fundraising-tech-ops, Operations, netops: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627399 (Jgreen) a:Jgreen>None [14:50:49] fundraising-tech-ops, Operations, netops: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627380 (Jgreen) p:High>Triage [15:30:29] fundraising-tech-ops: prometheus collector or exporter for banner log pipeline - https://phabricator.wikimedia.org/T176494#3627500 (Jgreen) [15:32:57] fundraising-tech-ops: prometheus collector or exporter for postfix metrics - https://phabricator.wikimedia.org/T176495#3627515 (Jgreen) [15:34:40] dstrine: hey... Yes T168673 has been out on production for a while. I think pcoombe did confirm it's good (maybe he mentioned generally the cache purge stuff and not this change individually, though, not sure) [15:34:41] T168673: CentralNotice: Add controls to purge banner content in Varnish for a specific language - https://phabricator.wikimedia.org/T168673 [15:35:16] (PS2) Mepps: Add 'ingenico' to error form definitions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379438 (owner: Ejegg) [15:35:20] (CR) Mepps: [C: 2] Add 'ingenico' to error form definitions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379438 (owner: Ejegg) [15:36:31] AndyRussG: I've not used the specific language purge myself that I can remember. Seddon, have you? [15:37:15] (Merged) jenkins-bot: Add 'ingenico' to error form definitions [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379438 (owner: Ejegg) [15:46:25] pcoombe (AndyRussG): I've pressed it but I haven't tested it in a rigorous fashion [15:52:46] pcoombe: Seddon: ah OK thx :) Just an administrative question, really, no problem I think in leaving the task open until we've seen the feature in action [16:04:50] fundraising-tech-ops, Operations, netops: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627580 (Jgreen) This also requires an updating to the firewall policy, I added the new database and generated the new policy. com... [16:26:12] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM Damaged message UI pagination is broken - https://phabricator.wikimedia.org/T176497#3627625 (Ejegg) [16:26:53] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM Damaged message UI should offer bulk requeue - https://phabricator.wikimedia.org/T176498#3627639 (Ejegg) [16:27:22] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: CiviCRM Damaged message UI should offer bulk requeue - https://phabricator.wikimedia.org/T176498#3627654 (Ejegg) p:Triage>Normal [16:36:42] fundraising-tech-ops, Operations, netops, Patch-For-Review: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627694 (ayounsi) a:ayounsi Vlan changed on pfw-eqiad (old) Vlan changed on fasw-c-eqiad (new) Security p... [16:40:17] PROBLEM - check_puppetrun on frlog1001 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 9 minutes ago with 26 failures. Failed resources (up to 3 shown) [16:45:17] RECOVERY - check_puppetrun on frlog1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:48:54] mepps did you want more CR on those WIP patches? [16:50:28] Yeah ejegg that'd be great [16:50:37] cool, will do [16:51:40] Fundraising-Backlog, fundraising-tech-ops: start testing mariadb 10.1.23 for fundraising - https://phabricator.wikimedia.org/T176489#3627750 (ayounsi) [16:51:57] fundraising-tech-ops, Operations, netops, Patch-For-Review: move frav1001's to the frack-fundraising VLAN so we can use it for database testing - https://phabricator.wikimedia.org/T176492#3627748 (ayounsi) Open>Resolved new policy file worked fine, committed. Don't forget to update rackta... [16:57:05] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: email address validation not clearing error on blur - https://phabricator.wikimedia.org/T176501#3627759 (Ejegg) [17:08:11] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico adapter: parse response errors - https://phabricator.wikimedia.org/T176502#3627785 (Ejegg) [17:08:58] (CR) Ejegg: "Just noticed an issue with my error parsing. Could save that for another ticket though: https://phabricator.wikimedia.org/T176502. Otherwi" (1 comment) [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 (owner: Mepps) [17:27:58] (CR) Ejegg: "The test looks fine. We should be storing the payment ID and updating amount + currency on either this call or the get...Status call, so l" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [17:28:15] XenoRyet: how goes the PayPal investigation? Want to talk anything out? [17:28:59] Just wrapping it up really. Looking up a few more missing predicessor types is all I really have left. [17:29:25] k. [17:29:58] So far it's been that paypal just seems to be sending us the wrong parent txn id. [17:30:02] i'mma re-queue all those deadlocked contribution imports from yesterday. [17:30:08] Whoa, wait, really? [17:30:11] good times [17:30:26] So... they're saying something is refunded and it isn't? [17:30:49] Can you write an email to our tech contacts about that? [17:32:05] (PS2) Ejegg: WIP: Prometheus metrics reporting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379442 (https://phabricator.wikimedia.org/T171524) [17:32:24] Not exatly? It appears to be a real refund. The donor details and amounts match donations that are in our system just under a different txn id. It's hard to keep the whole chain in my head to IRC about, but I'll do a write-up in the ticket. [17:33:06] and the parent_txn_id they give isn't in our logs at all, audits and IPNs included? [17:33:41] Let me go back and check that part of it again. [17:33:47] so either the refund messages are bad, or we're not getting info about original txns [17:34:02] Do they show up in the console? [17:34:29] I haven't looked in the console. [17:34:56] k, if they're not there, that would be a good indicator that the refund messages are just nonsense [17:47:40] (PS8) Mepps: Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 [17:54:13] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3627931 (cwdent) [17:54:16] fundraising-tech-ops, Operations: Long term storage for frack prometheus data - https://phabricator.wikimedia.org/T175738#3627929 (cwdent) Open>Resolved We will look into aggregated stats again later but there were spare 1TB disks on the lvs servers so I moved the prometheus backend there and set... [17:57:15] (PS3) Ejegg: Prometheus metrics reporting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379442 (https://phabricator.wikimedia.org/T171524) [17:57:21] (PS1) Ejegg: Delete ganglia_reporter, use new module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379806 (https://phabricator.wikimedia.org/T171524) [17:57:41] OK fr-tech, ^^^ should get us basic Prometheus reporting (with some work on the ops side) [17:58:36] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Update CiviCRM ganglia bits to point to Prometheus - https://phabricator.wikimedia.org/T171524#3627934 (Ejegg) a:Ejegg [17:59:51] (CR) Mepps: "Overall looks fine--maybe a test for PrometheusReporter though?" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379442 (https://phabricator.wikimedia.org/T171524) (owner: Ejegg) [18:00:22] Fundraising Sprint Quill Pencil, Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, and 2 others: Find source of unlimited dedupe queries, prevent them - https://phabricator.wikimedia.org/T175382#3627939 (Ejegg) We've deployed a couple of things to block the b... [18:04:57] ejegg should we be calling approvePayment in processDonorReturn? [18:05:15] yep, it's deep inside Confirm_CreditCard [18:05:25] oh gotcha [18:05:46] (PS1) Ejegg: Ingenico can't accept VEF anymore [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379807 [18:06:40] Fundraising-Backlog, MediaWiki-extensions-DonationInterface: Can we use Ingenico currencies 'Temporarily removed for Wells Fargo' - https://phabricator.wikimedia.org/T131407#3627956 (Ejegg) This would be nice to follow up on! [18:07:04] fr-tech mind reviewing that VEF patch? [18:07:14] as per Pats's email today [18:08:42] Yea, I'll take a look [18:09:21] (CR) XenoRyet: [C: 2] Ingenico can't accept VEF anymore [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379807 (owner: Ejegg) [18:13:45] (Merged) jenkins-bot: Ingenico can't accept VEF anymore [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379807 (owner: Ejegg) [18:15:58] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Create orphan rectifier for PayPal Express Checkout - https://phabricator.wikimedia.org/T172202#3627977 (Ejegg) @mepps sure, let's test it! [18:45:30] Fundraising-Backlog, fundraising-tech-ops: fundraising database replication lag master thread - https://phabricator.wikimedia.org/T173472#3628023 (cwdent) [18:45:32] Fundraising-Backlog, fundraising-tech-ops: One hour test Wed 20 Sep - https://phabricator.wikimedia.org/T176294#3628021 (cwdent) Open>Resolved We for the time being acquiesce to restarting mysql on the replicas when they start to lag. The server tuning line of inquiry has been rather fruitless.... [18:52:31] AndyRussG: heads up on https://gerrit.wikimedia.org/r/379468 [18:52:50] ejegg should we be overriding transactionConfirmCreditCard? It looks like a lot of that is specific to globalcollect [18:53:01] looks to stop returning a promise on EventLogging.log [18:53:21] mepps you think so? [18:54:27] in both cases, we have to get status, check AVS/CVV, run other filters, avoid retrying the same set of error codes, then approve the payment [18:54:43] yes ejegg there are references to GET_ORDERSTATUS and DO_FINISHPAYMENT transactions which ingenico doesn't use, correct? [18:54:47] It would be nice to delete that huge ball of code, though [18:54:59] yeah i'm less worried about the logic, it's more about the specific transaction calls [18:55:10] we could wrap them the way approvePayment is though [18:55:21] yeah, that's what I was thinking [18:55:48] the GET_ORDERSTATUS constant is still OK to use in the status code lookup [18:56:15] since we're reusing defineReturnValueMap [19:01:01] (PS9) Ejegg: Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 (owner: Mepps) [19:01:12] (CR) Ejegg: [C: 2] Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 (owner: Mepps) [19:03:07] Ugh, Donation consumer failmail [19:03:10] (Merged) jenkins-bot: Try out approvePayment [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/370656 (owner: Mepps) [19:04:28] weird, queue connection timeout [19:07:36] AndyRussG: never mind, I see now that thing I alerted you about is irrelevant to CN! [19:13:30] Fundraising Sprint RadioActivewear, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Email, Patch-For-Review: Reset on_hold in wmf_civicrm_message_email_update - https://phabricator.wikimedia.org/T170350#3628072 (Ejegg) Resetting on_hold when there's a change definitely sounds fine. Sin... [19:14:42] hey ejegg, i might be missing something but i'm not sure if there's an equivalent of DO_FINISHPAYMENT for ingenico? [19:15:40] mepps oh huh, lemme see... I guess I was conflating that with SET_PAYMENT [19:16:51] yeah SET_PAYMENT seems to map to approvePayment (given the wrapper) but DO_FINISHPAYMENT might map to capture payment? [19:17:08] assuming that's what the PENDING_POKE status means [19:17:19] the comment in the GC adapter mentions 3d secure [19:17:50] I think that's a thing where they have to redirect the user to e.g. Visa's page for an additional confirmation [19:18:29] lessee, status code 200 [19:18:54] I think that's payment status 200, not http 200 [19:18:58] hmm, does that apply to ingenico? [19:19:17] the connect API should support 3d secure somehow [19:19:27] I'll make another ticket for that [19:21:00] Fundraising-Backlog, Fr-Ingenico-integration_2017-18: Ingenico Connect: Support 3d Secure transactions - https://phabricator.wikimedia.org/T176512#3628078 (Ejegg) [19:34:38] (PS1) Ejegg: Fix amount warnings [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379826 (https://phabricator.wikimedia.org/T160962) [19:37:36] (CR) jerkins-bot: [V: -1] Fix amount warnings [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379826 (https://phabricator.wikimedia.org/T160962) (owner: Ejegg) [19:38:22] heh, that darn phpcs! [19:45:23] ejegg/AndyRussG are we the only ones who will be in standup today? [19:45:42] mepps I think so! [19:45:48] also, thanks for the reminder [19:45:56] cool ejegg, i'm hanging out there now [19:46:05] I seem to have missed the phone-ping [19:46:53] mepps it says I'm the only one in the queenmary hangout [19:51:58] (PS2) Ejegg: Fix amount warnings [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379826 (https://phabricator.wikimedia.org/T160962) [20:00:10] fundraising-tech-ops: frack syslog collection running 1/2 hour behind - https://phabricator.wikimedia.org/T176326#3628231 (Ejegg) @Jgreen did you get a chance to change the memcache logging back from debug? [20:10:27] (PS3) Mepps: WIP approvePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 [20:20:27] (CR) jerkins-bot: [V: -1] WIP approvePayment tests [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [20:26:35] fundraising-tech-ops: frack syslog collection running 1/2 hour behind - https://phabricator.wikimedia.org/T176326#3628277 (Jgreen) >>! In T176326#3628231, @Ejegg wrote: > @Jgreen did you get a chance to change the memcache logging back from debug? I did! Hopefully that made the difference. [21:09:56] (PS1) Ejegg: Fix Amazon token timeout [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379900 [21:37:28] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3628606 (Ejegg) Yikes! We do seem to be losing them... [21:51:10] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3628636 (Ejegg) The DonationQueueConsumer definitely... [21:54:52] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3628651 (Ejegg) Ooh, looks like that's a yes: https:... [22:15:26] (PS1) Ejegg: Search for invoice ID before inserting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379941 (https://phabricator.wikimedia.org/T171349) [22:19:26] Fundraising Sprint Navel Warfare, Fundraising Sprint Outie Inverter, Fundraising Sprint Prank Seatbelt, Fundraising Sprint Quill Pencil, and 4 others: Are we losing transactions witih repeated ct_id? - https://phabricator.wikimedia.org/T171349#3628697 (Ejegg) @XenoRyet Take a look at this - it ma... [22:34:36] (PS2) Ejegg: Search for invoice ID before inserting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379941 (https://phabricator.wikimedia.org/T171349) [22:34:50] (PS1) Ejegg: Alternate solution to losing duplicate invoice IDs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379945 (https://phabricator.wikimedia.org/T171349) [22:36:00] (CR) Ejegg: "The tests look pretty good! I'd still be in favor of splitting out the response arrays into json files." [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/379292 (owner: Mepps) [22:53:43] (PS4) Ejegg: Prometheus metrics reporting [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379442 (https://phabricator.wikimedia.org/T171524) [22:53:45] (PS2) Ejegg: Delete ganglia_reporter, use new module [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379806 (https://phabricator.wikimedia.org/T171524) [22:54:03] (CR) Ejegg: "OK mepps, added a test!" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379442 (https://phabricator.wikimedia.org/T171524) (owner: Ejegg) [23:31:56] (CR) jerkins-bot: [V: -1] Alternate solution to losing duplicate invoice IDs [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/379945 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg)