[00:00:13] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/381366 (owner: Ejegg) [00:01:10] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/381366 (owner: Ejegg) [00:07:14] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Patch-For-Review: Add currency symbol to email export file - https://phabricator.wikimedia.org/T156410#3645140 (CCogdill_WMF) @Ejegg thanks! I think I actually need more info on where this currency comes from. We rely on latest_currency... [00:16:16] !log updated CiviCRM 213886998cad62a37aab5b214290b8431ab41199 to 784756b4157517bf9639a67b8d0ddcb1db7c999d [00:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:19:35] (PS1) Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/381367 [00:19:39] (CR) Ejegg: [C: 2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/381367 (owner: Ejegg) [00:19:45] (Merged) jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - https://gerrit.wikimedia.org/r/381367 (owner: Ejegg) [00:25:12] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Patch-For-Review: Add currency symbol to email export file - https://phabricator.wikimedia.org/T156410#3645157 (Ejegg) Ah, thanks for clarifying the import process! I always though the mapping had to be set up first. The latest_currency... [00:26:09] !log updated fundraising tools from 69a96276fa8cd8f68aa1225ea48fff1a7c833101 to 6d4b6f388e64ff4dc9a6a501bc8339fc1294fe9f [00:26:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:27:28] Fundraising Sprint Synchronized Screaming, Fundraising Tech Backlog, Fundraising-Backlog, Patch-For-Review: look at possible drupal logging snafu causing excessive log volume - https://phabricator.wikimedia.org/T121799#1888591 (Ejegg) Open>Resolved Bam! That turned out to be pretty easy. [00:31:44] Fundraising Sprint RadioActivewear, Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, MediaWiki-extensions-DonationInterface, and 2 others: Are we overwriting good country data with bad? - https://phabricator.wikimedia.org/T176450#3645177 (Ejegg) Open>Resolved [16:23:07] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, FR-Paypal: PayPal IPN processing jobs: undefined index subscr_id - https://phabricator.wikimedia.org/T176152#3647000 (XenoRyet) a:XenoRyet [16:26:45] (CR) XenoRyet: [C: 2] Config setting to quiet down LOCK_EXISTS [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381288 (https://phabricator.wikimedia.org/T172303) (owner: Ejegg) [16:31:23] (PS2) Ejegg: Config setting to quiet down LOCK_EXISTS [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381288 (https://phabricator.wikimedia.org/T172303) [16:35:18] Thanks XenoRyet ! [16:35:35] No worries, just thought I'd get the ol' brain warmed up with some CR. [16:35:47] AndyRussG had some good ideas for cleaning up the separation of concerns with the lock stuff [16:36:02] I might have another patch later today [16:36:21] Good times, I'll be happy to take a look when it's ready. [16:38:06] lmk if you want to collaborate on the subscr_id stuff [16:38:36] Yea, will do [16:51:28] (PS2) Ejegg: Process-control code cleanup [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/381278 [16:51:51] (PS6) Ejegg: Allow slow-starting jobs [wikimedia/fundraising/process-control] - https://gerrit.wikimedia.org/r/368217 (https://phabricator.wikimedia.org/T171873) [17:07:51] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Patch-For-Review: Add currency symbol to email export file - https://phabricator.wikimedia.org/T156410#3647134 (Ejegg) @CCogdill_WMF How do the new columns look? [17:29:49] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, MediaWiki-extensions-DonationInterface: New payments-wiki errors: We don't have an order status after doing a GET_ORDERSTATUS - https://phabricator.wikimedia.org/T176355#3647199 (Ejegg) p:Triage>High a:Ejegg [17:32:07] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Patch-For-Review: Add currency symbol to email export file - https://phabricator.wikimedia.org/T156410#3647207 (CCogdill_WMF) Running the job now, should take 45 mins or so. I'll let you know! [17:52:38] Fundraising Sprint Synchronized Screaming, Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, Patch-For-Review: Import donations from audit files where the log lines are missing - https://phabricator.wikimedia.org/T176923#3647270 (Ejegg) Open>Resolved Yep, that makemissing mode did the t... [18:26:24] oh man, invoice_id importation is all kinds of silly [19:12:32] (PS1) Ejegg: Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) [19:16:24] (PS2) Ejegg: Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) [19:19:56] (CR) jerkins-bot: [V: -1] Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [19:21:31] (PS3) Ejegg: Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) [19:25:11] (CR) jerkins-bot: [V: -1] Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [19:29:30] (PS4) Ejegg: Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) [19:41:54] (PS5) Ejegg: Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) [19:55:14] fundraising-tech-ops: prometheus collector or exporter for postfix metrics - https://phabricator.wikimedia.org/T176495#3647532 (cwdent) Open>Resolved We have kicked the ganglia habit and are 100% prometheus [20:02:31] fundraising-tech-ops, monitoring: overhaul fundraising cluster monitoring - https://phabricator.wikimedia.org/T91508#3647546 (cwdent) [20:02:34] Fundraising-Backlog, fundraising-tech-ops, Operations: Port fundraising stats off Ganglia - https://phabricator.wikimedia.org/T152562#3647544 (cwdent) Open>Resolved Closing this, there are a few subtasks left but we have parity with ganglia [20:08:41] (PS1) Ejegg: Fix bone-headed errors in duplicate invoice handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381498 (https://phabricator.wikimedia.org/T171349) [20:09:54] (PS1) Ejegg: Fix a couple base test case things [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381499 [20:14:52] (CR) jerkins-bot: [V: -1] Fix bone-headed errors in duplicate invoice handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381498 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [20:17:07] (PS2) Ejegg: Fix bone-headed errors in duplicate invoice handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381498 (https://phabricator.wikimedia.org/T171349) [20:19:52] hey fr-tech! have you heard back about your NTEN submission? [20:20:40] whenever mepps is around, I bet she got the notification :D [20:21:25] nothing yet ccogdill :( [20:21:38] mmmmm okay keep us posted [20:21:45] I just got the email [20:43:42] ccogdill not accepted :( [20:43:48] no way!! [20:43:54] oh my gosh mepps I can't believe it [20:44:10] I heard feedback from people at other companies that they really wanted to see your talk [20:44:11] :( [20:48:03] maybe the next year! [20:48:48] Super lame! So many practical details in your proposal, sorry mepps. [21:41:47] man, what a bummer :( [21:43:01] XenoRyet: if you get a chance, I could use some review on https://gerrit.wikimedia.org/r/381498 and https://gerrit.wikimedia.org/r/381490 [21:43:19] I think the tests are pretty solid to avoid breakage [21:43:23] 10-4 [21:43:48] it's pretty broken right now - the message is null when it's dropped into the damaged db :( [21:46:22] That's actually the first test in the CRM repo which uses the mock queues [21:46:27] We totally need more like that [21:49:04] Good times [21:53:25] (CR) XenoRyet: [C: 2] Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [21:53:37] (CR) XenoRyet: [C: 2] Fix bone-headed errors in duplicate invoice handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381498 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [21:53:43] thanks! [21:53:48] No worries [21:54:30] I see we just got another couple failing duplicates totally lost - I think a Friday deploy to fix that is better than leaving it all weekend [21:55:30] Yea, I'm super alergic to friday deploys, but it does seem waranted in this case. [21:55:53] The tests looked good too. It should be plenty safe. [21:57:05] oh yeah, and the matching 'MISSING PREDECESSOR' refunds [21:57:07] :( [21:57:39] I should write a script to reconstruct queue messages from the IPN listener logs [21:59:33] (Merged) jenkins-bot: Prefer 'invoice_id' to 'order_id' for importing invoice_id [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381490 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [21:59:47] (Merged) jenkins-bot: Fix bone-headed errors in duplicate invoice handling [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/381498 (https://phabricator.wikimedia.org/T171349) (owner: Ejegg) [22:00:04] brb, gotta give a bottle to a baby. [22:01:26] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/381506 [22:01:31] (CR) Ejegg: [C: 2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/381506 (owner: Ejegg) [22:02:53] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/381506 (owner: Ejegg) [22:12:28] !log updated CiviCRM from 784756b4157517bf9639a67b8d0ddcb1db7c999d to 59e3e00ecbaa3557fd32dc49259a1ae12edf7414 [22:12:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:21:17] ejegg I saw your email to William re timeouts - are those PayPal donations straightup failing? [22:32:07] MBeat: I think we retry 3 times, but it's a long wait for the donors [22:32:49] ah, ok, thanks - no complaints yet [22:33:04] cool, let me know if you hear any! [22:34:16] XenoRyet: I'm going to try to clean up that damaged queue [22:34:48] I think I can get the original transactions into Civi (need to remember to suppress ty emails) [22:35:08] then we can put a retry_date on all the damaged message db entries and send those back through [22:35:20] (all the MISSING_PREDECESSOR ones, that is) [22:35:45] Makes sense [22:36:41] I worked out a sed script to turn the log lines from the payment listener into JSON that we should be able to feed back in to the jobs-paypal queue [22:36:54] I should put that stuff up someplace... [23:09:44] https://techcrunch.com/2017/06/13/the-chat-fiction-apps-that-teens-go-crazy-for/ [23:10:07] "chat fiction" [23:10:20] Looks like wp needs a new entry [23:21:54] XenoRyet: ok, I think I've got 137 of the original donations reconstructed, with 'no_thank_you' tacked on. Going to feed them into the donations queue! [23:22:23] * ejegg crosses fingers [23:22:45] * XenoRyet fingers crossed [23:38:19] OK, the duplicate invoice id handling seems to have gone ok for the first step [23:38:45] they all ended up in the damaged queue, with the invoice_id suffixed [23:39:01] I ran the requeue command [23:39:11] that really should say how many it requeued! [23:41:07] oh right, retry_date is still in the future. messing with that... [23:41:20] k, that's better [23:41:43] so the next donation queue consume should actually import those [23:42:09] as long as that 'DuplicateInvoiceId' tag is set up right [23:45:41] oh hah [23:46:07] well... that duplicate invoice ID logic wasn't meant for them all coming in on the same batch! [23:46:24] it suffixes the duplicate id with time() [23:46:45] and when we're pulling in the messages really fast, that's still not unique [23:46:57] microtime would have done it [23:48:51] Ah, right. [23:50:03] oh hey, that failmail is totally unrelated to all my current messing about [23:50:15] ...and now we've got Adyen API timeouts [23:50:31] well, are we handling them gracefully? [23:53:22] I would think we are, but I don't know it for a fact. [23:53:52] In any case, that boy isn't going to pick himself up from school. I've gotta sign off.