[00:02:19] https://lab.civicrm.org/dev/core/issues/1486 [00:03:04] Also - I tweaked your deadlock patch & think it should go out - if you do a double check - it doesn't solve the whole thing but I think it's correct https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/civicrm/+/368517/2..3 [00:03:07] (CR) Cstone: [C: +2] "Looks good, I saw the same speed increases with the queries." [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/559138 (https://phabricator.wikimedia.org/T241000) (owner: Ejegg) [00:03:42] (Merged) jenkins-bot: Add IsEndowment filter [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/559138 (https://phabricator.wikimedia.org/T241000) (owner: Ejegg) [00:27:06] eileen: oh really! [00:27:32] thanks cstone! [00:27:51] I've found I can replicate locally by running 2 test suites at once [00:28:08] having said that the wmf_contribution_xtra wouldn't fail locally due to speed [00:28:54] so I'm still trying to dig into handling too - because if we reduce the locks happening first then we might not get to the handling & the few that do happen will be an issue [00:30:35] (PS2) Ejegg: EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) [00:34:07] thanks for dusting off that old WIP - I +2ed it. I'll look for yours that sets the 'deadlock' error code next [00:37:00] (PS1) Eileen: Handle constraint errors on location insert as deadlocks. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559250 (https://phabricator.wikimedia.org/T182148) [00:37:13] (CR) jerkins-bot: [V: -1] EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) (owner: Ejegg) [00:38:12] (PS3) Eileen: Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) [00:38:14] (PS1) Eileen: Handle constraint errors on location insert as deadlocks. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559251 (https://phabricator.wikimedia.org/T182148) [00:38:27] ejegg: I think it was a good idea! [00:38:33] weird, why is the DonationStatsCollectorTest failing now? [00:38:37] I've built a little on the deadlock - https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/559250/ [00:39:18] cool, looks good! [00:39:38] (CR) Ejegg: "recheck" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) (owner: Ejegg) [00:40:17] (PS1) Eileen: Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559252 [00:42:01] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559253 [00:42:13] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559253 (owner: Ejegg) [00:42:51] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559253 (owner: Ejegg) [00:46:14] (Abandoned) Eileen: Handle constraint errors on location insert as deadlocks. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559250 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [00:48:43] gotta run! I'll review the rest of those deadlock patches tomorrow [00:51:08] (PS1) Eileen: Throw database contention exception on contribution insert fail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559257 (https://phabricator.wikimedia.org/T182148) [01:11:14] (PS2) Eileen: Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559252 [01:11:16] (PS4) Eileen: Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) [01:11:18] (PS2) Eileen: Throw database contention exception on contribution insert fail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559257 (https://phabricator.wikimedia.org/T182148) [01:11:20] (PS1) Eileen: Handle constraint errors on location insert as deadlocks. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559264 (https://phabricator.wikimedia.org/T182148) [01:50:00] (CR) Cstone: [C: +2] "Friendlier errors are better errors." [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/556745 (https://phabricator.wikimedia.org/T240524) (owner: Ejegg) [01:50:35] (Merged) jenkins-bot: Send error codes in data route [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/556745 (https://phabricator.wikimedia.org/T240524) (owner: Ejegg) [01:56:00] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/559287 [01:56:28] (CR) Eileen: [C: +2] Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559252 (owner: Eileen) [01:56:57] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/559287 (owner: Eileen) [01:58:26] !log - civicrm revision changed from 93037d6e35 to 6062da3ab5, config revision is 2e9bf6308b mostly eoy_summary stuff [01:58:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:02:11] (Merged) jenkins-bot: Submodule commit [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559252 (owner: Eileen) [02:26:44] (Abandoned) Eileen: Handle constraint errors on location insert as deadlocks. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559251 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [02:50:01] (PS5) Eileen: Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) [02:50:03] (PS1) Eileen: Throw database contention exception on constraint errors in places they are known to be deadlock caused [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559292 (https://phabricator.wikimedia.org/T182148) [02:50:17] (Abandoned) Eileen: Throw database contention exception on contribution insert fail [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559257 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [02:50:31] (Abandoned) Eileen: Handle constraint errors on location insert as deadlocks. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559264 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [03:44:01] (PS2) Eileen: Throw database contention exception on constraint errors in places they are known to be deadlock caused [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559292 (https://phabricator.wikimedia.org/T182148) [04:55:24] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559331 [04:55:38] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559331 (owner: Ejegg) [04:56:38] (Merged) jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559331 (owner: Ejegg) [05:24:17] (PS1) Ejegg: Update index.html for new script filename [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559343 [05:24:38] (CR) Ejegg: [C: +2] Update index.html for new script filename [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559343 (owner: Ejegg) [05:25:15] (Merged) jenkins-bot: Update index.html for new script filename [wikimedia/fundraising/dash] (deployment) - https://gerrit.wikimedia.org/r/559343 (owner: Ejegg) [14:59:11] (PS1) Ejegg: Merge branch 'REL1_31' into fundraising/REL1_31 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/559510 [15:03:41] (CR) Ejegg: [C: +2] Merge branch 'REL1_31' into fundraising/REL1_31 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/559510 (owner: Ejegg) [15:07:53] (Merged) jenkins-bot: Merge branch 'REL1_31' into fundraising/REL1_31 [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/559510 (owner: Ejegg) [15:10:39] !log updated payments-wiki from 7131303dba to 827e3235dc [15:10:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:12] !log updated fundraising internal dashboard from 913d690621 to b75f9074de [15:12:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:18:21] Fundraising-Backlog, Wikimedia-Fundraising, Readers-Web-Backlog (Tracking): [Bug] Banner causes misalignment in article - https://phabricator.wikimedia.org/T240641 (Pcoombe) [16:09:49] fr-retro skipped? [16:09:57] fr-tech^ [16:10:40] jgleeson huh that wasn't on my calendar [16:10:56] i think you have a copy of an old meeting on yours? [16:11:01] ah [16:11:02] maybe [16:11:21] jgleeson: reload page? [16:11:40] it still lives!!! [16:11:51] ohhhh hmmmm the retro retro [16:12:14] the ghost of christmas retro [16:18:23] jgleeson did you retro? i think it's telling it's time for some reflection :) [16:40:09] :) [16:55:26] retros roasting on an open fire [17:05:40] hehe [17:14:18] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for cstone - https://phabricator.wikimedia.org/T241153 (Dwisehaupt) [17:14:25] fr-tech there's one more EOY email patch to review, if anyone has a minute: https://gerrit.wikimedia.org/r/559229 [17:15:34] (Abandoned) Ejegg: Update data constraints for Ingenico Connect [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/556300 (owner: Ejegg) [17:17:07] (CR) Ejegg: "Got a thread going with Merchant Services to confirm the surname length. Documentation certainly says string (70), but one recent emoji-la" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/556079 (owner: Cstone) [17:17:09] ejegg: The English one says 'Your {{ year }} contributions to Wikipedia' but the rest seem to be translations of 'This is a receipt, but it's also so much more', is that intended? [17:17:21] XenoRyet: ohhh, weird [17:17:30] let's check with CDenes [17:17:44] Also, did anyone else just loose the staff meeting stream? [17:18:18] nevermind, auth monster got me. [17:19:07] (CR) Jgleeson: [C: -1] "Found the issue. It looks like the behaviour didn't break instead it was just the count output query missing a COUNT() call around the ast" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558794 (https://phabricator.wikimedia.org/T238691) (owner: Eileen) [17:19:38] XenoRyet: yeah all good here... also, this has your name on it: https://xkcd.com/2243/ [17:19:52] lol [17:22:07] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for khancock - https://phabricator.wikimedia.org/T241154 (Dwisehaupt) [17:29:48] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for shust - https://phabricator.wikimedia.org/T241158 (Dwisehaupt) [17:30:13] (PS3) Ejegg: EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) [17:37:17] (CR) jerkins-bot: [V: -1] EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) (owner: Ejegg) [17:43:23] oh right, I updated the tests to match the rolled back en subject line [17:56:37] (PS4) Ejegg: EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) [18:18:43] (CR) XenoRyet: [C: +2] EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) (owner: Ejegg) [18:24:05] so.. if you're not using unique passwords on ALL your accounts yet, this tool might convince you: https://monitor.firefox.com/scan [18:24:18] my personal address has featured in 9 leaks that they know about [18:25:14] (Merged) jenkins-bot: EOY summary: subject line translations [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559229 (https://phabricator.wikimedia.org/T240648) (owner: Ejegg) [18:25:26] They've got me on 10, but thankfully I was using unique passes for all of them, and everything in general. [18:25:27] thanks XenoRyet [18:27:54] Fundraising-Backlog, Analytics: Turnilo no longer showing sample-rate adjusted data - https://phabricator.wikimedia.org/T241162 (AndyRussG) [18:32:35] holy crap https://monitor.firefox.com/breach-details/VerificationsIO [18:40:42] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (EMartin) After much deliberation with Paypal, and challenging that these are spoof emails by some fr... [18:41:16] Fundraising Sprint X-rays, Fundraising-Backlog: Fredge search in Civi randomly scrambling search results - https://phabricator.wikimedia.org/T239967 (DStrine) [18:50:36] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (MBeat33) Thank you, Evelyn, I really appreciate your diligence with PayPal. Even if this change caus... [18:53:16] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (EMartin) Ok, i wanted your blessing here so glad to hear you agree. I still would like PP to troub... [18:54:52] Fundraising-Backlog, Analytics: Turnilo no longer showing sample-rate adjusted data for banner activity - https://phabricator.wikimedia.org/T241162 (AndyRussG) [18:54:59] Fundraising Sprint X-rays, Fundraising-Backlog: Fredge search in Civi randomly scrambling search results - https://phabricator.wikimedia.org/T239967 (EMartin) To be clear on this. This is still occurring but it has significantly reduced and is manageable. It would be good to resolve for next peak if po... [18:56:15] Wikimedia-Fundraising-Banners: Email input accepts invalid email address - https://phabricator.wikimedia.org/T240887 (EWilfong_WMF) Acoustic will not send to addresses it deems invalid (or those that have bounced before), so it shouldn't be affecting our bounce rates. We can review the bounce data though to... [18:57:21] Fundraising Sprint Trojan Horse Wisperer, Fundraising-Backlog, FR-PayPal-ExpressCheckout: Duplicate donations caused by PayPal's “Final Approval” messaging - https://phabricator.wikimedia.org/T235220 (MBeat33) Exactly. PayPal's words: "However, this will also cause some donations to go unfinished whe... [20:32:50] fr-tech looks like we have no blockers to deploying this: https://gerrit.wikimedia.org/r/558244 if anyone wants to review it [20:33:03] donatewiki already serves an x-frame-options: deny header [20:33:35] so I think the banners would be the last place sending people direct to paymentswiki that might have been in a frame [20:33:36] (CR) Eileen: "jgleeson - thanks for persevering & finding that! Big opps avoided. FIxed now" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558794 (https://phabricator.wikimedia.org/T238691) (owner: Eileen) [20:34:28] ejegg: unrelated but I'd like to see if I can deploy those new exception handling patches so I can monitor today [20:35:13] eileen: ah yeah, I was going to focus on that today, wasn't i [20:35:16] thanks for the reminder [20:36:07] ejegg: so I think the patches at the moment will make the deadlocks less problematic - I think I can also reduce them but would rather make sure we are handling them when they do occur before making them harder to track [20:43:10] (PS3) Eileen: Coding practice - use single quotes unless double quotes required [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558764 [20:43:13] (PS3) Eileen: Switch to getting the name of the temp tables by function. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558793 (https://phabricator.wikimedia.org/T238691) [20:43:15] (PS3) Eileen: Switch from drupal query function to civi (see previous commit for rationale) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558794 (https://phabricator.wikimedia.org/T238691) [20:44:42] (PS3) Ejegg: Add error code for deadlocks [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/558312 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [20:44:56] (PS4) Ejegg: Add error code for deadlocks [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/558312 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [20:45:39] eileen: we want both the PEAR fix that's already merged, PLUS that one ^^^, right? [20:48:07] ejegg: actually I think I found yours was enough [20:48:40] I was wrong about those code lines being unreachable - in fact they are reached & the other isn't - I misread the code [20:48:54] (I managed to step through it rather than guessing in the end :-) [20:49:02] ok, I'll stick with the patches in the parent repo then [20:50:30] yep [21:00:32] eileen hmm, don't you need that civicrm patch to make this one work: https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/558315/5/sites/all/modules/wmf_common/WmfDatabase.php [21:00:40] it's looking for the 'deadlock' code [21:05:25] ejegg: that comes from your patch - it can fetch that code from extraParams [21:05:52] hang on - that one might be superceded - let me check [21:06:20] ejegg: just these 2 https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/559292/ [21:07:18] oh it's still in there - hmm did I fix it in the other one? Needs to be getErrorCode [21:09:04] ejegg: yep - looks like those lines came back OUT again in the second patch because I decided that wasn't right - I can clean them up a bit [21:09:48] because what I wound up with is making sure the right exception was thrown in the first place so by the time they reached that point it can just re-throw [21:12:37] oh, ok [21:19:06] (PS6) Ejegg: Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:19:12] (PS3) Ejegg: Throw database contention exception on constraint errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559292 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:22:08] (PS7) Ejegg: Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:22:10] (PS4) Ejegg: Throw database contention exception on constraint errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559292 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:22:20] just prettying up the commit message linewraps [21:24:11] ejegg: cool - just an FYI - there is a setting in Civi that causes the stuff to ConfigAndLog to be sent to watchdog / syslog ( it would be both places so would increase write load but I guess it's not logging all the time like our queue consumers - unless something goes wrong) [21:24:59] yeah, for this truly exceptional stuff we needn't worry too much about verbosity [21:27:05] exceptional ! [21:27:23] well it's in the UI civicrm/admin/setting/debug?reset=1 [21:27:42] I can turn it on if we want to try it but I guess there is a case for not making changes right now too [21:29:06] (CR) Ejegg: [C: +2] Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:29:10] (CR) Ejegg: [C: +2] Throw database contention exception on constraint errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559292 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:32:21] ejegg: do you want to +2 this - is super trivial so nice to clear it out https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/558764/ [21:32:46] oh sure [21:33:25] (PS4) Ejegg: Coding practice - use single quotes unless double quotes required [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558764 (owner: Eileen) [21:33:31] (CR) Ejegg: [C: +2] Coding practice - use single quotes unless double quotes required [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558764 (owner: Eileen) [21:34:53] (Merged) jenkins-bot: Handle deadlocks as WMFExceptions. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558315 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:34:57] (Merged) jenkins-bot: Throw database contention exception on constraint errors [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559292 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:39:05] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for cstone - https://phabricator.wikimedia.org/T241153 (Dwisehaupt) Certificate generated and passed to cstone. Additionally, I found a logic bug with the new coterming that would make certs issued in December a little more of a... [21:39:10] (Merged) jenkins-bot: Coding practice - use single quotes unless double quotes required [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/558764 (owner: Eileen) [21:41:08] (PS1) Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/559597 [21:41:20] (CR) Eileen: [C: +2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/559597 (owner: Eileen) [21:47:33] (PS1) XenoRyet: WIP Fix fake PayPal cancellations on the fly [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559605 (https://phabricator.wikimedia.org/T217379) [21:49:28] (PS3) Ejegg: Simplify duplication check. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/556298 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:49:37] !log civicrm revision changed from 6062da3ab5 to 6ecdccd240, config revision is 2e9bf6308b [21:49:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:49:49] (CR) Ejegg: [C: +2] Simplify duplication check. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/556298 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:50:05] ejegg: I might tune up the deduping to try to get more deadlocks to happen so I can see what happens [21:50:24] also - the EOY summary patch went out with those [21:51:02] ok, great! [21:52:17] current is --batch=5000 --threshold=550 --threshold_period=4 [21:52:57] increase threshold to 1000? [21:52:59] Fundraising Sprint YAMLton, the Musical, Fundraising-Backlog: January Summary Emails - https://phabricator.wikimedia.org/T240648 (Ejegg) p:Triage→High a:CDenes_WMF [21:54:04] hangon - that threshold can't be the reason it's not running more [21:55:31] (Merged) jenkins-bot: Simplify duplication check. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/556298 (https://phabricator.wikimedia.org/T182148) (owner: Eileen) [21:56:42] OK - it's already deduping [21:58:22] I guess I just have to tail the CIvi log & hope I see a deadlock - I'm not sure what I can do to stress it if deduping is already running atm [21:58:44] I suppose I could run second dedupe job manually [21:59:02] I should start with major gifts since I think that is stil off [22:17:39] Wikimedia-Fundraising-Banners: Email input accepts invalid email address - https://phabricator.wikimedia.org/T240887 (jbolorinos-ctr) Hey Eric, cool sounds good! I added it to the list of items to review with Thea, so we'll discuss tomorrow! [22:24:35] (PS1) Ejegg: Add toggle to include endowment in Big English charts [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/559610 (https://phabricator.wikimedia.org/T241000) [22:29:03] fail mail was on jobs I'm running on the command line running overtime - so far I've triggered one deadlock but it was on a merge & self-recovered. [22:29:38] ah, ok [22:30:21] yeah it looks like because it was a simple query nothing rolled back except that one query [22:39:54] I feel like I'm hammering it but those deadlocks are staying away.... [22:56:03] I got a deadlock [22:56:40] recurring q consumer I think [23:05:15] oh hey, did it get kicked to the retry line? [23:05:19] eileen: i'm assuming i can ignore the redis warning emails about queues growing since it's related to your testing. please let me know if i shouldn't. [23:05:26] (no ui for that) [23:05:29] ejegg: I'm just trying to figure that out [23:05:43] eileen: it would be in the damaged table, but with a retry_date set [23:05:54] damaged table in the smashpig db [23:05:57] dwisehaupt: yes I've hammered it a bit on purpose to trigger a deadlock - it should start quieting down now [23:06:34] cool. not a problem but just wanted to double check. [23:07:07] eileen: hmm, nothing with retry date set [23:07:36] hmm this is the one less /var/log/process-control/recurring_queue_consume/recurring_queue_consume-20191219-224002.log [23:08:17] the subsequent fail is weird [23:09:47] eileen looks like it waited 14 minutes before throwing that deadlock error [23:09:55] and in that time, the redis connection died [23:10:02] ejegg: the db is accessible from staging not live? [23:10:23] eileen: yeah, there's a replica of smashpig on frdev [23:10:59] hmm does that mean it held our hope? So is the message currently 'lost' [23:11:25] kindof looks like it, unfortunately [23:12:49] Wikimedia-Fundraising-Banners: Email input accepts invalid email address - https://phabricator.wikimedia.org/T240887 (jbolorinos-ctr) Ok thanks Eric, I setup a meeting in Jan for us to review this then. [23:19:41] eileen: https://phabricator.wikimedia.org/T178104 [23:20:25] so we're attempting to use a Redis feature that leaves the data on the queue till we finish processing [23:20:40] but it's not working, and it's hard to figure out why [23:20:55] but not with full success? It mentions civi exit - but I''m not seeing that [23:21:12] hey all, saw the failmail wave. Anything I can help with? [23:21:37] eileen: I guess it's also happening on the fatal error [23:22:15] Yeah but we shouldn't hit that when using api - I can see Removing failed message from the queue: [23:22:22] BUT - this is actually a lock time out [23:22:55] I'm not sure my handling there - I was focussed on deadlocks cos I assumed lock time outs would get there on the retry & not roll back [23:23:37] ok, just read up and can see you are testing stuff [23:23:45] I can go back to bad santa [23:23:48] ;) [23:23:57] yeah I was trying to hammer the queue to see if our fix worked [23:26:21] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for khancock - https://phabricator.wikimedia.org/T241154 (Dwisehaupt) Certificate generated and passed to khancock. [23:29:13] Fundraising-Backlog, fundraising-tech-ops: Issue new SSL Client Certificate for cstone - https://phabricator.wikimedia.org/T241153 (Dwisehaupt) Open→Resolved Cert in use and verified. CRL updated. [frack::puppet::private] c8b3d8e Reissuing of cstone client ssl cert [23:37:09] (PS1) Eileen: Another constraint / deadlock catch [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/559627 (https://phabricator.wikimedia.org/T182148) [23:37:32] ejegg: I think there were 2 that happened & ^^ would have addressed one [23:42:03] That one was in the damaged q so I've re-sent [23:51:11] ejegg: so I AM seeing signs of things being requeued!