[00:03:13] (PS1) Ejegg: Switch data route to use promise [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554652 [00:03:15] (PS1) Ejegg: Cache query promises rather than results [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554653 [00:03:26] oops, left some debuggy stuff in there [00:03:49] (CR) jerkins-bot: [V: -1] Switch data route to use promise [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554652 (owner: Ejegg) [00:03:52] (CR) jerkins-bot: [V: -1] Cache query promises rather than results [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554653 (owner: Ejegg) [00:09:18] (PS2) Ejegg: Switch data route to use promise for queries [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554652 [00:09:48] (CR) jerkins-bot: [V: -1] Switch data route to use promise for queries [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554652 (owner: Ejegg) [00:11:57] (PS3) Ejegg: Switch data route to use promise for queries [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554652 [00:18:09] (PS2) Ejegg: Cache query promises rather than results [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554653 [00:34:19] ok, we're in the slowdown times [00:34:24] and the queue is growing [00:34:29] might need to tweak things [00:37:53] fr-tech those dash patches seem to work for me! I can share the same query between two different browser windows [00:40:04] ejegg: is it the sheer volume of recurrings we are processing that's the issue [00:45:18] ejegg i can take a look i just got my local dash working again [00:46:25] ooo.. big english dashboard loaded fast. [00:50:05] heh, it was probably cached by someone else's request then - I haven't deployed anything new! [00:50:40] and the cache seems to have emptied by now - it's hanging on 'Chart Loading' for me [00:51:18] ejegg: just looking at watchdog - I think given we log the whole normalised message we could probably just log the fact the contribution is created & the id rather than the $contribution array - what do you think? [00:51:36] eileen yeah, probably [00:52:17] what do folks think of logging any arrays json encoded? [00:52:27] would be a little less readable for tailing logs [00:53:05] but when grepping it would get all the info out without needing to mess with the context switches [00:53:32] I like json [00:53:40] and would be more useful if we ever needed to use logs to recover a bunch of data [00:54:01] I want a terminal plugin that recognizes json and lets you right click to expand / pretty-print it [00:54:27] (PS1) Eileen: Reduce watchdog output from successful contribution.create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554655 [00:54:55] I think json is OK [00:55:24] the print_r format gets a bit mangled in syslog [00:55:33] and takes lots of lines in process-control logs [00:56:33] in terms of what to log I think it makes sense to log what we try to put into civi - but less so what civi returns from our efforts [00:57:03] ie ids created then we can use db logs from there [00:58:16] (CR) jerkins-bot: [V: -1] Reduce watchdog output from successful contribution.create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554655 (owner: Eileen) [00:59:33] (PS2) Eileen: Reduce watchdog output from successful contribution.create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554655 [01:00:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Big english queue speeds - https://phabricator.wikimedia.org/T239678 (Ejegg) We've changed the timing of the queue consumer and the thank you mailer so that they run at 3 minute intervals, with 2:30 of overlap. With this timing the queue delay has been... [01:06:46] $job[-2] is just going through the process of killing all unstructured logs (that aren't system logs). they've gone all in on using elasticsearch/logstash for all that. [01:07:02] seems crazy, but times have moved on since the advent of logging. [01:07:53] the other nice thing about it is they have a nice single place to tail/query logs and can do much more complicated queries to get better insight. [01:10:11] cool! [01:10:26] Yeah, the main production cluster here is nicely searchable [01:10:39] over at https://logstash.wikimedia.org [01:13:43] i remember one of the first things we did when testing it was testing VPN logins from multiple places to verify that you couldn't violate "speed of light" rules. used it as an additional account protection. [01:14:02] oh fun [01:14:28] heh, though geolocation can be pretty inaccurate [01:14:57] Even my hardwired net here in Cali shows up as Bogotá. [01:15:39] yeah. the main point was to test things that shouldn't be possible. like logging in from sf and then one minute later from ny [01:16:02] it has it's limits, but it was just one of many checks. [01:20:55] ejegg: that one https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/554655/ will slightly reduce our logging [02:01:41] (CR) Ejegg: [C: +2] Reduce watchdog output from successful contribution.create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554655 (owner: Eileen) [02:01:48] thanks eileen [02:03:09] (PS1) Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554664 [02:03:22] (CR) Ejegg: [C: +2] Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554664 (owner: Ejegg) [02:07:23] (Merged) jenkins-bot: Reduce watchdog output from successful contribution.create [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554655 (owner: Eileen) [02:07:41] (Merged) jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_31) - https://gerrit.wikimedia.org/r/554664 (owner: Ejegg) [02:12:55] !log updated payments-wiki from f61c9f0692 to 81921bd04a [02:12:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:44:53] Fundraising Sprint Visual Basic Instinct, Fundraising-Backlog, MW-1.35-notes (1.35.0-wmf.5; 2019-11-05): Donors being asked to log in during donation (INGENICO) - https://phabricator.wikimedia.org/T236803 (MBeat33) Resolved→Open Zendesk 674304 reported this today: //"...after I click on the... [04:04:54] (PS1) Eileen: dev/core#1444 Permit activities with campaigns on contact tab for contacts without 'administer CiviCampaign' [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/554667 (https://phabricator.wikimedia.org/T239374) [05:20:35] Fundraising-Backlog, FR-Smashpig: Amazon RecordPaymentJob is mangled on the way to the damaged table - https://phabricator.wikimedia.org/T239306 (Ejegg) [05:20:37] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Damaged message UI breaks nested arrays on re-queue - https://phabricator.wikimedia.org/T205316 (Ejegg) [08:58:15] Fundraising-Backlog: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (CCogdill_WMF) [08:58:30] Fundraising-Backlog: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (CCogdill_WMF) [08:58:41] Fundraising-Backlog: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (CCogdill_WMF) p:Triage→High [14:00:06] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Exception recovery - https://phabricator.wikimedia.org/T239679 (Ejegg) see also: T182148 [14:09:08] (PS1) Ejegg: Update English TY email [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554861 (https://phabricator.wikimedia.org/T239889) [14:31:42] Fundraising-Backlog, Patch-For-Review: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (Ejegg) OK, this is up on staging (and looks good to me). I've sent a copy to @CCogdill_WMF and to @MBeat33 . [14:31:57] Fundraising Sprint X-rays, Fundraising-Backlog, Patch-For-Review: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (Ejegg) a:Ejegg [14:35:23] Fundraising Sprint X-rays, Fundraising-Backlog, Patch-For-Review: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (CCogdill_WMF) Thank you! That looks good to me. [14:37:37] fr-tech anyone around to peek at the TY update? https://gerrit.wikimedia.org/r/554861 [14:38:32] ejegg: I'm here... [14:38:43] mornin AndyRussG ! [14:38:45] This one, I guess? https://gerrit.wikimedia.org/r/554861 [14:38:51] yep! [14:39:20] buongiorno! [14:39:35] turns out that a whole lot of donors are taking that question mark as an invitation to write to donate@wikimedia.org with their motivations [14:40:08] which would be great, but they're mixed in with actionable complaints [14:40:19] so it was really messing with Donor Services' workflow [14:40:36] :O [14:41:23] howdy Jack [14:42:53] does anyone else with debian on a thinkpad keep getting notifications for firmware updates, even after installing them? [14:43:09] hmmm no, where do you get those? [14:43:33] it's via the 'Software' thing that's sort-of in the control panel [14:44:01] I'm on ubuntu [14:44:04] I think I might have gotten something like that when i had a kernel from unstable but the firmeware package didn't support it yeti [14:44:15] btw howdy ejegg AndyRussG :) [14:44:32] another great day yesterday [14:44:42] :) [14:47:29] jgleeson: helo [14:50:24] Fundraising Sprint X-rays, Fundraising-Backlog, Patch-For-Review: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (MBeat33) Thanks, all, this is greatly appreciated! [14:51:21] ejegg: the ty change looks fine! I haven't tested it... I guess it's OK to +2 even so? [14:53:05] AndyRussG: I can test it [14:53:15] I just put through a local donation [14:53:22] so can pull in the new email to test it [14:53:31] if that helps! [14:53:59] jgleeson: ah ok sounds great, thanks! [14:54:23] I'll just +1 it then with the same note, and let you take it from there, then, maybe? [14:55:10] AndyRussG: It's up on frdev [14:55:19] so you can you the email test form there [14:55:37] ejegg: ah cool, how do I do that? [14:56:27] looks good to me [14:56:30] https://phabricator.wikimedia.org/F31458522 [14:56:41] old on the left, new on the right [14:56:52] nice [14:57:17] (CR) Jgleeson: [C: +2] Update English TY email [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554861 (https://phabricator.wikimedia.org/T239889) (owner: Ejegg) [15:03:05] (CR) AndyRussG: [C: +2] "Cool, looks great! -Testy McTestersen" [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554861 (https://phabricator.wikimedia.org/T239889) (owner: Ejegg) [15:03:12] (Merged) jenkins-bot: Update English TY email [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/554861 (https://phabricator.wikimedia.org/T239889) (owner: Ejegg) [15:04:23] jgleeson: ah cool, nice way to check! [15:10:49] (PS1) Ejegg: Update filter values for 2019 [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554887 [15:12:26] (PS1) Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554888 [15:13:55] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554888 (owner: Ejegg) [15:32:14] Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Campaign archived and yet enabled - https://phabricator.wikimedia.org/T238895 (Jseddon) [15:39:05] (CR) Ejegg: [V: +2 C: +2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - https://gerrit.wikimedia.org/r/554888 (owner: Ejegg) [15:41:39] !log updated Fundraising CiviCRM from 4a72ad4e63 to 30cdc5fa59 [15:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:45:40] Fundraising Sprint X-rays, Fundraising-Backlog: Update TY email copy, pt 2 - https://phabricator.wikimedia.org/T239889 (Ejegg) OK, this is deployed [15:46:10] fr-tech now that TY emails are not running behind any more, I'm going to try turning the CiviMail records back on [15:46:26] great ejegg! [15:46:39] Let's keep an eye on that graph [15:46:51] this may be a dumb question but will we have any idea how to backfill old data? [15:47:18] to create civi activities from the thank_you_date of the contributions? [15:47:45] We could write some code to do it if we really thought we needed to [15:47:52] hmm [15:48:05] but for example we would have to restore the old TY template somewhere [15:48:18] to synthesize the text that we actually sent [15:48:29] i'm just thinking the data will be uneven and might throw off DS [15:48:38] Fundraising-Backlog: Ingenico changes to Ideal flow for our roadmap - https://phabricator.wikimedia.org/T239924 (Aklapper) @EMartin: Assuming this task is about #Fundraising-Backlog, hence adding project tag so others can find this task when searching for tasks under that project or looking at that project w... [15:48:51] but if the thank_you_date is there hopefully that will help [15:49:09] i'm less worried about the email text and more thinking about if a DS agent is responding to a thank you email reply [15:49:24] I guess we can just mention the gap in CiviMail activities in standup today [15:49:49] !log re-enabled creating CiviMail activities when sending Thank You emails [15:49:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:06] heading out for an errand, back for standup [17:07:37] Fundraising-Backlog: Transactions stopped for Fraud are unscored - https://phabricator.wikimedia.org/T239769 (EMartin) More of these 600's appearing in the fraud search today 12.5: 73517561.1 73485102.1 73484895.1 73484088.1 73516052.3 73481987.1 73480663.1 73515520.2 73514644.1 73513909.1 [17:07:48] fr-tech anyone else looking at those orphan slayer fails? [17:10:16] nope [17:10:19] but I can do [17:11:30] ooh, looks like it's trying to warn us of low minfraud queries, and that codepath uses a mediawiki-only function [17:11:57] let's see if we can disable that warning in the orphan slayers [17:12:17] hmm, though that makes me wonder why we aren't getting that warning from payments-wiki [17:15:13] so the alarm limit message caused the unhiding of the wfMemcKey() failure? [17:25:17] jgleeson yep! [17:25:30] we want to send an email when there are too few queries remaining [17:25:39] but we don't want to sent one for EVERY donation at that point [17:26:02] so we note in memcache that we have sent the warning [17:59:26] fr-tech here's a related teeny bugfix: https://gerrit.wikimedia.org/r/554621 [18:13:53] Fundraising Sprint Visual Basic Instinct, Fundraising Sprint X-rays, Fundraising-Backlog, MW-1.35-notes (1.35.0-wmf.5; 2019-11-05): Donors being asked to log in during donation (INGENICO) - https://phabricator.wikimedia.org/T236803 (jgleeson) [18:32:12] Fundraising Sprint X-rays, Fundraising-Backlog: Transactions stopped for Fraud are unscored - https://phabricator.wikimedia.org/T239769 (DStrine) [18:32:15] Fundraising-Backlog: Ingenico changes to Ideal flow for our roadmap - https://phabricator.wikimedia.org/T239924 (EMartin) Yes, sorry for leaving that off. [18:37:30] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM: Weird FK constraint problems in Civi - https://phabricator.wikimedia.org/T182148 (DStrine) [18:46:54] Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface: Sending invalid currency codes to Ingenico - https://phabricator.wikimedia.org/T239941 (Ejegg) [19:13:35] dstrine: I'm afraid I'm not going to have much more info on those 'stuck at 600' txns in 15 min [19:13:49] just helping get lunch ready here in fact [19:13:56] fr-tech can anyone else attend? [19:15:47] I was just about to jump off but if it's important I can attend whatever it is ejegg ? [19:16:29] maybe cstone? [19:16:53] hey yeah I can I just finished eating [19:16:58] ejegg it looks like this is the 1:1 with evelyn, dstrine do you have mroe info on what is needed? [19:16:59] thanks cstone ! [19:17:09] mepps yeah, that's the one [19:17:27] cool thanks ejegg [19:18:24] Fundraising Sprint Visual Basic Instinct, Fundraising Sprint X-rays, Fundraising-Backlog, MW-1.35-notes (1.35.0-wmf.5; 2019-11-05): Donors being asked to log in during donation (INGENICO) - https://phabricator.wikimedia.org/T236803 (jgleeson) I'm looking into this. Not found anything as of yet [19:19:54] cstone so I just looked in the logs for 73513909.1 [19:20:07] and that person came back to the ResultSwitcher with a dead session [19:20:28] but in that case, the orphan rectifier should have caught the payment at 600 and pushed it through [19:20:37] so the next thing to do is to look in the orphan rectifier logs [19:20:47] and to see how bad the orphan rectifier backlog is [19:21:04] because if it's trying to push through payments from a day ago, it's not going to work [19:21:16] so maybe we need to delete some old messages from the pending table [19:21:32] dstrine: can you plz send cstone that invite? [19:21:45] thanks ejegg [19:22:05] thank you! [19:30:05] Wikimedia-Fundraising-Banners: Donate Later link does not work after donation amount has been selected - https://phabricator.wikimedia.org/T238604 (Pcoombe) [19:32:59] Wikimedia-Fundraising-Banners: Donate Later link does not work after donation amount has been selected - https://phabricator.wikimedia.org/T238604 (Pcoombe) Oops, I actually meant to do that merge the other way round! Copying Sam's comment here: > For the sake of expedience, I'm choosing the former (letting... [20:03:56] fr-tech reminder that there's a tech-talk on right now about "the manager's path" [20:38:43] Fundraising Sprint X-rays, Fundraising-Backlog: Transactions stopped for Fraud are unscored - https://phabricator.wikimedia.org/T239769 (Ejegg) OK, so most of these people got back to payments-wiki with a dead session. Not sure if that means our session storage is filling up or if it's just browser quirk... [21:08:26] Fundraising Sprint X-rays, Fundraising-Backlog: Transactions stopped for Fraud are unscored - https://phabricator.wikimedia.org/T239769 (Ejegg) 73485102.1 - orphan rectified failed because it was trying to warn us of low MinFraud queries and hit a codepath that doesn't work under Civi. 73484895.1 - looks... [21:11:11] (CR) XenoRyet: [C: +2] Fix error handling on minfraud emergency mail [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/554621 (owner: Ejegg) [21:12:42] (Merged) jenkins-bot: Fix error handling on minfraud emergency mail [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/554621 (owner: Ejegg) [21:19:07] fr-tech here's the simple patch for dash (just updating filter values): https://gerrit.wikimedia.org/r/554887 [21:19:42] its looking good for me ejegg only the filters arent bring up any of my test data, i just wonder if its too test-datay [21:19:56] cstone I'd bet [21:20:20] yeah [21:20:23] what do the utm_medium rows look like in your contribution_tracking table? [21:20:29] errr values that is [21:21:16] hah they are null for the ones i was trying to get to show up, that explains things [21:21:49] mepps do you remember anything about ingenico donations coming back with just {"status":"IN_PROGRESS"} [21:22:16] does that mean we've redirected them and they haven't submitted anything? [21:22:54] we seem to go ahead and run the minfraud score after that, so we should quit doing that [21:23:26] argh, we should pay this much attention to the logs during more of the year [21:24:37] (CR) Cstone: [C: +2] "Looks good!" [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554887 (owner: Ejegg) [21:25:22] (Merged) jenkins-bot: Update filter values for 2019 [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554887 (owner: Ejegg) [21:25:58] Fundraising-Backlog, FR-Ingenico, MediaWiki-extensions-DonationInterface: Ingenico orphan rectifier should not run fraud filters when status result comes back {"status":"IN_PROGRESS"} - https://phabricator.wikimedia.org/T239951 (Ejegg) [21:26:11] Thanks cstone! [21:26:22] looking back at the other ones now [21:26:33] cool, I'll just deploy this stuff for right now [21:29:11] (PS1) Ejegg: Quit redundantly setting SQL text [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/554953 [21:30:58] ooh right, deploying dash logs ppl out, and that isn't handled too well by the chart refresh [21:31:13] i'll wait till you weigh in on the data route stuffs then [21:34:30] cstone (and fr-tech): here are the mockaroo schemas to generate more test data for dash: https://mockaroo.com/projects/346 [21:35:09] looks like utm_medium is always one of sitenotice,waystogive,email [21:35:35] hmm that link just brings me to the homepage, should i make an account? [21:35:39] heh, and the ts is always in 2015 [21:35:45] *2014 [21:35:50] cstone oh, I guess so [21:36:48] today just took #5 spot on top days chart [21:37:01] so this year has 1, 2, and 5 [21:37:13] now i get "The page you were looking for doesn't exist." [21:37:18] hrm? [21:37:23] maybe I need to share it with you [21:37:33] lemme remind myself how that site works [21:37:54] oh hey, I guess I do need to share it [21:38:18] fr-tech who else has a mockaroo account? [21:38:24] cstone I just added you to the project [21:39:26] cstone ok, so there's a couple things to tweak to get some consistent data there [21:39:51] ok i dont see anything in projects yet [21:40:05] does this link load anything ? https://mockaroo.com/projects/346 [21:40:23] oh, and did you sign up with your @wikimedia.org address? [21:40:33] yep the @wikimedia and still the doesnt exist lnk [21:40:38] grr [21:41:11] weird, looks like the share setting didn't take [21:41:17] trying to save the share settings again [21:41:29] i see it! [21:41:35] great! [21:41:45] OK, I'mma at least update that ts column definition [21:41:58] I think it's using the wrong format, and it's 5 yrs out of date [21:43:23] hmm, which of these formats is right? [21:43:40] Wikimedia-Fundraising-Banners: Desktop Sm Nag is too large and layout issues observable on IE 11 - https://phabricator.wikimedia.org/T239953 (jbolorinos-ctr) [21:51:08] Wikimedia-Fundraising-CiviCRM: Benevity to change the reports we receive for Benevity import - https://phabricator.wikimedia.org/T239954 (RLewis) [22:12:54] looks like a db lock [22:12:56] let's see [22:13:13] Fundraising Sprint X-rays, Fundraising-Backlog: Transactions stopped for Fraud are unscored - https://phabricator.wikimedia.org/T239769 (EMartin) Hi Elliott, The examples I sent were from yesterday so definitely longer than 2 hours delayed. [22:13:29] on the address table - maybe it's that proximity search smart group? [22:14:40] nope, it's a manual dedupe [22:14:49] funny tho, looks like it has decent limits [22:16:28] oh wait, that subselect is not limited [22:16:46] and is a self-join on the address table [22:16:47] gross [22:23:35] huh, that blip shows up on all the graphs except for the outgoing mail queue [22:28:17] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Civi-Dedupe: Runaway Civi dedupe query on first 5 chars of address table - https://phabricator.wikimedia.org/T239956 (Ejegg) [22:29:07] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Civi-Dedupe: Runaway Civi dedupe query on first 5 chars of street_address field - https://phabricator.wikimedia.org/T239956 (Ejegg) [22:30:53] XenoRyet: any ideas on that recurring mistiming? [22:31:04] hi eileen! [22:31:16] hey just saw the runaway query [22:31:30] that shouldn't happen unless someone created a new rule.... [22:31:32] heh, i thought that might have been the batsymbol that brought you here [22:31:48] ah, let's see if there's a new rule then [22:32:47] blech, also, let's delete all this paypal_ec pending from the damaged table so we can make sure we're not missing important stuff that we should requeue [22:33:21] That's weird the user id is not an expected one [22:33:49] eileen it's a DS rep [22:34:31] guessing you saw https://phabricator.wikimedia.org/T239956 [22:34:41] yep [22:35:43] ok contact 20561252 edited the dedupe rule it looks [22:36:05] that's mepps [22:36:10] ooh [22:36:15] the plot thickens! [22:36:27] rule 17 [22:36:49] Hmm I created a new rule but never edited an existing one [22:37:00] yeah the odd thing is your rule is 7 chars [22:37:00] That I know of.. [22:37:04] * ejegg takes a second to realise rule 17 is not an internet universal [22:37:10] It was in a DS call [22:37:32] so here is the thing - if you use the length field in a dedupe rule the resulting query is unindexed [22:38:02] Huh [22:38:16] but the one created has a length of 7 & the query is looking at 5 [22:38:29] see (SUBSTR(t1.street_address, 1, 5) = SUBSTR(t2.street_address, 1, 5) [22:38:55] because SUBSTR is a calculation it's comparing the results of 2 calculations -not doing an indexed join [22:39:34] yep, problematic on a db this size [22:39:56] some must have tried to use dedupe rule 3 - households [22:40:11] I'm going to remove the length var from both rules [22:41:32] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Civi-Dedupe: Runaway Civi dedupe query on first 5 chars of street_address field - https://phabricator.wikimedia.org/T239956 (Eileenmcnaughton) A DS agent tried to use dedupe rule 3 - see how it has 5 in length? That results in it comparing calcul... [22:41:57] Fundraising Sprint X-rays, Fundraising-Backlog: Transactions stopped for Fraud are unscored - https://phabricator.wikimedia.org/T239769 (Ejegg) @EMartin I can look up yesterday's examples too, but if you search in the Ingenico console now I believe you will see 73513909.1, 73514644.1, 73515520.2, and 735... [22:42:42] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Civi-Dedupe: Runaway Civi dedupe query on first 5 chars of street_address field - https://phabricator.wikimedia.org/T239956 (Eileenmcnaughton) I actually wrote a whole blog post about this a while back https://civicrm.org/blog/eileen/deduping-for... [23:04:38] (PS1) Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/554968 [23:04:55] (CR) Ejegg: [C: +2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/554968 (owner: Ejegg) [23:05:27] (Merged) jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - https://gerrit.wikimedia.org/r/554968 (owner: Ejegg)