[00:04:40] (PS20) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [00:04:48] ejegg: have you got a few minutes? [00:07:56] eileen1: sure, what's up? [00:08:32] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [00:08:53] so, discussing with Leanne about the import [00:09:10] I think she wants something a bit closer to what we did with orgs for those contacts with no email [00:09:17] i.e force there to be a match [00:09:32] (CR) Ejegg: Matching gifts import. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [00:09:39] so, I'm thinking along the lines of either using the contact source field [00:09:48] (easy but maybe a bit greedy) [00:09:53] or a tag [00:10:08] for them to mark specific contacts as benevity givers [00:10:23] oh huh [00:10:35] so, if we have first name & last name [00:10:40] and in that case we count them as a match on just first & last? [00:10:54] yeah - but if there are already multiple matches in the db [00:10:56] hmm, might want to match the org too [00:11:26] hmm are they always employers in Benevity [00:11:26] which means using the employer field, I guess... /scopecreep [00:11:53] I mean - can we say if someone has a matching gift from ACME corp we should create an employer relationship to ACME corp? [00:12:06] yeah, that seems totally legit [00:12:15] ok - I think I can work with that [00:12:34] if that is true then I feel like using that field doesn't set me back from a performance point of view [00:12:43] then if we have John Smith from acmeCorp with no email, and John Smith from BetaCorp with no email, we can tell them apart [00:12:52] yes [00:13:08] and having that relationship data recorded would actually improve the data in Civi [00:13:12] cool, as long as that doesn't seem to painful to implement! [00:13:16] *too [00:13:17] I'll email Leanne but I think that is good [00:13:23] no, not too painful. [00:13:34] right on [00:13:44] perhaps some pain in initial data wrangling - but I think that is ok [00:14:09] there is also a transactional issue [00:14:15] these things are monthly files, right? Definitely will pay off before too long [00:14:31] yeah - and same contacts over & over with a few new ones [00:14:57] ohhh right, could possibly create the one donation but not the match if there's a failure [00:15:02] good catch [00:15:40] yeah that - but also the issue is that if you create the contact & then can't create the organization you can get a proliferation of contacts [00:15:49] sorry - can't create the contribution [00:16:05] ah, yep. [00:16:35] dang, there is already a bit of duplication with the core wmf_import code, but I guess we need to pull in the transactionality too [00:16:53] so - I remember last time I looked at the transactions I got a bit tripped up on it - but I have another ticket in my queue that relates so I guess I need to get my head around it [00:17:22] BTW leanne was pleased with the import overall & also we looked at how she could mark a bunch of contacts with a tag [00:17:29] wait, if we are creating a new contact, we can just let the normal import function handle that [00:17:47] and then get the soft credit ID out [00:18:03] so we don't have to handle the location update and contact/contribution transactionality [00:18:24] except we are being tighter on contact creation [00:18:46] right, if there's a match we don't create a new one [00:18:57] but more particularly we want the 2 contributions in one transaction [00:18:58] but if there's no match, we could have the getContactId return -1 [00:19:32] and just send the right contact & address fields to wmf_import [00:19:37] yes, but it doesn't solve the 2 contributions part [00:19:41] yeah [00:19:58] I do think that's worth doing just to simplify this code [00:20:23] hmm does it simplify it - let me think [00:20:25] and in case we add something else in the wmf_import having to do with the contact [00:20:50] a la update_location [00:22:17] yeah - although we call update location anyway [00:23:20] Fundraising Tech Backlog: fix silverpop database updater not to crush the master database with long queries - https://phabricator.wikimedia.org/T157600#3011755 (Ejegg) So, I think we're just using the wrong isolation level. from https://dev.mysql.com/doc/refman/5.7/en/innodb-locks-set.html : > INSERT INTO... [00:27:09] (PS1) Ejegg: Fix transaction isolation level [wikimedia/fundraising/tools] - https://gerrit.wikimedia.org/r/336739 (https://phabricator.wikimedia.org/T157600) [00:28:43] eileen1: yeah, that feels like duplication [00:29:11] I'll think about it when I go through this [00:40:19] (CR) Eileen: "Thanks for the review - I think I've responded to your comments." (4 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [00:41:15] (PS21) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [00:43:09] going to head out for the evening. AndyRussG, I'll get to reviewing your code first thing tomorrow! [00:44:41] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [01:18:59] (PS22) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [01:22:51] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [02:22:38] Fundraising Tech Backlog, Patch-For-Review: fix silverpop database updater not to crush the master database with long queries - https://phabricator.wikimedia.org/T157600#3011913 (Jgreen) IMO the approach of monolithic select-insert queries with huge tables does not scale, is already a problem, so that's... [02:47:33] (PS23) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [02:51:31] (CR) Eileen: Matching gifts import. (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [03:59:13] Fundraising Tech Backlog, Patch-For-Review: fix silverpop database updater not to crush the master database with long queries - https://phabricator.wikimedia.org/T157600#3010439 (Eileenmcnaughton) In my performance tests on other queries ON DUPLICATE KEY UPDATE has been slow. I have found something like... [05:12:06] (PS24) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [05:26:29] (PS25) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [05:33:48] (PS26) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [05:57:24] (PS27) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [06:08:02] (PS28) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [07:18:32] (PS29) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [07:39:44] Fundraising Sprint Autotune Earphones, Fundraising Sprint Baudelaire Bowdlerizer, Fundraising Sprint Costlier Alternative, Fundraising-Backlog, and 3 others: Create an import method for matching gifts and payroll deductions - https://phabricator.wikimedia.org/T115044#3012232 (Eileenmcnaughton) Af... [08:55:13] (PS30) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [08:59:54] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [09:55:18] (PS31) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [09:58:11] (PS32) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [10:01:49] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [10:30:02] (PS33) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [10:37:35] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [10:40:33] (PS34) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [10:47:42] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [11:13:28] (PS35) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [11:21:16] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [12:41:43] (PS36) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [12:46:32] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [12:56:14] (PS37) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:00:38] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [13:04:04] (PS38) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:07:19] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [13:12:37] (PS39) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:17:36] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [13:23:20] (PS40) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:28:22] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [13:30:57] (PS41) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:34:14] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [13:35:39] (PS42) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:42:27] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [13:47:23] (PS43) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [13:48:53] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [17:43:55] back in 10 min [18:00:31] fr-tech: Yes, I've now got this nice little apartment in New York, one of those [18:00:31] L-shaped ones. Unfortunately, it's a lower case l. [18:00:31] -- Rita Rudner [18:00:31] -- discuss. [18:20:48] fr-tech anyone want to discuss stuff? [18:22:15] ejegg: whatcha got? [18:23:45] nothing new from me [18:25:16] ejegg: as far as that transaction isolation level, i think any write to the master will block replication regardless [18:25:44] so any long running write will eventually be a problem [18:25:48] hmm [18:25:55] do we replicate the silverpop tables? [18:26:01] I feel like we don't need to [18:26:16] they're just derived data, and nothing uses them on the replica [18:27:10] yeah, i need to learn more about how replication works [18:27:56] but i don't think it cares about transactions [18:28:30] it think it's more like, whenever the master finishes a write it queues it up on the replicas [18:29:33] (PS3) XenoRyet: Add argument to setGatewayDefaults base function [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336683 (https://phabricator.wikimedia.org/T157618) [18:29:50] k. so we could maybe avoid the table locks with that txn isolation level, but replication would still block [18:29:57] Jeff_Green: am i understanding that correctly? [18:30:21] let's make a list of tables we don't have to replicate [18:30:29] ejegg: that's my impression of the situation, and replag is what the fail mail is about [18:30:55] well, replag is one failmail, and dedupe failing coz of locks is another [18:31:15] yeah [18:31:32] that's true [18:31:43] so txn might help dedupe out [18:32:21] reading backscroll [18:33:09] ok so there are a couple problems, i think what you've both said is correct so far [18:33:11] Jeff_Green: oh wait, is it even possible to replicate selected tables? [18:33:34] it is possible to exclude tables and databases from replication [18:33:38] but still [18:33:48] but with statement based replication wouldn't that make some queries shit? [18:33:57] if it joined on a table we don't replicate [18:34:02] do you understand what I'm saying about the paradigm of insert...select construct? [18:35:47] probably not! [18:36:18] that necessarily locks all the tables involved the whole time [18:36:28] and it constrains the activity to having to happen entirely on the master [18:36:42] yes you can reduce the impact of locking with the isolation level trick [18:37:09] and you can reduce the impact on replication in a couple of ways, but you still create contention that doesn't need to be there [18:37:38] so the locks are the same whether you're writing to a replicated table or a temp table? [18:39:10] i think they'd have to be [18:39:41] ah, ok [18:39:44] i think the isolation trick improves things by maybe making the locking more granular, i.e. by row, I'm not sure [18:40:38] so that's a huge help for the locking part, but I think you still burn a bunch of disk/cpu time on the master to do it, where we've got a whole equivalent mostly-idle machine nearby [18:40:57] so does that thing from the docs not apply because of replication? 'InnoDB does the search on S as a consistent read (no locks)' [18:41:15] lemme re-read, I don't remember that part [18:43:37] sorry, I'm just being lazy about changing the python code to dump and reload - I got used to just tweaking that one SQL file [18:45:13] i'm having a hard time wrapping my head around the explanation of "InnoDB does the search on S" documentation, it ~looks~ like it's saying you can essentially go from roughly row-locking to roughly no-locking depending on the isolation level [18:45:41] that's what it sounds like [18:46:26] elsewhere it sounds as though due to replication the starting point is the entire source table is read-locked [18:47:23] ah, gotcha, I didn't read any of the replication-related docs [18:47:54] lemme see if i can find that one [18:48:22] https://www.percona.com/blog/2006/07/12/insert-into-select-performance-with-innodb-tables/ [18:50:51] ok, and we're still doing statement-based replication, so the 'fix' in 5.1+ doesn't apply [18:51:36] we're doing 'mixed' replication so it switches back and forth depending on a blackbox of mystery [18:52:46] i'm still sorta baffled how a 35s select plus a 6-10min insert bloats to 40 min when done as a single combined query [18:53:12] yeah, that seems insane [18:56:53] eileen said something about a missing index [19:00:37] cwd|brb: the first query i looked at, the "40min" one just pulls email addresses from a log table in the civicrm database, that table has about 33M rows, but only about 11M unique email addresses [19:01:36] she posted a one-shot query, based on a join of source+dest tables that would be good for backfilling new addresses from the civi source table [19:02:08] but since there's no index on the email column in the civi table, doing that join would be painful [19:04:36] she also suggested a different construct than using ON DUPLICATE KEY, which is slow, but I'm not sure why we don't just do INSERT IGNORE, it seems like the end effect is the same [19:04:50] no index on the email column? [19:04:58] oh, log table, nvm [19:08:19] i just tried the original query switched to INSERT IGNORE and it ran in 31 min, so that might help a little [19:09:46] ejegg something else I was pondering is whether doing read on slave and write on master could just be done on the fly, i.e. by opening two db handles and passing the data through python [19:12:40] Fundraising Sprint Baudelaire Bowdlerizer, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: DjangoBannerStats: Handle banner loader error - https://phabricator.wikimedia.org/T152723#2858103 (DStrine) [19:12:53] hmm, still seems like a lot of network overhead. Any reason not to just do all the writes on the slave too (and not having a silverpop db on master)? That export script is seriously the only thing that needs it. [19:13:16] hang on, there was some reason we didn't use insert ignore [19:13:24] Fundraising Sprint Costlier Alternative, Fundraising Sprint Qwerty Thwacking, Fundraising-Backlog, MediaWiki-extensions-CentralNotice, Unplanned-Sprint-Work: CentralNotice banner sequence: implement feature for MVP - https://phabricator.wikimedia.org/T144453#2600241 (DStrine) [19:13:55] Jeff_Green: oh right, insert ignore actually ignores all sorts of potential errors, not just duplicate keys [19:14:47] also, insert ignore still emits warnings, just not errors [19:17:35] Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, fundraising-tech-ops: What fundraising dbs and tables to we not need to replicate? - https://phabricator.wikimedia.org/T157717#3014249 (Ejegg) [19:18:29] oh wow, looks like eileen was up till the wee hours working on that matching gift import [19:18:53] ejegg ah that makes sense [19:20:22] so what do you want to try first? [19:20:32] just turning off replication for silverpop? [19:20:41] i meant the insert ignore thing [19:21:00] ah yeah. you can end up with nulls in not null columns and stuff [19:21:44] I think either we would have to dedicate a slave just to this, or you'd still have to fix the core problem of queries that block replication [19:25:11] I think it wouldn't be as bad as having it on the master, i.e. even if it's 40 minutes behind that should only be in query execution--i.e. if the master blew up we'd still have the latest activity already transferred and in binary logs on the slave [19:26:50] is it worth trying turning off replication for just the destination db? [19:27:07] and seeing if the locks are still happening [19:28:45] how hard would it be to do it all in temporary tables? [19:29:49] not hard at all - i think we were doing that for a while b/c we thought it would avoid these locks [19:30:17] ohhh, but it does make it much harder to debug [19:30:26] seems like that would have the same effect [19:31:04] meh yeah i see what you mean [19:31:05] hmm, we did have a hell of a time figuring out issues when we didn't have any of the intermediate tables to look at [19:31:34] i was thinking there's a type of temp table that can be shared across sessions [19:31:40] oh? [19:31:57] afaict i was wrong :-) [19:35:27] trying to figure out how this would work. the way we're ignoring tables now is done on the slave side, I don't think the master is aware of the change [19:36:37] ah binlog-ignore-db [19:40:02] ok, so the master would still lock as though it was replicating everything [19:40:16] well, how about ignoring replication for a whole db? Would master know about that? [19:40:37] binlog_ignore_db looks like it might do it [19:41:02] but i keep finding caveats to using it.... [19:42:24] the main caveat seems to be that it makes it easy to accidentally break replication [19:43:10] apparently the filter happens on statements/queries that are made while the client using the database in question [19:43:46] i.e. so after "use silverpop" everything doesn't get logged, but say you've done "use civicrm" and then "update silverpop.foo set blah", that query would replicate [19:43:54] seems insane but there it is [19:45:29] backscrolling... [19:46:40] so...perhaps let's try to stay away from that route [19:46:57] i'd rather move it back to lutetium than that [19:47:21] but you'll still have to figure out a way to make it stop blocking replication for more than a couple minutes [19:50:17] actually now that I think of it, we already know that changing what goes into binlogs doesn't fix locking--if it did it we'd see the difference on lutetium which doesn't have binlogging enabled [19:51:05] imo having the python script just shell out what jeff did yesterday - dump to file, sort -u, make silverpop db - seems sane and fast. could even push the new db to somewhere that doesn't get replicated since that's not necessary, then i think we'd only be looking at the 35s lock [19:51:28] also could easily chunk it [19:51:53] there is some network overhead but the network at the dc is pretty dang fast [19:52:19] might not even be the bottleneck in that situation [19:52:45] i've done this kind of thing in perl via DBI, but haven't done a ton of python database stuff [19:53:49] you mean the db server switching part? [19:54:32] figuring out how to sling around large data without horrible buffering surprises [19:54:43] ah yes [19:54:45] mysql_use_result() is the one thing that is forever burned into my brain [19:55:30] nodejs might be ideal for this [19:55:38] but i'm guessing you can achieve the same thing with python [19:56:00] but yeah i do not know how the buffering would work [19:57:25] i'm positive it's doable, just may have to avoid high-level libraries that don't give you control over how things are buffered [20:00:04] yeah [20:00:19] sounds kinda fun :S [20:01:49] BIAB i gotta go shovel snow, we've got about 10" already today! [20:06:14] (CR) Ejegg: [C: -1] "I really like how this now treats the individual donation as the 'real' message all the way through the doImport function, and lets the st" (8 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [20:08:45] (CR) Ejegg: "recheck" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336683 (https://phabricator.wikimedia.org/T157618) (owner: XenoRyet) [20:16:43] (CR) Ejegg: [C: 2] Add argument to setGatewayDefaults base function [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336683 (https://phabricator.wikimedia.org/T157618) (owner: XenoRyet) [20:18:44] (Merged) jenkins-bot: Add argument to setGatewayDefaults base function [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336683 (https://phabricator.wikimedia.org/T157618) (owner: XenoRyet) [20:33:35] finally found a good trick for dealing with the greenhouse.io comment box: keeping a note of the people i want to @mention that i can cut and paste [20:33:46] the autocomplete is broken in several ways [20:36:32] ejegg: i would volunteer to do that python stuff if you agree it's a good approach [20:36:37] break up the export [20:37:18] the snow is going to murder me. I got maybe 1/3 done and I'm exhausted [20:37:42] Kinda feels like exporting American parts to China for assembly and shipping the finished product back [20:38:09] but hey, if it makes sense for the global economy... [20:38:13] Jeff_Green: wet and heavy? [20:38:27] heh efficiency is weird [20:38:28] Can we try just turning off silverpop db replication first? [20:38:33] just lots [20:38:40] oh jeez [20:38:55] (PS1) XenoRyet: Add payment method to PayPal EC [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336888 (https://phabricator.wikimedia.org/T157618) [20:38:55] time to hire some guy with a snowblower? [20:39:05] i saw this thing where they send these scottish shrimp to china to be shelled and then back to scotland and they advertise "local shrimp" [20:39:28] guess that's accurate [20:39:52] ejegg probably a good idea [20:39:57] snowblower guy [20:40:15] homecoming shrimp? Grand tour shrimp? [20:40:19] heh [20:40:35] ejegg replication doesn't seem to affect the length of the query [20:40:41] oh, bummer [20:40:59] lutetium doesn't have binlogs enabled, takes just as long there [20:41:25] it would help replag that's true, but so would just kicking it off the master [20:41:35] How about with isolation level READ COMMITTED ? [20:42:02] isn't the network at the DC faster than the disks? [20:42:09] write wise anyway [20:42:21] good question [20:42:38] i mean gig E is faster than writing to any disk [20:42:45] i know it takes 35s to dump 33M rows to a flat file across the wire :-) [20:43:19] yeah i feel it's safe to treat the DC as one big computer with respect to that [20:43:26] of course to codfw is different [20:44:20] but i imagine we could have the python script xfer across the network without writing to disk [20:44:37] that's worth trying [20:45:32] our goal should be to get everything we can off the master, to reduce times we're scraping logs in search of contention that blocks civi users [20:49:47] Fundraising-Backlog, Analytics-Kanban, Patch-For-Review: Productionize banner impressions druid/pivot dataset - https://phabricator.wikimedia.org/T155141#3014664 (Nuria) Open>Resolved [20:52:35] Fundraising-Backlog: Update on call protocols - https://phabricator.wikimedia.org/T157731#3014668 (ggellerman) [21:28:33] (PS1) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/336898 (https://phabricator.wikimedia.org/T115044) [21:37:25] what is with the new * menu in phab? not very intuitive [21:38:53] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/336898 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [21:40:34] (PS44) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [21:40:43] (Abandoned) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/336898 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [21:44:20] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [21:44:46] Jeff_Green: do you know if there are automated processes that access the silverpop db, or does ccogdill export it by hand? [21:50:23] cwd the same script that generates the db also exports it to file and uploads it [21:51:04] ah so it's ending up on the jenkins box anyway? [21:51:10] yep! [21:51:24] (CR) Eileen: "As you can see the failing test drove me to distraction last night - mostly because I can't get it to fail locally so I can't figure out w" (6 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [21:51:44] So i guess it's like the ore for the parts was mined in China [21:52:28] heh [22:11:57] (CR) Ejegg: [C: 2] "Looks like that should do it!" [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336888 (https://phabricator.wikimedia.org/T157618) (owner: XenoRyet) [22:13:51] (Merged) jenkins-bot: Add payment method to PayPal EC [extensions/DonationInterface] - https://gerrit.wikimedia.org/r/336888 (https://phabricator.wikimedia.org/T157618) (owner: XenoRyet) [22:17:05] sorry, just back from round 2 of shoveling, we can finally get one care out of the driveway [22:17:18] brutal [22:17:46] then a solid five minute coughing fit when i came back in. why do I like snow?? [22:18:06] there are a couple jenkins jobs that I think may be the only things talking to that db [22:18:49] oh ha I didn't see that ejegg had responded already [22:19:07] carry on, i'm just going to lie on the floor and croak [22:19:31] :( [22:19:41] oh jeez, don't have a heart attack before you teach cwd everything! [22:19:57] but seriously, don't [22:20:13] yeah long term snow shoveling is also not the type of activity that leaves you without back problems [22:21:17] our current driveway is long, steep, and gravel. it's horrible to shovel. [22:21:36] but if you drive over it one time in the snow without shoveling it's over with [22:21:51] won't be getting back up [22:23:26] yeah, even our short paved driveway has that problem, gotta get a running start in the car with traction control or it won't climb it [22:25:02] (PS45) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [22:29:27] (PS46) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [22:34:45] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [22:37:11] Fundraising-Analysis, MediaWiki-extensions-DonationInterface: Accept GNU Taler for donations - https://phabricator.wikimedia.org/T157529#3015145 (Koavf) [22:39:33] GNU Taler? [22:39:53] stallman-approved cryptocoin? [22:43:02] Fundraising-Analysis, MediaWiki-extensions-DonationInterface: Accept GNU Taler for donations - https://phabricator.wikimedia.org/T157529#3008266 (Ejegg) Hi @Koavf, we depend on an external servicer to accept cryptocoins and immediately cash them out, so this will depend on someone like Coinbase supportin... [22:45:22] Fundraising-Analysis, MediaWiki-extensions-DonationInterface: Accept GNU Taler for donations - https://phabricator.wikimedia.org/T157529#3008266 (MZMcBride) This task is stalled, then? [22:48:14] eileen2: those tests pass for me locally too :( [22:49:26] hmm, but I did manually create a bunch of the orgs in the test csv when I was smoke testing [22:52:59] fr-tech I'm heading to a language exchange meetup, but I'll be back online in an hour or two! [22:53:12] nice [22:53:14] have fun! [22:53:14] Sounds fun [23:29:09] Fundraising Tech Backlog, Patch-For-Review: fix silverpop database updater not to crush the master database with long queries - https://phabricator.wikimedia.org/T157600#3010439 (cwdent) It sounds like the least disruptive fix for this might be to move the job back to the staging server, but @Ejegg menti... [23:34:22] (PS47) Eileen: [WIP] - this version is just for debugging test hatred [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [23:35:13] Fundraising Tech Backlog, Patch-For-Review: fix silverpop database updater not to crush the master database with long queries - https://phabricator.wikimedia.org/T157600#3015346 (Eileenmcnaughton) No dedupe doesn't touch the silverpop db. The dedupe job fails when silverpop runs (but is scheduled not to... [23:36:27] (CR) jerkins-bot: [V: -1] [WIP] - this version is just for debugging test hatred [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:38:02] (PS48) Eileen: Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) [23:39:04] (PS1) Eileen: [WIP] - this version is just for debugging test hatred [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/336940 (https://phabricator.wikimedia.org/T115044) [23:41:42] (CR) jerkins-bot: [V: -1] [WIP] - this version is just for debugging test hatred [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/336940 (https://phabricator.wikimedia.org/T115044) (owner: Eileen) [23:42:10] (CR) jerkins-bot: [V: -1] Matching gifts import. [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/334235 (https://phabricator.wikimedia.org/T115044) (owner: Eileen)