[08:08:43] (PS4) AndyRussG: WIP API for campaign allocation choices [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/170843 [08:09:57] (CR) AndyRussG: "This PS: Added some missing properties." [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/170843 (owner: AndyRussG) [15:51:25] (CR) Ejegg: [C: 2] support cross-database references [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/171979 (owner: Awight) [15:52:57] (CR) Ejegg: [C: 2] Make compatible with raw SQL instrumentation [wikimedia/fundraising/crm/civicrm] - https://gerrit.wikimedia.org/r/171980 (owner: Awight) [16:05:51] * awight grumbles a bit about national holidays inconsistently x [16:05:56] being applied to Monday [16:06:07] ... and hijacking of both May 1 and Armistice day [16:06:24] ... and add International Women's day to that list [16:12:21] awight: boo! :) [16:12:29] morning! [16:12:48] * AndyRussG 's scary [16:13:16] :) [16:13:38] nothing can scare me for the next few minutes... I just emptied a day of diapers and stuff [16:13:53] uhhhhhh hmmm [16:13:54] * awight rocks slowly, clutching self [16:14:21] Yeah it does get better, and different :) [16:14:33] oh hey, speaking of scary futuristic things though [16:14:41] ? [16:14:52] I realized we don't have a roadmap for CentralNotice [16:14:52] Considering you're numbed to scariness, if u'd like to take a peek at progress on the bucket bootlegging... [16:14:57] hehe exactly [16:15:00] Ah that is indeed tru [16:15:15] yeah cos we have features to implement on several time scales now... [16:15:26] We need a theory of relativity [16:15:28] https://gerrit.wikimedia.org/r/#/c/170843/ [16:15:31] aaaah [16:15:38] ^ has new patchset [16:15:40] lemme look at that after commuting [16:15:50] if you want [16:15:56] K thanks! :) That'd be fantastic [16:16:06] The following change is almost out of the factory [16:16:53] It's a darn complicated change we're pulling out of our collective... sleeves... [16:16:55] it looks really slick, but I'm confused--I thought u were doing a ResourceLoader module? [16:17:08] Yes that's the next change in the pipeline [16:17:29] The API is because the info is on Meta, not on whatever wiki is serving the user [16:17:48] So the RL module makes an in-cluster HTTP call to Meta [16:17:52] whoa. [16:17:58] It's in the doc! [16:18:04] Here's what I'm modelling it on: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/includes/RemoteSchema.php [16:18:33] Don't worry, it's very cacheable [16:18:33] I'm pretty sure bits.wmo can read from the metawiki database. but I hadn't thought about it, that would be tricky you're right [16:18:48] do api responses get cached? [16:18:56] Oh! Hmm that's another thought [16:19:00] ... and another one [16:19:32] I hope that the local call can bypass the cache... [16:19:51] Lemme ask about it on ops [16:20:10] * AndyRussG wonders how long a person can survive on smoothies + coffee [16:20:25] ok this looks great, lemme sniff out some of ^^ and catch up w u after that [16:20:52] K thanks much [18:13:46] AndyRussG: I'm gonna get some input re: cross-WMF database requests, in #wikimedia-dev [18:15:26] Ah awight I did in ops [18:15:33] oh sorry [18:15:39] * awight reads logs [18:16:02] ah np thanks also [18:16:36] one sec I'll send a next WIP too [18:16:36] (tl;dr: 120s cached in Varnish) [18:17:39] Well, I wasn't as concerned with the cache interval, as with the potential loss of 150ms to make an API request, when the data is actually sitting on a db server we're already connecting to. [18:18:47] Ah! right I didn't ask much about that 8p [18:18:47] I did poke at it [18:19:00] anomie just replied, "don't do what TimedMediaHandler does" [18:19:03] awight: ^ [18:19:09] didn't dig into it much more... [18:19:16] hehe I just got to that [18:19:29] Maybe marktraceur has been down that rabbithole? [18:19:43] * marktraceur peeks [18:20:23] awight: TMH is the thing that loads a video player and audio player for "thumbnailed" media on article pages. [18:20:36] marktraceur: we're wondering about making API calls between bits and meta, vs accessing the remote db directly [18:21:33] marktraceur: if we were to go the API route, here is what I'm copying: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/includes/RemoteSchema.php [18:21:52] awight: as in that case, I'm using memcached [18:24:45] awight: yeah sorry I didn't mean to deter you from asking more questions anywhere :) [18:27:10] AndyRussG: the API patch is definitely something we should have, we might fall back to it, but if there's a way to shave the API round-trip... [18:27:24] K makes sense... [18:27:54] Since it's all on-cluster, I was thinking it'd be negligibleish [18:28:05] which is definitely a word [18:28:13] that's a good point :) but there's still the PHP spin-up [18:28:32] troo [18:28:41] / How long to cache the responses to calls to the banner choices API [18:28:41] $wgBannerChoicesCacheExpiry = 360; [18:28:47] I think the non-response on -dev has to do with there not being an agreement [18:29:07] wikimetrics for example is liberal about reading from every db [18:29:11] lemme look at TimedMediaHandler... [18:29:25] u might have to use the pinger [18:29:29] hehe [18:29:33] * awight flushes IRC [18:30:41] Last night I was pinging wildly on -dev (check logs from quarter-past midnight or so) [18:32:15] lol [18:32:23] * awight imagines the scene [18:34:06] AndyRussG: Sorry, what does this have to do with TMH? [18:34:06] (PS1) AndyRussG: WIP Get banner choices via server, choose on client [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/172299 [18:35:11] marktraceur: it was suggested that TMH needs something similar w/ regard to... uh... I think pulling data between WMF wikis, tho that it doesn't take the optimal route (?) [18:35:23] Uhh...I have no idea [18:35:42] Hmmmmmm oh well... [18:35:52] AndyRussG: Media functions use the fancy-dancy shared DB media repository filerepo class on our cluster [18:36:29] Hmm that must've been something like it... [18:36:38] Know the classname offhand? [18:36:59] awight: ^ very drafty, not complete [18:37:20] AndyRussG: looking at TMH, it subclasses ApiQueryAllPages, in handlers/TextHandler/TextHandler.php [18:37:37] seems like a nice way of cross-db querying, now I want to know what the problem is. [18:38:17] I guess ask anomie... ;) [18:38:20] yah [18:41:37] Jeff_Green: there was something terrible we were going to look at... making the silverpop db available from production. I can move forward with just a "yes it's a good idea", don't need the work done right now. [18:41:44] I think you already said this makes sense? [18:42:47] * awight blinks at pizzza spawn [18:44:06] awight: yeah, imo that makes sense generally sortakinda [18:44:18] remind me how that db is populated and used? [18:46:28] (PS6) Ssmith: update gauge library and presentation [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/171486 [18:46:30] Jeff_Green: hehe. So, it munges the CRM database and produces nicely digested summaries, which we export to Silverpop to help with spamming people. [18:46:59] These tables contain stuff like, did the contact donate this year? last year? [18:47:11] Very expensive to calculate. [18:47:57] AndyRussG: anomie is holding forth a bit in #wikimedia-operations [18:48:16] awight: a thanks [18:50:07] (CR) Ejegg: [C: 2] update gauge library and presentation [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/171486 (owner: Ssmith) [18:50:37] awight: the code is in gerrit and it's running via jenkins or cron? [18:50:50] awight: hmmm... I didn't get what anomie said about "A straight DB query is fine, IMO, as long as you're not caring to support setups that don't involve the local wiki being able to directly query the central wiki's DB"--by local he means, non-WMF? [18:50:54] Jeff_Green: via jenkins [18:51:11] But it looks like a satisfactory answer--i.e., both methods are doable, no? [18:51:33] AndyRussG: yeah I think the idea there is that 3rd parties might be using CN in a multi-wiki setup, but the db might not be directly accessible in tha tcase. [18:51:55] is the expensive part happening in the db or is it loady on the script host? [18:52:18] awight: ah yes ur right [18:52:26] Jeff_Green: very much db [18:52:37] Jeff_Green: which is where this gets tricky... [18:52:52] Jeff_Green: if you care to see, https://github.com/wikimedia/wikimedia-fundraising-tools/tree/master/silverpop_export [18:53:07] awight: I' [18:53:11] err [18:53:11] update_table.sql is the meat, it takes about an hour, all db server effort [18:53:23] i've got an ops meeting in a minute [18:53:37] Jeff_Green: kbye! [18:53:46] I have plenty else to keep busy [18:54:48] what I'm wondering though, is if we could move it off of lutetium but still send brutal read queries there. [18:55:11] or is it nasty on the writes too? [18:55:38] writes are strictly into the silverpop db [18:55:56] we can't really lock up the master though, even if it's just that db [18:56:11] I don't see how we would read and write across servers anyway, though [18:56:59] awight: if we can't split up db stuff so it doesn't destroy the master, then I would probably look for other ways to teleport the data over besides replication [18:57:13] oof [18:57:33] replication db1025-->lutetium-->db1025 is about the worst thing I can think of :-) [18:57:38] right. [18:58:32] federation...maybe? I haven't worked with that yet, we could consult Sean [18:58:35] (PS2) Ssmith: make gauge range selection more clear [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/171982 [18:58:37] argh [18:58:43] oop.mtg time [19:14:44] (PS2) AndyRussG: WIP Get banner choices via server, choose on client [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/172299 [19:15:25] ^ awight ejegg|away sucessfully sends banner choices into JS via RL!!! [19:15:37] and eats the peel [19:15:40] woot! [19:16:33] atgo: also ^ :) now getting to e-mails... 8p [19:16:47] thanks AndyRussG :D [19:16:54] likewise! [19:24:12] (PS3) AndyRussG: WIP Get banner choices via server, choose on client [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/172299 [19:25:11] K4-713: boo! [19:25:22] mmm.. and good morning :) [19:25:45] https://gerrit.wikimedia.org/r/#/c/172299/ ... almost there on Step 2... [19:26:45] awight: what is your verdict on API vs. DB call, especially for now? [19:27:21] AndyRussG: My instinct is that it should depend on how long an API call takes. [19:27:28] 150ms would be... way too slow [19:27:46] hmmm [19:27:50] If it's fast enough, I think the API is cleaner [19:27:56] why that limit? [19:28:16] not a limit, but the number I remember from... before the api startup code was optimized [19:28:30] this has to be served before the page is rendered [19:28:37] yes [19:28:41] we really can't slow down page load... [19:29:53] The memcache bit seems to work, so it'd only actually call the HTTP API once every whatever # of seconds we set (I put a default of 360s) per project/langauge/logged-in-status combination [19:31:08] lines 42-45 and 67: https://gerrit.wikimedia.org/r/#/c/172299/3/includes/CNBannerChoicesResourceLoaderModule.php [19:32:38] AndyRussG: I think all we have to do to make a direct DB query, is to parameterize AllocationChoicesProvider to pass it the $dbr [19:33:20] Ḱ... also fine uv course! [19:33:28] AndyRussG: but yeah the only criterion I have to evaluate which approach we need to take, is how many people's experience we will slow down, and by how much. [19:33:42] Unfortunately, I don't think there is a MediaWiki rubric for that [19:33:50] Yeah [19:34:05] Mmm since I've never done interwiki DB calls I guess I have no idea how easy/complex they are [19:34:07] Perhaps there's a person with responsibility for site performance who u can ping [19:34:28] I mean, if it's just as fast to do the DB version, that's certainly cool, I think... [19:34:40] AndyRussG: well I think it's as simple as, replace wfGetDB with another function call. [19:34:53] Ah hmm! K that ain't hard... [19:35:07] And maybe yeah just leave the API as fallback for 3rd parties that don't have such cross-db access then... [19:35:11] maybe ask Ori who to ping about performance? [19:35:47] I think you're right that it's a drop in the bucket, at most... like 40,000 people every 6 minutes who are affected [19:36:11] Remember a lot will also get the Varnish cache [19:36:29] How long will you cache the RL response? [19:36:33] probably the standard 5 min? [19:36:38] But if the DB is the better approach and it's easy, we might as well do it, I think.. :) [19:36:48] I think, lemme re-check [19:37:34] Pretty sure we can set that cache expiry independently for your new module [19:39:33] ^ a hmm, maybe not caching, it looks now like for the banner choices module ResourceLoader is putting in the "version" parameter, which is, as Roan specifically said, a "cache-buster". [19:40:02] errr [19:40:05] One sec I'll get the varnish config [19:40:19] I thought he meant, it busts the cache when we increment the number in the request? [19:40:29] which is not quite germane I think [19:40:54] I thought he meant, it's so it doesn't cache, but I could be completely wrong [19:41:07] Also I'm running w/ debug=true [19:41:24] that will change things :) but I think caching is the same [19:43:03] whole lot of crickets on -dev [19:43:50] hmmm [19:44:06] see "caching details" on page 7: https://docs.google.com/a/wikimedia.org/document/d/150lrUt8b4aiPK19XCcoE6Yy3TwN_wo529UGQjk0eTbo/edit [19:44:28] yeah now I remember, varnish just passes thru the cache headers set in PHP [19:46:38] awight: in core, see includes/resourceloader/ResourceLoader.php, sendResponseHeaders() [19:47:26] and in DefaultSettings.php:$wgResourceLoaderMaxage = array( [19:47:26] 'versioned' => array( [19:47:26] 'server' => 30 * 24 * 60 * 60, // 30 days [19:47:26] 'client' => 30 * 24 * 60 * 60, // 30 days [19:47:26] ), [19:47:28] 'unversioned' => array( [19:47:30] 'server' => 5 * 60, // 5 minutes [19:47:32] 'client' => 5 * 60, // 5 minutes [19:47:35] ), [19:47:37] ); [19:47:58] versioned doesn't really get cached 30 days since the version parameter is basically a timestamp [19:55:21] oh dear [19:55:32] (PS3) Ssmith: make gauge range selection more clear [wikimedia/fundraising/dash] - https://gerrit.wikimedia.org/r/171982 [19:56:39] AndyRussG: hmm, we could make the timestamp low-resolution though... that's interesting, cos it synchronizes the cache expiry across clients [19:57:05] Oh dear? [19:57:16] "oh dear" that people do that :) [19:57:32] Ah hmmmmm ;p [19:57:45] It seems a pretty internal resource-loader-y thing, not sure I'd really want to mess w/ it... [19:57:48] But like 5-min absolute time expiration windows would give us really nice properties for the ramp-up and ramp-down [19:57:54] ah ok [19:58:03] Hmm [19:58:18] Well w/ the DB query, should we remove the memcache cache then? [19:58:23] erm. [19:58:43] iono. probably not necessary? [20:00:06] awight atgo K4-713 lunchy lunch? [20:02:05] pizzzacat: they are in a closet for... 30 min, I think [20:02:16] yep. [20:02:31] ooh ok [20:02:54] wait or waft food in front of them when they escape? [20:03:19] I can't food. Have to go do another thing. [20:03:43] ooh [20:06:18] awight: BTW, have any easy-to-copy examples of the cross-wiki-DB query on hand, by chance? [20:06:50] no! [20:07:06] At least, I know of examples, and they're horrible [20:07:41] (CR) Ejegg: [C: -1] "No workee. Am I missing a CRM_BAO_SilverpopExport commit from someplace?" (1 comment) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/170268 (owner: Awight) [20:08:04] AndyRussG: check out docstring for wfGetDB [20:08:13] It looks like you can just supply the wiki id as the third parameter? [20:08:20] ejegg: try "drush en wmf_reports" [20:08:25] and, hi! [20:08:39] OK, let me give that a shot [20:09:02] yeah, it was already enabled. [20:09:03] awight: wow! OK that is not bad at all :) [20:09:15] AndyRussG: work/mediawiki-extensions/AbuseFilter/AbuseFilter.hooks.php [20:09:26] at "wgAbuseFilterCentralDB" [20:09:31] Can't find any other mentions of that classname though [20:09:38] ejegg: oof? ok lemme check [20:09:42] awight: cool thx :) [20:09:50] also, hi! [20:10:07] ejegg: hehe, yes git add fail [20:10:13] git add --fail [20:10:18] heh [20:10:46] 'stages all but the crucial chunk of the given files' [20:10:58] (PS9) Awight: Customized LYBUNT report [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/170268 [20:11:16] git checkout -f --scrub-all-work [20:11:23] git stash --lose [20:11:33] pizzzacat: you might need these ^^ [20:12:06] awight oh? [20:13:08] git rebase --lose-freaking HEAD^ [20:13:33] combined with --scatter-marbles [20:13:44] I think ejegg already pointed out that these are documented at http://git-man-page-generator.lokaltog.net/ [20:13:56] e.g. "git-pocket-subtree pockets any remote subtrees below some configured unstaged branches, and all merged logs are exported to DEVELOP_STAGE by git-thrash-origin." [20:14:18] awight: ejegg: BTW anyone wanting to start thinking about unit tests for the new banner choices... might be fun... [20:14:41] freaking awesome manpages [20:22:12] (CR) Awight: [C: -1] "Looks really great, only one blocker, for campaigns with geolocation=0" (13 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/170843 (owner: AndyRussG) [20:22:35] awight pizzzacat lunch before standup? [20:23:25] atgo: yes. [20:23:25] awight pizzzacat lunch before standup? [20:23:28] AGH [20:23:30] lingo. [20:23:31] hehehe [20:23:33] blargh. let's do it. [20:23:36] YES LUNCH [21:02:28] AndyRussG|brb: ejegg: standup in 30", fwiw [21:29:04] Jeff_Green: u have further thoughts about the silverpop db paradox? [21:29:18] hey AndyRussG - on your last email, who are you expecting to have the answer to your open question? [21:29:48] I thought it was a great idea, but it looks like Megan+Ellery should decide? [21:30:08] awight is that about my question or yours to jeff? [21:30:18] atgomez: about AndyRussG's question [21:30:25] cool :) [21:30:31] maybe the-wub will want to weigh in, also [21:30:39] it's a... technical production question ::p [21:30:52] awight, not really [21:31:05] Jeff_Green: yeah we're screwed, so far, aye? [21:31:34] it's ahhm, too late for my coding though, I've already assumed we can do this somehow :) [21:31:42] ? [21:31:59] I wrote a report which relies on this data being available from the production Civi server... [21:32:16] well [21:33:16] the more we complicate the database setup the more likely we are to hurt ourselves [21:33:31] so my suggestion is to first look for ways to make the code safe for production db's [21:33:40] yeah, sorry to learn that at your and Sean's expense [21:34:17] Jeff_Green: maybe we can look at the query log while this thing runs? [21:34:33] atgomez: maybe I understood wrong? I thought the idea was to try to get some e-mail conversation going... Or, I took the wrong approach maybe? [21:34:41] no yeah, that's all good [21:34:48] just want to highlight who needs to answer the question [21:34:50] awight gimme a minute to finish something, and let's look at the code & queries [21:34:54] Hmmm [21:35:00] Well all together, maybe :) [21:35:14] Jeff_Green: /me blinks real hard at the fact that this job is *already* running on production [21:35:34] it's running on barium but using lutetium as the db? [21:35:34] Jeff_Green: I think that just means the db connection is made to the QA box though [21:35:41] yeah, creo que si [21:35:43] yeah [21:35:45] AndyRussG just trying to throw the ball squarely into someone's court so that we can get this sorted out as quickly as possible :) [21:36:40] atgomez: It may have technical side but the non-technical question is what information is important for whom [21:36:40] ...and how that information will impact on the stuff owned by different groups/people [21:36:40] does that make sense? [21:36:42] Jeff_Green: if query logging is enabled on lutetium... this script ran from 0801-0922 this morning [21:36:44] So like, all the stakeholders could tell us what info that sticks around across campaigns they care about and want to use, and how [21:37:18] yeah sounds good :) [21:37:29] AndyRussG: we already need at least one cross-campaign variable, eh? The saw_fullscreen thing... [21:37:36] I think that's enuf to say there is in fact a use case [21:38:07] and from there we find technical options (of which there are a few I think) [21:38:07] atgomez: aaah I see [21:38:23] got it :) [21:38:35] fueling the fire :P [21:38:37] maybe Peter, or ewulczyn, I think Megan is off [21:39:11] awight: atgomez: the "feature" I'm guessing at is that it might make sense to have it per-campaign-category [21:39:46] AndyRussG: is there any other option, for the saw_fullscreen variable? [21:40:05] ure [21:40:06] sure [21:40:07] I think the only alternative is a random cookie, which is exactly what you are wisely cleaning up [21:40:37] * awight quickly retracts "wisely" and substitutes "ruthlessly" :p [21:41:11] mm basically that's the idea yeah [21:41:11] There's nothing stopping anyone from doing this ad-hoc [21:41:11] I mean, you could write a commodore 64 emulator in a banner [21:41:26] ino... but we're proliferating ridiculous cookies [21:41:41] argh, wr1 parser. [21:41:42] if we look at what people are looking for I think there are commonalities in use cases [21:41:59] ? [21:42:03] I understood your question as, do we have info that applies across an entire campaign category, and I think the answer is definite yes [21:42:12] K [21:42:32] yeah the cookie proliferation is a technical issue. We can store stuff in several different places [21:42:35] implementation is up to you, like you say, banners can do whatever, but we will at least provide a nice framework if desired [21:42:46] Also there are the community/ethical/legal issues around storing [21:42:51] yesss [21:43:10] unfortunately, noone is watching <_< [21:43:45] (CR) Ejegg: "Looking good. 2 strictmode static warnings, and potential extra credit, but I'd +2 this if you don't care about those." (3 comments) [wikimedia/fundraising/crm] - https://gerrit.wikimedia.org/r/170268 (owner: Awight) [21:44:13] yeah so if we factor our the general common bit in these needs we can make it available for even more related stuff, have tests, keep it up to date, etc. [21:44:15] awight: @_@ [21:44:31] heheh [21:44:32] oh crap I forgot about roboK4-713 [21:44:43] quick, throw the switch back to "evile"! [21:44:44] That's what happens when I don't say anything for a while. [21:44:47] that's what she wanted the wr1 parser for, egads [21:47:02] awight: (and ejegg? maybe K4-713?) when is a good time, via IRC or hangout, whatever is easiest, to make a quick game plan CN work today and tomorrow? that is assuming you have time to and it would make sense, np if not [21:47:09] I'm just gonna throw a bit of food in my belly and would fine about 10-15 minutes after that :) [21:51:19] (CR) Awight: "Just to confirm, this is a noop, and wouldn't actually change banner selection?" (10 comments) [extensions/CentralNotice] - https://gerrit.wikimedia.org/r/172299 (owner: AndyRussG) [21:51:36] AndyRussG: any time! [21:51:54] * awight blushes at taking 4 hours to CR your patches. urrrgh, out of practice. [21:52:25] question for the universe: why are some contacts in civi listed with the email address "nobody@wikimedia.org"? [21:53:03] CaitlinVirtue: that's the dummy address we jam in if the donor doesn't supply a contact email [21:53:20] CaitlinVirtue: it's not actually supposed to make it all the way to the database, but if it does... at least it's recognizable ;) [21:53:28] awight: looking at queries now [21:53:36] Jeff_Green: awesome. [21:53:44] subselects [21:53:45] I don't really know what to look for. something about read locks [21:53:50] hateses subselects [21:54:03] me too. I prefer intermediate tables. [21:54:09] always? (assuming it's an online donation) [21:54:29] It's in there a ton. [21:54:37] Jeff_Green: just cos subselects end up confusing me, and often I and other evile devs try to make them coordinated, using info from the outer query. [21:54:52] CaitlinVirtue: yes, that's definitely why that dummy address appears. [21:55:04] not a fraud thing, if that's what you were thinking. [21:55:21] CaitlinVirtue: are there any suspicious entries we should check up on? [21:55:22] and if you have a huge select that takes a long time, and it explodes, you don't know where it died [21:55:37] Jeff_Green: yep. and it probably isn't doing what you intended, anyway :) [21:56:13] I think I would start with getting feedback from Sean [21:56:25] Jeff_Green: hehe don't tell him I'm behind this :p [21:56:27] @awight no, nothing to check. we're just doing the fine-grained de-dup of the MG annual appeal list and it appears quite a lot [21:56:35] my kneejerk not-a-DBA suggestion is to break up the huge select and insert into separate queries [21:56:40] CaitlinVirtue: aah. yeah it will be harder to dedupe those [21:56:49] and use a temp table, and only do the insert if the select went well [21:56:50] just wanted to make sure I understood what it meant before we started the outreach [21:56:55] thx [21:57:12] Jeff_Green: well. hmm, it would be hard to get simpler than those insert...selects. [21:57:12] i don't *think* a long process creating a temp table interferes with replication [21:57:36] I think we might be able to tweak the read isolation level to "hella sloppy", and reduce locking. [21:58:50] awight: which query is/queries are painful? [21:59:09] Jeff_Green: iono! I'd have to look at the query log [21:59:20] how about putting some logging in the code instead? [21:59:40] so we can tell what the heck it's doing without spelunking and trying to line up events? [22:00:04] no, we need the info about what actually happens in the db. I think the big deal is not the overall execution time of each query, but it's the amount of time the CRM db tables are being locked [22:00:17] Pretty sure that's easy to grok from the query log [22:00:25] there isn't even basic event logging in this [22:01:09] Jeff_Green: afaik, this is the number we care about: Lock_time: 0.000081 [22:01:45] oof. no, I just got slapped by SO. http://stackoverflow.com/questions/19036176/how-to-interpret-slow-query-log-information-generated-by-mysql [22:01:51] that's not at all what I want to know. [22:01:55] lutetium:/var/log/mysql/ -- you've got access [22:02:07] reading slow log is a pain in the arse [22:02:21] yeah I just learned that I know less than nothing. [22:03:55] the queries are nicely commented though [22:04:12] that's python foo? spiffy [22:04:25] really?? [22:04:45] look at lutetium:/var/log/mysql/lutetium-slow.log [22:05:05] Jeff_Green: I think those comments *are* the slow log [22:05:31] the query comments match the "-- foo" comments above the queries in the script [22:05:44] but now that I read this thing about "lock time is the time spent waiting for a lock *before* the query" I don't know where to turn. [22:10:36] Jeff_Green: I have this other pretty important thing I'd like to pull your coat about: https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2138?referrer[q]=utm_source&referrer[q_type]=&referrer[query_id]=q_4e6e4b804b540132dff022000ae213f7&referrer[rank]=2&referrer[size]=23&referrer[ts]=11%2F10%2F14+22%3A10%3A30 [22:10:41] grrrr [22:10:44] https://wikimedia.mingle.thoughtworks.com/projects/online_fundraiser/cards/2138 [22:11:06] Jeff_Green: I want to add some triggers to contribution_tracking... [22:11:19] we'll know right away if it doesn't work :) [22:11:35] let's look at that on weds [22:11:40] Jeff_Green: sure [22:11:57] tomorrow's PTO right? [22:12:05] TO at least [22:12:06] yeah [22:12:09] ha [22:12:11] Armistice Day [22:12:26] yeah my 30hr/wk schedule is not nicely compatible with vacations, afaict [22:13:25] It isn't? :o I thought benefits started at 30 h [22:13:51] which db is this utm_source column thing about? [22:13:59] hehe. Well it's just that the vacations seem to fall through the holes [22:14:07] Jeff_Green: drupal.contribution_tracking [22:14:08] :/ [22:14:51]