[13:51:33] o/ schana [13:51:59] Was thinking about setting up a testing env for ORES where we could safely knock down the redis host and see what happens. [13:52:58] It seems like setting it up in the `revscoring` project could make sense. We're well beneath our CPU quota there. [13:53:31] We've never seen the issues I'm worried about on our staging server -- which hosts all services within one VM. [13:54:00] So I'm thinking that we're going to want to split the redis service to it's own VM, but we could have celeryd & uwsgi on the same host. [13:54:05] Thoughts? [14:03:10] o/ sabya [14:03:18] o/ halfak [14:03:33] How's experimenting with rabbitmq et al. going? [14:04:03] Been working on trying to address our short-term redis concerns with schana while you look at the longer-term. :) [14:04:35] hi halfak [14:04:50] rabbitmq not started yet. plant to start after I get the precached daemon started. [14:05:28] precached daemon coded. but finding out ways to test it. [14:05:44] halfak:^ [14:06:14] Oh yeah! I forgot you were looking at the precached daemon [14:06:16] My bad! [14:06:26] Re testing, you could aim it at our staging server. [14:06:33] ores-staging.wmflabs.org [14:07:24] The staging server will likely not be able to keep up with the rate of edits coming through for long, but it should serve as an ORES endpoint you could point to. [14:07:24] ok. going through puppet basics :) [14:07:42] Gotcha :) [14:08:04] * halfak will likely learn a lot about puppet from reviewing your changes. [14:08:06] how many edits/sec do we get? is there a place I can see these stats? [14:08:29] Plain edits/sec is hard, but we can look at the activities of ORES on graphite.wmflabs.org [14:08:53] I'm not sure how to deep link, but you can find "ores" on the left side under "metrics" [14:09:17] * halfak tries to deep link [14:10:10] don't bother. I found out already. [14:10:17] kk :) [14:10:28] Look for "precache_request" [14:10:39] ores-web-0[1-4] [14:10:52] are they servining production traffic? [14:11:02] *serviing [14:11:11] aah. *serving [14:13:04] ores-web-0[1-2] are serving production traffic, yes. [14:13:09] Do you see 03, 04? [14:13:24] no, my bad. [14:13:31] OK. :) [14:14:35] any idea if we have grafana (a nice frontend to graphite) configured? [14:14:44] sabya, BTW, I made this for schana yesterday: https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores#Cluster_layout [14:14:53] sabya, not sure, but I remember hearing about it. [14:16:13] what's is running in lb? haproxy or something else? [14:16:47] nginx [14:17:21] sabya, https://wikitech.wikimedia.org/wiki/Grafana.wikimedia.org [14:18:03] ok [14:19:35] * sabya going through the cluster setup graphics made by halfak [14:24:46] question: why is "store" not saving the results directly in the cache? [14:24:51] halfak:^ [14:26:26] if I understood correctly, we are doint like like this: store, notify available, then the web request gets from one redis, puts into another redis then write the response to HTTP client. [14:27:02] this could be reduced to: store in cache, notify available, web request now gets from cache, serves the response. [14:28:05] sabya, celery has it's own "results backend" for managing the results of a computation. [14:28:23] It is a finicky, non-configurable (at least trivially) store. [14:28:37] Where-as our cache is fully configured and predictable. [14:29:10] got it. [14:29:40] very nice pictorial representation, halfak [14:32:07] :D [14:32:11] I'm really happy they are helpful. [15:06:25] * apergos peeks in [15:06:32] did someone say "celery"? :-) [15:09:41] :D You have that for a ping word apergos? [15:11:15] no but I've though about it :-D [15:11:59] :) I was just talking to sabya about how we use redis as a backend results storage for celery [15:13:07] did someone say sabya [15:13:18] :) [15:16:11] :-) [15:16:25] do you find that you run into any troubles with that? [15:16:42] you're using that only for the storage piece right? [15:25:12] apergos, using redis for both broker (job queue) and backend (results store). [15:25:34] We're dealing with some instability issues. These might just be because we've misconfigured our redis connections and therefor they are flakey. [15:25:44] It could also be that having redis as a SPOF is just a bad idea. [15:25:50] apergos, ^ [15:26:15] I should note that our "instability issues" are 2 minutes of downtime once a month. [15:26:25] So not major. [15:27:51] good morning lzia :) [15:31:08] oh it's a SPOF hm [15:31:23] I have heard that using redis as a broker is not recommended, though I don't know why that is [15:31:38] Yeah. We're looking into tewmproxy and RabbitMQ as alternatives to the single redis node. [15:31:42] I will likely play with the storage end of it [15:31:51] apergos, we went with redis since ops has experience with redis. [15:31:54] sure [15:31:58] well redis is easy-ish [15:32:14] But it seems that RabbitMQ is best supported and will work in a nice, distributed way. [15:32:23] I guess it's a pub sub sort of model? [15:32:28] gah I read some of the docs and yet [15:32:38] no brain no pain, or at least no memory when I need it [15:33:24] and rabbit is written in erlang! [15:33:41] yeah that put me off it a bit :-D [15:33:59] but I'll probbaly go that route for proof of concept [15:34:03] If I'm not contributing code, then I'm generally a fan of things written in erlang :) [15:34:10] oh gee thanks [15:34:26] well the thing is that like just about everything else that ops touches [15:34:32] if we run it we'll wind up debugging it [15:34:39] Yup [15:34:39] and that means code brrrrr [15:36:55] So, I was trying to look up a nice term for a programming principle the other day. Maybe ya'll know it. Something along the lines of "Program so that the person who comes after you will assume the right thing without having to think." [15:37:07] kiss [15:37:11] :) [15:37:12] Something like "Program like your code will be maintained by an axe murderer." [15:37:44] I've heard the quote before, but don't remember where it came form [15:38:18] I want to be able to say, "Have deep thoughts about your code, but don't require that your maintainer does too." [15:38:57] well [15:39:09] so I've heard this thing that is sort of like the peter principle [15:39:11] and that is [15:39:18] we like to hire really smart coders [15:39:29] and have them write code that is to the best of their ability [15:39:31] sure [15:39:38] I mean, on the face of it, makes sense [15:39:52] but now consider that debugging code is always harder than writing that same code [15:40:06] so if you write code at the edge of your ability [15:40:15] it will be outside of your ability to maintain it well [15:40:25] But when I code to the best of my ability, it's easier to debug than otherwise. [15:40:42] Maybe some problems I can barely code for, let alone debug the code for. [15:40:47] * halfak thinks about that for a bit. [15:40:55] I guess you mean: I make it more readable [15:41:02] I put work into refactoring [15:41:02] etc [15:41:06] Yeah. [15:41:08] but I see code a lot of the time that is [15:41:14] 'let me use this cool trick' [15:41:19] I've heard that same thing in terms of writing 'clever' code [15:41:20] because I can, I'm smart and I figured it out [15:41:21] and [15:41:27] boom [15:41:41] so it's about avoiding cleverness in favor of readability/maintainability [15:42:08] dunno if that made sense, the writeup I saw explained it better [15:44:05] yeah here it is [15:44:10] I totally butchered the quote :-D [15:44:13] “ Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. ” - Brian Kernighan [15:46:28] here's the axe murderer one: [15:46:34] apergos, yeah. I guess if we thinking about it as "coding cleverly" vs. "engineering cleverly". A clever engineer is thinking about how she'll be lazy come time for maintenance. [15:46:37] “ Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. ” - Rick Osborne [15:46:56] a good programmer is a lazy programmer [20:12:15] schana, some thoughts I posted to IRC this morning: http://pastebin.com/5Stwuhwp [20:12:22] When you have a chance to look :) [20:12:53] halfak: I saw them, and have been thinking about it [20:13:44] kk [20:14:34] I need to get sleep before [20:14:42] I continue on scap3 [20:14:59] kk [20:15:07] Can you submit a WIP with what you have? [20:15:14] tomorrow I work more on this [20:15:25] I made a commit in scap branch [20:15:39] Great. Can pick up from that if we beat you to it. :) [20:15:44] Sleep good. :D [20:15:49] thanks [20:15:59] o/ [20:16:22] * halfak is just putting some finishing touches on `ores`. Will be bumping ores-wikimedia-config to use the new models next. [20:43:47] no dartar? [20:55:41] greg-g, he's not on IRC often [20:55:57] But I know he is *in* today [21:00:06] halfak: yeah, was just going to ask about https://phabricator.wikimedia.org/T103700#2124344 [21:00:44] greg-g, seems right to me. [21:00:46] ggellerman, ^ [21:01:38] greg-g, we're just about to start a staff meeting. I'll make sure DarTar sees this. [21:02:08] halfak: if you think it's right, I can change it real quick for ya [21:02:32] * greg-g is here to serve [21:02:43] Na. Will have DarTar do it. I assume he has the rights to do so. [21:03:15] * greg-g nods [21:03:27] * greg-g parts, to save his window list