[15:32:19] halfak: you used celery on a recent project right? [15:32:30] yup. [15:32:52] so I'm celery-curious as I think about dump rewrites... can you [15:32:59] o/ [15:33:16] well do you have a few minutes to explain what problem you were trying to solve by using it, and how well it met/failed your expectations? [15:33:23] yeah I looked at the docs but [15:33:35] tl;dr syndrome etc [15:34:01] hello/goodbye guillom (not sure which that is :-D) [15:34:16] It's hello :D [15:34:35] all righty then! hello [15:34:39] apergos, just started a meeting. Can I ping you in an hour? [15:34:42] oh sure [15:34:48] I"ll be around, just holler. [16:53:26] o/ apergos. I ended up going meeting to meeting. [16:53:33] But I've got a few minutes to talk celery. [16:53:39] I figured it was something like that [16:53:51] Do much programming in python? [16:53:54] yes [16:54:00] pretty much all of it any more [16:54:28] OK. So celery is fancy distributed `multiprocessing` [16:54:34] uh huh [16:54:44] You start up "workers" that operate a lot like a `multiprocessing.Pool` [16:55:03] but across hosts [16:55:08] You'll likely do most of your work through an `apply_async` call that returns a future. [16:55:15] Yeah. horizontally scalable. [16:55:39] Essentially, there's an task queue and a key-value store for output values that serves the futures. [16:55:58] and that store can be redis or rabbitmq or other unstable stuff I guess? [16:56:09] That's right. [16:56:13] Celery also offers some complex patterns for setting queue priority and killing tasks that run too long. [16:56:29] Those come in handy for setting up for robustness. [16:56:48] how about resource management, how does a worker know how many jobs to run on a host? [16:56:56] or are there multiple workers per host too? [16:57:09] Yes. You can start as many worker processes as you like. [16:57:27] We're a mixture of IO/CPU bound for ORES, so we start up 4 workers per CPU core. [16:57:30] so you put a cap on the number of workers in order to manage the load (for example0 [16:57:32] right [16:57:53] I'm really happy with the documentation and support from the celery devs. [16:58:07] Except for a few crazy requests, they've been very quick to help me solve problems. [16:58:17] what sort of issues did you run into? [16:58:21] well let me backtrack [16:58:29] what did you want to get done with it, first? [16:59:05] So, we have a nice stateless problem for ORES: fetch data, extract features, give features to prediction model and report the score. [16:59:22] ok [16:59:26] We encapsulated this function into a "task" and submit it to the celery cluster. [16:59:28] (nice blog post btw) [16:59:31] :D [16:59:31] ok [16:59:33] Thanks [16:59:44] So, this allows us to service many parallel requests at a time. [16:59:53] Or to offer a small set of requesters high throughput. [17:00:05] right [17:00:11] Right now, we have 16 cores across 4 VMs for the workers. [17:00:33] Celery is also very good for long-running jobs. [17:00:38] how so? [17:00:45] You can store a job ID and come back to ask for it later. [17:00:49] ah [17:00:51] Celery will report the status. [17:01:01] Quarry also uses celery to run queries. [17:01:03] your long running and my long running are likely quite different :-D [17:01:23] Well... could be hours, but that generalizes to days/weeks just fine. [17:01:28] uh huh [17:01:31] * halfak runs to next meeting. [17:01:34] ok [17:01:38] I'll read scrollback and respond when I get back :) [17:01:44] thanks! [17:01:52] no problem [17:01:59] happy ... meeting :-P [17:37:17] halfak: so the story for me is that reliability is crucial, and if a job doesn't run by a worker I have to know about it and be able to resubmit [17:38:54] I have to be able to say 'don't submit this til that's run' (maybe) [17:39:10] I have absolutely to be able to get the return code and any exceptions [17:39:30] and I can't have two of the same exact job running at once [17:39:35] it's not a web service sort of thing [17:39:45] so wondering what sort of issues you ran into with it [17:47:04] apergos, so, there's a good way to check if a job is running. [17:47:21] Regretfully, there's a race condition possible if you've just submitted the job. [17:47:30] grrrrrrrrr [17:47:38] I've done some work to bring this down to ~100ms [17:47:49] in celery itself you mean? [17:47:59] Yeah. It tracks job IDs. [17:48:06] hm [17:48:17] So if you can start a job and store the ID (or if the ID is deterministic) you can check later before starting up a new job. [17:48:53] got a link? (might as well start looking at the code from somwhere :-P) [17:49:48] so hrm I would expect the worker to claim the job first. then start it. then report that it's been started [17:50:08] http://stackoverflow.com/questions/5544611/retrieve-a-task-result-object-given-a-task-id-in-celery [17:50:13] and if the worker manager (whatever that component is called) doesn't see that within a bit of time... [17:50:28] also! [17:50:36] can the manager request the jobs be shot? [17:50:47] apergos, you can either poll when a worker starts or attach an event. [17:50:57] ok [17:50:58] Killed? I think so. [17:51:02] yeah killed [17:51:07] You can also set a time limit when starting the task. [17:51:07] ok that's a plus for sure [17:51:20] cancel [17:51:24] http://stackoverflow.com/questions/8920643/cancel-an-already-executing-task-with-celery [17:52:04] using the terminate option is "a last resort for administrators" because you may accidentally terminate another task which started executing in the meantime [17:52:06] uhhh [17:52:23] well we'll see [17:53:26] what does the manager do about workers that are unresponsive (does it even know that?) [17:55:33] apergos, depends on what you mean by "unresponsive" [17:55:53] well it depends on the model [17:56:13] whether the manager assigns jobs to specific workers or waits for them to get done as they are magically grabbed from the queue [17:56:16] if the later [17:56:18] *latter [17:56:33] then it won't know if a host is overly loaded or has gone on the blink [17:56:57] or if a worker is out to lunch, I guess? [17:57:37] apergos, the broker should repair if a worker is not accepting tasks. I have had worker nodes go down and the system doesn't blink. [17:57:43] hm ok interesting [17:57:54] ah so what issues did you run into anyways? [17:57:54] I don't know if it will resubmit the task though. [17:58:13] right, well we can have the submitter check that it wasn't done and resubmit [17:58:13] Some serialization issues. It's important that your exception are Pickleable [17:58:16] so that's not an issue [17:58:19] oh huh [17:58:23] probably not at this point :-D [17:58:32] so that would be true for every...uh [17:58:41] Another thing I'm frustrated about is that there's no good way to put a limit on the queue size for celery. [17:58:49] can jobs be arbitrary scripts (bash wrapper around who knows what, C program, etc)? [17:58:56] I'd like to have the task queue implement backpressure. [17:58:56] no limit? [17:58:59] huh [17:59:19] Yeah. You can set all sorts of limits, but not queue size. [17:59:37] So re. running other scripts, I suppose you could use the subprocess module. [17:59:44] groan [17:59:56] really, I have to have a python wrapper around everything? [18:00:06] that is going to be really a drag [18:00:07] Well, it will be executing a python function. [18:00:12] ic [18:00:28] popen, how I hate you [18:00:30] ok well [18:00:39] Really is just a beefy multiprocessing.Pool ;) [18:00:49] I got it I got it :-P [18:00:57] :D [18:01:26] ok, well this has been very helpful believe it or not [18:02:03] if we can find any alternatives we;ll have to do the whole eval and list features/issues on a grid [18:02:32] Sounds reasonable. :) [18:02:43] but I have at least a general picture for celery now [18:03:09] apergos, I recommend you ping yuvipanda to get his sense as well since he's been dealing with the long-running use-case for longer than I have. [18:03:15] ok sure [18:03:20] Might have some stuff to add :) [18:03:24] great! [18:03:41] Oh! And milimetric [18:03:44] oh? [18:03:46] For WikiMetrics [18:03:48] :) [18:04:02] We have a bit of in-house expertise I guess. [18:04:12] cool [18:04:28] I"ll check that out [19:34:33] apergos: halfak soooooo I think celery is totally the wrong thing for dumps generation [19:34:43] because it's a queuing system, and not a batch job execution system [19:34:55] the latter is just 'make sure X of these run to completion with this data' [19:35:00] which celery does not give you [19:35:10] yuvipanda: it's not always X of these [19:35:19] tbh it's going to be something much different in the next go round [19:35:25] apergos: BUT [19:35:29] apergos: https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/jobs.md is what you want [19:35:30] apergos: I think [19:35:48] > A job creates one or more pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the job tracks the successful completions. When a specified number of successful completions is reached, the job itself is complete. Deleting a Job will cleanup the pods it created. [19:36:02] ah well you are welcome to tel me what is good/bad about celery and what's good/bad about kybernetes job handling too [19:36:09] I think they're just different things [19:36:17] discuss :-P [19:36:23] celery is great when you have X *incoming* discrete units of work [19:36:24] if you have a few free minutes obviously [19:36:32] and have to distribute them across hosts [19:36:37] where the units of work are roughly the same [19:36:45] and when they fail, you have code written to handle them [19:36:55] hey halfak, before I forget again… seen this? https://meta.wikimedia.org/w/index.php?title=Meta:Requests_for_comment/Enable_flow_in_the_Research_talk_(203)_namespace&diff=14841130&oldid=14134081 [19:37:35] I'm not sure if we'll have "roughly the same" or not, some might finish much faster than others [19:37:39] but that's tbd [19:38:00] right [19:38:00] the idea is exactly that, to have all dumps of anything be done in little pieces that get reassembled later [19:38:07] we don't know how many little pieces [19:38:11] so the question is how is the re-assembly being done? [19:38:20] ah not by the job queue [19:38:23] that's out of the scope [19:38:23] do you have one thing at the end that waits for task completion? [19:38:35] and what do you want to do on failure? [19:38:44] there will be a monitor that reports when the pieces for X dump step are all available [19:39:06] when jobs fail we will retry a couple of times, maybe one of them with some delay like an hour [19:39:13] all of this is up for discussion of course [19:39:19] and then alrt/email/whatever we decide [19:39:46] we have to cover cases like "the network's gone" or "the server's gone" or "mw deploy broke the dumps and no one noticed" or etc [19:40:00] as well as "this db was depooled but you'll get a new one on a retry right away" [19:40:19] the retries I expect to be out of the job queue manager scope too [19:40:52] apergos: right, so I think kubernetes has much better engineering for dealing with failure on long running jobs than celery does. celery's use case is mostly 'distributed multiprocessing', which implies you deal with a lot of the recoery / error handling type stuff [19:41:18] while kubernetes is a job scheduler, which is at a much higher level and you can just express what you wanna do about failures via manifests [19:41:27] give me an example if you like? or point me at something [19:43:03] apergos: so you can say: 'run this command to completion X times' [19:43:13] 'with at most Y parallel tasks' [19:43:21] and then one of them fails, it'll just start a new one. [19:43:47] so you'd write a small script [19:44:01] that basically splits the work input range into X bits [19:44:22] and writes out the YAML manifests with that info and hands it to kubernetes... [19:46:09] not sure if that made full sense [19:46:30] it did [19:46:40] but we won't know how many bits [19:46:49] something will feed jobs into whatever the manager turns out to be [19:47:05] but it will also be checking outputs and feed new jobs in, depending [19:47:07] oh. in that case you can just say 'Run Y instances of this' [19:47:16] oh I see [19:47:26] so if you want to take that much control celery might be a better option [19:47:27] well it will always be "run one instance of this" now run one instance of that" [19:47:33] but a lot of instances [19:47:39] right. [19:47:51] so at that level I think celery might be a good fit conceptually [19:47:55] hm [19:47:58] where it breaks down is per-node scheduling I guess [19:48:11] which I hope to bypass completely [19:48:16] if the 'this' and 'that' are different resource constrained - one is taking up 3cores and 1 is 5 cores... [19:48:19] all these jobs should be indifferent to where they run [19:48:22] you are going to end up with lopsided distribution [19:48:28] well I want to avoid that if possible [19:48:31] the current model doesn't [19:48:42] the redo should have every job want one core period [19:48:43] so if they are all the same resources (or similar enough) celery will work [19:48:45] right [19:48:48] with a max runtime of X (1 hour say) [19:48:59] * yuvipanda nods [19:49:07] and if it would keep going the job will write out a file with start/endpoints (we do this now0 [19:49:33] and the feeder will see that the time estimate was bad and a) reschedule the rest of that piece, [19:49:49] b) redo the splitting up of any related jobs that haven't been claimed [19:49:51] something like that [19:50:03] so with celery you can do that easily without files I guess. just have the celery task add more celery tasks :) [19:50:09] with appropriate info [19:50:20] the task adds tasks [19:50:21] right [19:50:47] so the file with start and endpoints is actually the dump output file I shoudl clarify [19:50:53] right [19:51:12] and we know we asked for X to Y but got back only X to X_n say [19:51:15] right [19:51:21] so it can just recurse [19:51:24] yep [19:51:26] and file a task for X_n to Y [19:51:36] well it will make a better guess first but yes that's the idea [19:51:56] I want ot never have to do anything manually with dumps again [19:52:10] just dropthings into the mill and have the sausages come out the other end [19:52:43] hmm well I will add kybernetes [19:52:48] (I always go to spell that wrong) [19:53:01] to the list of things to look at, even if the fit may not be right [19:53:27] no guarantee that my notions for the redo willl wind up being in the final architecture [19:55:18] apergos: right. I don't think kubernetes is the right level with this kind of architecture, but there are probably other architectures where kubernetes could be the right fit [19:56:53] ok. thanks for the heads up! [19:57:02] I should really take the time to play with it soon [19:57:22] apergos: have you seen tools.wmflabs.org/paws/hub/oauth_login [19:57:28] apergos: it's our first kubernetes native application [19:57:37] no, though I know you have said you had the first native app [19:57:40] login, click 'my server', and then 'new -> terminal' [19:57:52] and you got a bona-fide terminal on your browser [19:58:03] contained in your own docker container but with a persistant $HOME [19:58:03] no way [19:58:13] apergos: you can even run emacs in it :) [19:58:16] home is.. where? [19:58:16] (or vim or nano) [19:58:21] home is $HOME [19:58:30] (internally it's implemented on NFS) [19:58:39] but as a user you don't feel the difference [19:58:41] ah nfs the bane of our existence [19:58:41] apergos: check it out :D [19:58:47] I shall! [19:58:59] apergos: yes but it's abstracted away. once we have ceph / cinder I can switch to that and give each user their own cinder block [19:59:06] oh yay [19:59:12] what do you think you'll move to, any ideas? [19:59:22] because I need to shop around for somethign usable as an object store [19:59:29] ...to replace nfs :-P [19:59:29] apergos: I don't know yet. [19:59:44] I had ceph in the back of my mind but really no idea if it would be suitable [20:00:31] right [20:00:33] lots of options [20:00:35] yuvipanda: i'll be 10-15 min late. Sorry! [20:02:30] I"m just waiting for the server startup now [20:02:35] J-Mo: np [20:02:54] apergos: try refreshing. I added a new node so it doesn't have the container in cache yet so you probably hit that node. [20:03:27] the refresh button didn't seem to do anything [20:04:23] apergos: your container has started! try refreshing? [20:04:47] the refresh button stil does nothing [20:05:34] I reloaded the page and eventually it came back with something else [20:06:46] labstore.svc.eqiad.wmnet:/project/tools/project/paws/userhomes/104 on /home/paws type nfs4 [20:06:49] ah there it is indeed [20:09:57] apergos: yeah :) [20:09:59] can't run emacs on it, it's not installed and I don't have root perms :-P [20:10:02] apergos: oh? [20:10:05] apergos: I INSTALLED EMACS! [20:10:10] damn I didn't push it I think [20:10:11] try vim :) [20:10:14] or nano [20:10:24] tools.paws@jupyter-arielglenn-104:~$ which emacs tools.paws@jupyter-arielglenn-104:~$ emacs [20:10:34] yeah I didn't push it I think [20:10:38] sorry bad formatting [20:13:04] so where's the code for this fine service? [20:13:09] :-) [20:13:37] apergos: https://github.com/yuvipanda/jupyterhub-kubernetes-spawner [20:14:12] yuvipanda: https://en.wikipedia.org/wiki/Wikipedia:Teahouse/Questions/Archive_412 [20:15:51] one init py file? that's the whole thing? nice!! [20:29:11] hm that's twice now I've had one of the terminals hang on me [20:29:18] I went off to do something else and when I came back, no joy [20:29:42] but it's pretty cool anyways [20:33:04] apergos: yeah that's http I think. refreshing works [20:33:11] apergos: this is a 5d project so far :D [20:33:21] 5d? [20:33:55] 5 days [20:38:58] very nice! [20:39:03] what's up next? :-D [20:39:45] Unicorn-riding pandas! [20:41:12] I initially misread that as.... [20:41:15] http://www.chinadaily.com.cn/photo/images/attachement/jpg/site1/20110531/0013729e48090f4e8f900e.jpg [20:42:13] hahahahahaha [20:42:15] wow [20:42:17] that's great [20:42:29] apergos: what's next is to get emacs on there :D and also to provide access to dumps and install halfak's mw utilities in it [20:42:42] apergos: and then see how stable it is when more than 10 people use it :D [20:42:59] image ten people all grepping through xmldumps :-D [20:43:15] apergos: I'm hoping to work with the pwb community to write docs about using this instead of setting up pwb locally on their own computers [20:43:27] oohhhh [20:43:50] apergos: do you know of quarry.wmflabs.org? :D [20:43:52] I admit for years, really years, I ran an irc bot for the el wikt channel off of my laptop [20:44:00] then I ran one out of toolserver [20:44:10] one of several hundred other copies :-D [20:44:25] ofc [20:44:27] :) [20:44:30] I know the name but I don't remember what it's for [20:44:57] apergos: it's like paws/jupyterhub except just for SQL [20:45:08] usign the lab dbs? [20:45:11] so instead of having ot get a tools account, learn gridengine commands and write 'sql' commands from the commandlines [20:45:19] you just write your query and press a button [20:45:21] apergos: yes [20:45:29] and get your one-off answer instantly [20:45:31] nice [20:45:49] is it running now/ [20:45:50] ? [20:46:10] apergos: yes [20:46:26] apergos: we have a few hundred active users, with about 20-30 new queries per day :) [20:46:43] apergos: you can also share the link with others to show them your answers [20:47:21] so let's assume you don't know the db schema and you want to wander around, does it support that, or should you show up with a query already written? [20:47:39] apergos: you can do 'DESCRIBE ' and explore around that way [20:47:42] (I've got the window open now, no wait like the notebook service) [20:47:50] yeah it's instant [20:47:52] it uses celery to [20:47:55] ah so it will let me run all queries not just a series of selects [20:47:55] *too [20:47:56] good [20:48:06] apergos: yeah anything. there is a time limit of 30min [20:48:32] I imagine the code is in one of your github repos? [20:48:40] (don't need to link me, I can poke around later) [20:48:49] apergos: no on gerrit :) [20:48:55] apergos: it's analytics/quarry/web [20:49:17] ok! [20:49:31] oh so this is en wp is it? can you use dabatabase to switch to something else? [20:49:44] dabadabadoo database [20:49:52] typo fingers [20:50:25] apergos: yes [20:50:25] for that matter can one submit a few lines of sql or only one? [20:50:30] apergos: 'use trwiki_p' [20:50:34] apergos: any number of lines [20:50:44] apergos: you can run multiple select statements and you'll get back multiple resultsets [20:50:48] so a full tool [20:50:50] excellent [20:50:52] basically yeah [20:51:03] I wrote it so nobody would have to use the 'sql' commandline tool on toollabs :D [20:51:21] so you wouldn't have to give help to anyone using 'sql' on the commandline in toollabs [20:51:25] which is a fine goal [20:52:25] vagrant and not a container? [20:54:09] sqlite? huh [20:54:35] apergos: that's all the test setup :) [20:54:41] ah [20:54:49] apergos: this is puppetized. there's a quarry module :) [20:54:56] it's not my best puppet since I wrote it so long ago [20:55:03] yeah all old code is bad [20:55:12] yeah [20:55:18] which means code we write now will be bad in a few years too [20:55:20] shrug [20:55:29] yeah what this means is that code needs care and love [20:56:18] right [20:57:06] oh there's the source code link on the bottom, durrr [20:57:27] apergos: like, https://etherpad.wikimedia.org/p/gridengine-uses is my list of things that people use tools for now (sql is no longer there :D) and I've to find individual specific solutions for all of them [20:57:59] you want to do away with tools commandline you mean? or? [20:58:35] or you want to do away with gridengine? :-P :-D [20:59:12] yes [20:59:15] kill gridengine [20:59:18] and make ux better [20:59:21] basically :) [20:59:53] that would make some things easier on the users for sure [21:00:05] and more importantly expand the number of people who can do it [21:00:12] far more people can use quarry than can use 'sql' [21:00:26] I ran into a case this weekend where the grid reported a job as running (qstat) but in fact on the host it was not running (checked via ps) [21:00:37] stuff like that could drive a user crazy, they would never know [21:00:52] halfak: J-Mo I've conned bd808 to helping setup elasticsearch [21:00:57] apergos: yeah happens lal the time [21:01:11] yuvipanda, \o/ [21:01:21] A true professional finds other people to do his to-do items :) [21:01:33] yes!!!! [21:01:34] halfak: I suspect we'll have our infrastructure this week or next (probably next) [21:01:42] why is sql (cli) so much harder for folks? it must just be the setup [21:01:45] this is just the backend stuff [21:01:46] apergos: yes [21:01:49] because honestly typing at the terminal is no different [21:02:05] I'll agree that setup is a legit barrier to entry [21:02:55] basically anything you can do via the web browser is going to be easier [21:05:18] ^ [21:05:20] yeah [21:06:39] hm 11 pm and I have promised myself to go to bed "early" tonight (midnight) [21:06:48] so I'm going to wander off to do winding down things [21:07:10]