[03:00:15] YuviPanda: I discovered https://atom.io/packages/hydrogen and https://atom.io/packages/jupyter-notebook yesterday; haven't tested them yet but they look pretty cool. [03:00:29] * YuviPanda nods [03:00:45] eventually we should be able to allow people to hit our jupyterhub cluster from things like that [03:00:47] not yet tho [03:01:15] * YuviPanda is deep in yak shaving territory [03:01:26] am writing a python script that builds a debian image from scratch... [03:01:35] That sounds... fun? [03:01:45] it currently is a bash script [03:01:49] that's very fucking long [03:01:56] and written by upstream [03:02:03] after spending a couple of hours trying to make it do what I want [03:02:09] I decided to give up and just rewrite it in python [03:02:15] timeboxed it to one hour only today, let's see how that goes [03:03:09] I spent my day on strategy stuff and meetings and annual planning. That is unusual for me and I hope I can go back to doing Real Work soon. [03:03:29] +1 [03:04:09] * guillom is also considering a vacation. [03:07:37] guillom: +1 [03:07:44] * YuviPanda has a few weeks next month [03:08:23] On the one hand, vacation time means less time for making progress on those quarterly goals. On the other hand, one of my quarterly goals (postponed from last quarter) is "take a vacation" so there's that :p [03:08:43] I need minions!!! [03:09:00] Sorry, I mean, unpaid interns. [03:17:24] :D [03:23:36] guillom: you could make them take your vacation for you! [03:24:17] Emufarmers: Always so helpful <3 [03:24:49] :> [03:25:29] Fun fact: It's really difficult to search vacation rentals by type ("cabin in the woods with trails, hot water and internet") instead of by location. [03:27:07] Take a sabbatical to found a startup for that! [03:28:44] Step 1: Find interns. Step 2: Have them find where to go on vacation. Step 3: Go on vacation, let the interns do my job. [12:12:55] guillom: or you could let the interns go on vacation, and save yourself the bother of having to find the right place at all. [12:13:12] They are relaxing for you. [14:15:08] o/ [14:26:46] morning halfak :) [14:27:22] Hey Ironholds! [14:27:32] Just about to start my trip today. I'm off to Omaha first [14:34:23] halfak, cool! What's there? [14:42:52] Ironholds, it's a workshop for open science practice. [14:43:05] Mostly for people leading other initiatives. [14:43:23] E.g. https://meta.wikimedia.org/wiki/Research:Breaking_into_new_Data-Spaces [14:43:35] cool! [16:13:47] _o/ [16:15:40] o/ kjschiroo [16:15:50] Good morning guillom :) [16:16:29] kjschiroo, if you are around, please meet joal. [16:16:52] I'm here [16:16:53] joal is an engineer at the WMF who I work with on big distributed processing jobs like the onces we have been experimenting with. [16:17:01] guillom == joal? [16:17:08] nope :) [16:17:11] Oh [16:17:18] oh joal == joal? [16:17:21] nope. Just happened to want to do the intro and say Hi to guillom at the same time. [16:17:29] Sorry for the confusion [16:17:38] * guillom greets kjschiroo nonetheless. [16:17:40] * halfak doesn't know how to aim his head in different directions on IRC. [16:17:56] hi kjschiroo o/ [16:18:12] * joal greats everybody in the room in the mean time [16:18:27] Hi joal, I may be asking you some questions shortly about spark [16:18:35] sure ! [16:18:37] specifically on altiscale [16:18:43] hm [16:19:10] Are you familiar at all with that environment? [16:19:35] more or less :) [16:19:44] depends on the complexity of the question I guess [16:20:19] Right now I am still struggling to find the entry point so I can get my feet under me. [16:20:38] Ah, I think I can help on that :) [16:21:03] The first question I'll have is: what type of job/shell do you want to work with? [16:21:44] also kjschiroo, we are talking of the Internet Archive cluster, right ? [16:22:35] joal & kjschiroo: indeed. the IA/Research cluster [16:22:41] I'm just starting to dive into this so I'm not sure that I can say what type of job/shell I should want to work with. I am talking about the Internet Archive one. [16:22:46] Thanks halfak for the confirmation [16:22:47] Just got kjschiroo set ip and running hive queries last week. [16:23:11] Awesome halfak and kjschiroo :) [16:23:23] Hive is a cool entry point for hadoop stuff :) [16:23:36] kjschiroo expressed interest in experimenting with pyspark too [16:23:47] Which is on my todo list for working with joal [16:23:57] Yeah [16:24:01] That was my question halfak, scala or python ? [16:24:07] So, python it will be :) [16:24:41] That is my personal favorite! [16:25:15] cool kjschiroo :) [16:25:34] Oliver isn't around, the Python cabal can take over ;) [16:26:20] kjschiroo: on altiscale IA cluster, (as on any altiscale cluster actually), the core of stuff is installed in /opt folder [16:26:32] You might be interested to have look over there :) [16:26:50] I'll take a look there. [16:27:07] Is pyspark setup on the IA cluster? [16:27:37] kjschiroo: Fast-tracking on the spark path: /opt/alti-spark-1.4.1.hadoop24.hive13/bin/pyspark is the command you are after [16:27:59] Yes!! That is what I was looking for! [16:29:40] Then, there are some parameters to give pyspark in order for it to work nicely with Hadoop-Yarn: [16:30:05] An example: /opt/alti-spark-1.4.1.hadoop24.hive13/bin/pyspark --master yarn --driver-memory 2g --num-executors 4 --executor-memory 2g --executor-cores 1 [16:30:26] I let you try to guess what that means in term of cluster resource usage :) [16:30:30] kjschiroo: --^ [16:31:14] I think I have a pretty good idea. What is --master yarn doing? [16:32:06] It tells spark to use yarn as the resource manager (instead of doing local) [18:12:07] * halfak runs to airport [18:12:16] See you folks later [18:17:28] safe travels, halfak! [20:09:51] schana: are you working with the hold laptop or a new one? :D [20:09:54] I see patches [20:11:54] leila: the old one - I haven't rebooted it since I got X running and it's holding out so far :) [20:12:40] haha. great, schana. [20:12:48] It was very exciting to see the patch. thank YOU! [20:13:27] np [20:13:34] okay, signing off to go to transit mode. see you in a bit again. [20:13:37] * leila waves [20:51:56] * YuviPanda waves at schana [20:52:09] howdy [20:53:46] schana: finally I get to meet you :) We had a (planned) maintenance window this morning that was threatening to spill over, but worked out ok and just in time :) So I'll see you shortly! [20:58:15] sounds good YuviPanda! [21:01:24] schana: you aren't in the office right? [21:02:14] YuviPanda: right [21:02:27] cool :D [21:02:33] * YuviPanda is already in the hangout [21:05:10] schana: http://wikitech.wikimedia.org/wiki/Labs_labs_labs [21:07:17] schana: https://wikitech.wikimedia.org [21:08:59] YuviPanda: Nschaaf [21:10:14] schana: ores project [21:12:41] schana: https://github.com/wikimedia/operations-puppet [21:13:30] schana: https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/labsores.pp [21:19:38] schana: phabricator.wikimedia.org/paste [21:20:10] https://phabricator.wikimedia.org/P2535 [21:21:20] schana: schana-test.ores.eqiad.wmflabs [21:21:24] or schana-test.eqiad.wmflabs [21:23:08] schana: sudo puppet agent -tv [21:33:40] schana: quarry.wmflabs.org [23:03:58] hey YuviPanda. Is PAWS down? I'm unable to start my server. [23:04:36] J-Mo: someone is hammering NFS so possibly related. what error are you getting? [23:05:27] no error yet, it just hangs after I try to start the server. 'refresh' has no effect. [23:05:54] ok let me fix my current fire I'll look at that after [23:05:57] http://takemehome.tv/images/pawsup.jpg [23:06:12] We clearly need a PAWS status gif. [23:06:13] thanks guillom, that helps [23:06:36] and YuviPanda: no rush on this. feel free to triage this request as you see fit ;) [23:06:52] ooh! now I get 500 Internal Server Error [23:24:39] J-Mo: do you have a sample message for surveys that involve subject recruitment? [23:25:09] J-Mo: there is usually a landing page where potential participants can learn about the Research and have a more informed way of deciding whether to participate in the survey or not. [23:26:01] leila: are you in the office today? [23:26:13] YuviPanda: unfortunately, not. [23:26:27] YuviPanda: if tomorrow is as good as today, I will be there tomorrow. [23:26:57] ellery and I are working from Stanford today, YuviPanda. We are trying to make it a more regular thing again, to work from campus once a week. [23:27:03] ah [23:27:06] nice [23:27:09] and you're welcome to join, YuviPanda. :-) [23:27:12] I should find a way to sneak into standord some day [23:27:13] hah [23:27:16] should do, leila :D [23:27:17] YuviPanda: in a meeting… back in 20 [23:27:20] let me know if you're oding it next week [23:27:24] J-Mo: yeah, just ifnished my other fire [23:27:28] let me look at this one :D [23:27:34] excellent ;) [23:27:34] YuviPanda: will do. We owe you at least a tour. ;-) [23:28:17] YuviPanda: are you in the office? [23:28:25] leila: I'm considering going to the office :) [23:28:41] so was looking for reasons [23:28:43] might go anyway [23:28:50] talk to ori, YuviPanda. ;-) [23:29:00] haha [23:29:02] mysterious :D [23:29:16] I will!