[00:21:11] DarTar, halfak, where should I go if I need to access some of my old cards? :D [00:21:25] you don’t yet :) [00:21:31] they are coming back soon [00:21:36] great! [00:21:49] and they will be re-assigned to you [00:22:03] we accidentally removed all assignments by shuffling cards around, yay [00:22:13] but they’re coming back [00:22:25] cool! I noticed I suddenly have much less to do. ;p [00:55:52] Ironholds, did you say that WMutils has code for parallelizing? [14:08:23] morning [14:28:10] whelp, ellery is using 50% of stat1002's memory. I get to go tattooing! [15:10:43] baah [15:10:51] Hey dude. [15:10:57] Just saw your email re. trello board. [15:11:02] halfak, region-/city-/country-/timezone-level geolocation, all in C. [15:11:05] but I cannot port netspeed! [15:11:09] do you know why? [15:11:10] Did you actually lose any cards? [15:11:26] Cool! [15:11:32] because the underlying API has a bug. A bug identical to...the bug I fixed in the Python API. [15:11:40] when I tried to use it there, and ran into exactly the same issue. [15:11:53] the lesson is: ports are not feature-parity, they are also bug-parity. [15:11:59] * Ironholds meditates on that. [15:12:11] and yeah, one card ended up in -archive - I shifted it back to the main board. [15:12:31] -archive is only stuff that was already in a "completed" list. [15:12:42] So it must have gotten moved there by accident. [15:12:46] bah, sorry [15:12:49] ended up in -backlog [15:12:55] * Ironholds needs caffeine this morning [15:12:56] brb [15:13:07] Oh. Yeah. Then it was not in "in-progress". [15:13:48] When you get back, can you link me to the card? [15:13:58] I'd like to examine the history to see how we made the mistake. [15:15:55] so, it looks like Toby added it to "New" [15:16:01] but I don't get why "New" means "move it to the backlog. [15:16:13] New means "I haven't had a chance to assess how big of a deal this is and move it into staging yet". [15:16:38] "new" == "backlog" [15:16:46] Same thing. [15:16:49] yeah, my brain doesn't do that mapping [15:17:01] backlog == We're not ready to start this yet. [15:17:04] backlog implies, to me, "stuff we really should get to but we haven't yet", with an implicit recognition that the stuff isn't vital. [15:17:14] maybe this is a legacy of working on the Flow team and their specific workstyle. [15:17:19] Na. All new things go in the "backlog" [15:17:34] huh. Okay; that...seems really confusing. [15:17:54] I like to have all of my cards visible so I don't lose track of em. Having to transition between multiple boards to do that (and a gdoc, for mobile, and another gdoc, for the board) is non-ideal. [15:17:55] :P I think it's the commonly accepted definition of backlog. [15:18:00] * halfak googles [15:18:41] https://en.wikipedia.org/wiki/Scrum_%28software_development%29#Product_backlog [15:18:59] yeah, fair point [15:19:04] Seems about right [15:19:22] So, at some point, we need to sort the backlog. [15:20:19] I think we should wait until Grace shows before more tasks like this. [15:20:36] getting things non-crazy for her is a good idea, but she may have her own ideas about process that we need to factor in. [15:20:49] (I mean, I hope she does. I dunno why we'd hire her if she doesn't.) [15:20:53] For now, it is at least categorized [15:20:53] And all of the crusty cards (like 40% of what was there) are gone. [15:20:53] I found a lot of dupes and consolidated. [15:21:35] Ironholds, we didn't implement a new process. We just cleaned up our mess so that we could show Grace where things are. [15:21:49] How would she know what cards were real and which ones were crusty old nonsense? [15:23:30] no, totally! [15:23:38] I'm not saying "let's undo things!" [15:23:46] I'm saying we should wait on prioritisation/ordering until she's here. [15:23:54] that's the process I meant. [15:24:20] Oh. Well, I don't think that's going to be her job, but I'm OK with waiting the two hours until she's in the office. :) [15:24:30] hehe [15:24:40] I think she'd find it useful to see how we do it. [15:24:41] I think that we'll still be responsible for prioritizing. [15:24:48] Oh sure. that's a good point. [15:24:52] totally, I just want to give her an eye into the dynamics [15:24:56] snap! [15:25:28] okay, may have got netspeed working... [15:28:05] Hey Ironholds, do you know a good way to check on the progress of a hadoop job? [15:29:02] only hadoop job -list [15:29:16] there's a mapred equivalent I'm warned about every time I use it, but it helpfully doesn't tell me what that equivalent /is/ [15:29:29] yarn application ...? [15:29:38] or, you can look at that web gui and see stuff [15:29:50] What web gui is this? [15:29:59] hue! [15:30:00] * halfak has been poking at docs on wikitech [15:30:07] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Access [15:30:16] here try this [15:30:17] ssh -N bast1001.wikimedia.org -L 8888:analytics1027.eqiad.wmnet:8888 [15:30:18] its fun! [15:30:22] then go to [15:30:25] https:/localhost:8888 [15:30:36] log in with your shell username and your ldap pw [15:30:57] actually, yeah, go to [15:30:58] https://localhost:8888/jobbrowser/ [15:32:33] wheee! https://gist.github.com/Ironholds/38ba1e27017e544925df [15:32:40] okay, netspeed ported. Massive speed increase. [15:32:48] Thanks ottomata. This works great. [15:32:55] Now to find WTF errored. [15:33:06] I've never seen map 100%, reduce 100%, but error. [15:33:21] I have! [15:33:23] Oh wait. It shows all of the jobs as 100% [15:33:24] oh, I've seen some /shit/. [15:33:41] I've seen jobs on fire off the analytics 1010 gate [15:33:45] HDFS files glitter in the dark [15:33:58] mind you, that second one might have been the acid, but regardless. [15:34:12] halfak: sometimes i see exceptions at the end of jobs too, but those jobs still compelted [15:34:17] also, your alternative gui: [15:34:26] ssh -N bast1001.wikimedia.org -L 8088:analytics1010.eqiad.wmnet:8088 [15:34:28] and then go to [15:34:32] http://localhost:8088/cluster [15:34:37] it isn't as friendly, but sometimes you can get more info [15:35:08] I'm seeing: java.io.IOException: Cannot send data to logstash1002.eqiad.wmnet/10.64.32.137:12201 [15:35:16] That wouldn't cause a crash, would it? [15:35:20] no [15:35:29] that means that your logs aren't being sent to logstash [15:35:33] :( [15:35:38] did it crash your job, or just show you the exceptoin? [15:35:53] I'm not sure how to tell what caused the crash. I'll try /cluster [15:36:06] what is your application_id? [15:36:27] application_1415917009743_5137 [15:40:54] Progress: 10000% [15:41:07] DONE'R THAN DONE! [15:41:54] Got an error. [15:41:59] Now that's weird. [15:42:04] oh? [15:42:14] Most mappers just worked, but some had an error while starting up. [15:42:27] It looks like my packaging of virtualenv failed. [15:42:33] But only for some mappers. [15:43:31] * halfak gets out the brain shovel and starts digging. [15:43:58] hm [15:46:19] So, I take it back. It looks like all of the mappers for that job failed. [15:46:28] But the same code worked for the tiny sample. [15:46:39] application_1415917009743_5136 [15:47:38] * halfak seriously considers just resubmitting the job and seeing what happens. [16:05:51] halfak, I blame versioning! [16:06:01] we had a problem for a while where queries would sometimes fail, and sometimes not. identical queries. [16:06:08] turned out to be Diedrik manually upgrading Java on one machine. [16:09:48] Hmmm. I'm running a new test now with a setup I know worked. [16:10:01] Bah! And it worked again. [16:10:11] The only thing I am changing is the input XML dump! [16:10:12] Bah! [16:50:01] diedrik? [16:50:43] halfak: is it possible there is sometihng in the other xml dump input that is causing some exception [16:50:48] hm, no but you are saying all mappers fail [16:50:49] hm. [16:51:04] Yeah. The error is an import at the top of the file. [16:51:08] oh [16:51:24] oh strange, but it works with the small xml dump? but not the real one [16:51:25] Doesn't make sense. [16:51:26] that is strange [16:51:27] ghm [16:51:27] hm [16:51:28] Yeah [16:51:28] ottomata, waybackwhen [16:51:31] ah, ok [16:51:32] I just ran another test [18:06:33] yo Ironholds [18:07:22] yo dude [18:07:58] we just welcomed bob to the office [18:08:15] and grace is here too, feels like christmas :) [18:09:41] bob? [18:10:11] bob west [18:10:32] helpful! [18:10:34] bob west? [18:10:39] himself [18:10:45] aka robert west [18:10:54] I'll ask him to join the IRc [18:10:58] I still have no idea who you're talking about [18:11:04] could you just tell me what this person does? ;p [18:11:12] the stanford collaboration Leila has been working on [18:11:16] we meant to talk about him in the staff meeting yesterday Ironholds [18:11:20] but then we ran out of time [18:11:20] :D [18:11:21] and he’s done some anazing Wikipedia research [18:11:22] just got a stat account too, I think. [18:11:24] I'll send out an email [18:11:29] to the research-list [18:12:28] Ironholds: wanna jump on a quick with call with da toby? [18:12:37] sure [18:12:48] just send me ze invite [18:59:42] hey leila, halfak, can I ask a favour? [18:59:56] I'm out Friday recovering. DarTar should ALSO be out Friday. If you see him online, scold him for me? [19:01:45] why? cuz you feel you may fall behind, Ironholds? :D [19:01:54] enjoy the time off [19:02:05] you've been working really hard [19:02:20] no, no [19:02:29] I'm saying he has been working off and should take the time [19:02:33] if he tries to work, tell him off ;p [19:02:42] *has been working hard and [19:12:34] halfak, you remeber editor_month_by_namespace table you made a while back? [19:32:43] Yeah leila. What's up? [19:32:55] Ironholds, will do. [19:33:07] DarTar, you best be out on Friday. >:( [19:33:08] halfak, ta [19:33:09] halfak: I’ll have to cancel our 1:1 for today [19:33:20] S'ok I think we're good. [19:33:20] halfak: yes, it’s tempting [19:34:07] Mia started making horrible noises every time she sees me past 8pm or before 6am with RStudio instead of La Repubblica on my laptop [19:34:33] What's La Repubblica [19:34:35] ? [19:34:57] DarTar, you just need to use Shiny to make it into a web app called Dora the Rsplorer [19:35:03] then you can call it "educational" [19:35:07] halfak, so I made another table with user_id, page_namespace, month, reverts. I can add reverts as a column to your table, or let it be where it is in mine. [19:35:13] thought to check with you. [19:35:17] halfak: it’s Italy’s NYTimes [19:35:26] leila, feel free to add. [19:35:39] You should check that the table is not myISAM first [19:35:57] if it is MyISAM, then you should re-create it as innodb. [19:36:08] MyISAM gets angry when you try to add columns. [19:36:47] got it. I'll check before doing that. okay. then know that we have a table for reverts now if I can't add the column to yours [19:37:09] :) Reverted or reverting? [19:37:25] reverted [19:38:09] you mean based on whether user_id has done the revert or his revision is reverted, right? [19:38:14] yes [19:38:32] Ironholds: check out the captions that I added to the new slides, any thoughts? [19:39:02] DarTar, in 1:1 :( [19:39:13] sure, no worries [20:54:31] hey halfak, you like brain-puzzlers? [20:54:53] My favorite type of brain- [20:55:35] you ever encountered the MIT Mystery Hunt? [20:56:47] Negative [20:57:33] five day long brain-puzzling competition in Boston, run by whoever won the previous year [20:57:40] https://en.wikipedia.org/wiki/MIT_Mystery_Hunt [20:58:34] the team names are the best [20:58:41] "the team formerly known as [the entire text of Atlas Shrugged]" [21:01:05] YuviPanda: Hi! How long does it take for a fix (such as the one for bug 69544) to be deployed on quarry? [21:01:14] hi [21:01:17] I manually do deployers [21:01:20] *deploys [21:01:22] let me do one now [21:02:15] no hurry... [21:02:31] I was just wondering how it works.. [21:03:00] Helder: done now [21:03:18] Helder: I use fab, it's on the repository. I do a 'fab update_git restart_uwsgi' on my machine and it does the rest of the steps [21:03:25] a bunch of other people also have deployment access [21:03:39] cool! [21:03:49] :) [22:05:53] Ironholds, was in meeting. Now I got to read. This is pretty awesome. [22:05:57] Re. MIT Mystery [22:06:21] (not sure if that just double posted. my IRC client is acting up) [22:07:08] it didn't! [22:07:11] and yeah, should be fun. [22:07:18] 2 months to go /me cracks knuckles [22:15:53] DarTar, can you release R35 for our 1:1?