[00:28:00] alright, I'm off for the evening [03:44:59] evening leila :) [03:45:09] hello Ironholds. how's it going? [03:45:15] same as always :) [03:47:05] what are you up to Ironholds? [03:47:31] writing yet another pageview retriever [03:47:40] comms decided they want some data on monday they couldn't ask me for earlier [03:47:42] ditto fundraising [03:47:53] I love my coworkers, I love my coworkers, I have nothing but warm feelings towards all of my coworkers [03:47:56] * Ironholds repeats the litany [03:48:46] depending on what the nature of it is, Monday may not be that bad. [03:48:58] you were used to passed due dates. ;-) [03:51:58] yeah, but I hated them ;p [05:48:59] Ironholds, I'm signing off. have a good night. [05:49:24] leila, take care :) [05:49:34] you, too. :-) [05:49:35] ciao [05:53:13] how's it going, Ironholds? [05:54:50] fhocutt, oh, that's a very complex question ;p. [05:54:56] Let's go with: same as always ;p. You? [05:55:23] all right. My afternoon got eaten, so I'm working on stuff now. [06:07:28] Cool! [06:07:33] fhocutt, awesome presentation, btw :) [06:07:39] thanks, Ironholds! [06:07:58] I didn't have any thoughts except really wanting to build a library in R for fake http requests [06:08:01] glad you enjoyed. Was there anything you found particularly interesting? [06:08:05] that! [06:08:09] like, it's the bane of the language [06:08:19] there's a HTTP library, but there's no library for non-live requests [06:08:29] huh. [06:08:31] and that means that unit tests of actual calls either genuinely call out or don't...work. [06:08:39] so, I've decided, based on that presentation, to build one. [06:08:41] interesting. [06:08:46] very cool. [06:08:46] htttest, I'm going to call it. [06:08:52] because: imaginative. [06:08:54] * fhocutt hasn't done anything with R yet [06:09:01] also I thought a lot about the things I really hate about our API :/ [06:09:01] hey, principle of least surprise :) [06:09:07] oh, do tell [06:09:16] the XML-inherited key-pair structure? [06:09:22] yep. [06:09:29] If I say "grab me the last 10 revisions for [page]", it returns revID as the key [06:09:58] in languages where the JSON structure isn't integer-based - i.e., where I have to say "give me item foo", not "give me item 5" - which is a fairly common paradigm, I think [06:10:15] that means that I have to parse up and down the JSON structure extracting actual names [06:10:20] yep. At least there's the "grab a list of keys" option, but it is a workaround [06:10:29] yeah, and it's not fully documented [06:10:37] ...story of the API, really [06:10:37] or, wasn't when I wrote WikipediR [06:10:39] heh [06:10:51] the only other frustrating thing is how the API treats non-cached diffs [06:10:56] though to be fair, it's gotten better. [06:11:32] what does it do with those? [06:13:03] so, you request a cached diff. Fine. It returns a cached diff. [06:13:14] you request a cached diff and an uncached diff. Fine. It returns a cached diff and an uncached diff. [06:13:23] You request a cached diff, an uncached diff, and another uncached diff. [06:13:30] It returns a cached diff, an uncached diff, and an empty string. [06:13:41] no error. No warning. An empty string. Which real diffs can naturally consist of. [06:13:43] whoa. [06:13:51] not so good, no. [06:13:55] and, of course, the fact that this behaviour is caching-based, means it's impossible to write tests for. [06:14:02] because by dint of running the test, the test becomes invalid [06:14:09] it's very heisenburg-ish [06:14:10] right [06:15:34] this is, as with all things, Roan's fault [06:15:41] I mean, I tend to blame him for anything, but this one genuinely IS roan's fault. [06:16:20] so yeah, those are my only big bugbears. [06:16:30] I would really like it if we totally screwed the API syntax, of course, but. [06:17:02] there's the rewrite that's being worked on [06:17:14] and/or planned, I'm not sure of the status [06:17:27] yay! [06:23:14] Ironholds, are you coming to the dev summit? [06:25:48] almost certainly not, I'm afraid. [06:26:54] ah, well [06:28:32] I will be at the overall event, though; we should chat :) [06:28:49] I'd say "I'll show you around R!" but momma always told me not to make people hate me on a first impression [06:29:24] * fhocutt grins [06:29:56] I'll probably be introduced to it one of these days [06:44:52] enjoy the four classs systems, four assignation methods, and NULL. [06:44:55] NULL is my favourite thing. [06:45:49] favorite or "favorite"? [06:45:55] yes [06:46:09] okay, the latter. Let's be honest, it's the latter. [06:46:24] that's what I thought. [06:47:00] has anyone EVER said "X is my favourite feature" without a facial expression that communicates "I'm going to make your internal CS person curl up and die of shame"? [06:47:36] someone must have. [06:48:26] this is my most recent favorite feature: http://docs.python-guide.org/en/latest/writing/gotchas/ [06:48:39] the way Python handles mutable default args [06:49:01] ...what [06:49:06] okay, here's NULL, then. [06:49:13] So, NULL. Like None, right? Ternary logic [06:49:14] NOPE. [06:49:18] R has NA for ternary logic. [06:49:25] in fact, if you compare something to a null, what do you think happens? [06:49:26] okay... [06:49:31] it's never true? [06:49:51] that too [06:49:59] but, so, TRUE == FALSE is FALSE. Doy. [06:50:05] TRUE == NA is NA. Doy. Ternary logic. Cool. [06:50:09] TRUE == NULL is? [06:50:18] FALSE? [06:50:31] ...TRUE? [06:51:11] oh, not true [06:51:18] ......NA? [06:55:21] nope! [06:55:23] logical(0) [06:55:26] it does not produce an output [06:55:29] at all [06:55:40] you'd be amazed how much you can fuck a codebase by passing an if-statement /nothing/ [06:55:49] * fhocutt blinks [06:56:15] the wrong sort of nothing, of course. [06:59:58] yuuup [15:11:38] sup [15:16:03] hey halfak :) [15:16:16] g'morning Ironholds [15:16:45] how goes? [15:20:37] hey halfak, i was minding, but i think others are now, do you mind if I kill your long running streaming job? [15:23:27] hey halfak, i was minding, but i think others are now, do you mind if I kill your long running streaming job? [15:23:42] i was *not* minding [15:25:44] which others? [15:25:47] I mean, I don't mind [15:28:05] ottomata1, that would be frustrating. I'm also curious who I'm hurting and how they know I'm causing them pain. [15:28:48] (sorry to be slow, I'm in a meeting. Will be done in an hour) [15:29:08] qchris_meeting: was noticing that it has been taking up half of the cluster's memory for many weeks now, and it is making it harder for other jobs to find containers to run [15:29:24] depending on how busy the essential jobs are, it makes it harder for others to develop and test [17:04:51] ottomata, checking on my progress with that job now. [17:05:21] bah. I think I need a reboot. BRB [17:05:29] halfak: qchris_away estimated how long he thinks it would take for that job to finish based on current rate [17:05:33] he thinks at least 2 more weeks [17:06:06] Yeah. Something is broken about it. The same job for simplewiki takes a few hours. [17:06:13] Re. memory usage, what can I do? [17:06:21] My streaming process isn't using the memory. [17:07:59] hi everyone. [17:08:42] anyway, going for a reboot [17:08:47] o/ leila [17:08:56] back in a bit! [17:13:02] Ironholds: do you know if all the schema should automatically come to this page? [17:13:03] https://meta.wikimedia.org/wiki/Research:Schemas [17:13:35] it's last updated in May but I'm guessing we've had more schema built since then (Content Translation if, for example, not in this list) [17:13:41] yes, iff they've been categorise [17:13:41] d [17:13:52] see https://meta.wikimedia.org/wiki/Schema_talk:EchoMail for cat examples [17:14:08] cool! thanks! [17:18:41] Yup. Can't seem to port forward through Bastion all of a sudden. [17:18:56] * halfak does the googles [17:22:03] ottomata, that process is not using any more memory than it ever was (according to job -list) [17:27:08] halfak: , i'm not exactly sure if I am lookint at this right, but is it telling me it is using 2.6 TB? [17:28:39] I see 500GB for both "UsedMem" and "NeededMem". [17:28:53] job_1415917009743_47379 RUNNING 1419091740811 halfak root.default NORMAL 51 0 417792M 0M 417792M http://analytics1010.eqiad.wmnet:8088/proxy/application_1415917009743_47379/ [17:29:07] I guess that's more like 400GB [17:29:17] that would make more sense [17:29:23] i did [17:29:24] mapred job -status job_1415917009743_47379 [17:29:27] Physical memory (bytes) snapshot=2422320500736 [17:29:33] anyway, yeah, 400GB makes more sense [17:30:10] that is close to half of the cluster's memory [17:30:51] halfak: i only asked because some folks wanted to in standup, and milimetric seemed to think it was ok. it has been running for quite a while... [17:31:21] It seems that memory usage corresponds to the # of containers used. [17:31:50] So, I might not be able to use 50 reducers and use any less memory. [17:32:01] my reasoning was: once this thing ran longer than it did on stat1003, the load test failed in my opinion [17:32:03] If I'm using any less than 50 reducers, I might as well just run on stat3 [17:32:13] I think that's fair. [17:32:32] But I want the data more than I want to know Hadoop performance. [17:32:43] oh I thought you got the same data on stat1003 [17:34:55] Negative. It never finished. That was my estimate based on similar runs I have done in the past (years old data) [17:36:27] oh, I didn't know that was an estimated run time. But, estimated run time for Hadoop was much lower as well, so we may not be comparing apples and apples here :) [17:36:58] Indeed. But we don't get that many new revisions in two years [17:37:11] halfak: something is fishy here and i'm not sure what, do you think that this job should actually take this long in hadoop? [17:37:18] No [17:37:29] definitely not, something is completely wrong [17:37:38] but it's not obvious, we need to all look closer [17:38:01] is there any way to get a sense of how much data it's looked through already? [17:38:28] I've considered shutting it down for a lot time. There's two conditions that would make that an easy decision. (1) it is actually preventing others from doing work or (2) we know what the problem is and how to solve it. [17:39:10] (1) is definitely true, Christian estimates this hurts most aspects of working with the cluster for everyone using it [17:39:40] (starting new jobs for example takes longer because there's less memory and more competition) [17:39:57] maybe let's try to look for a problem for one full day [17:40:12] tomorrow seems like a good day? [17:40:52] halfak: can you make time tomorrow? or this afternoon? [17:41:56] :( My calendar sucks this week. I can look at it after 4PM EST today and after 4:30 EST tomorrow. [17:42:38] If (1) is really clearly true (it seems that there was some confusion about my memory usage), then I'd like to stop the job. [17:42:41] ok, mine sucks too but this seems urgent. let's do today at 4 for an hour [17:42:43] But I want to know that we know that. [17:42:48] Thanks milimetric [17:42:58] i'll set it up [18:32:42] ottomata|lunch, when you get back, I'm down for kill that hadoop job now if we feel like we *know* that it is causing problems (note that, if we're wrong, this will cost me up to 20 days). [18:33:19] Then again, I'm looking at running this job outside of our cluster, so it may not really matter. [18:34:28] qchris: ^ i say let's wait, i halfak, i guess we have a meeting with toby and appan about that in a few hours anyway? [18:34:44] Indeed. [18:35:04] Sounds good [18:35:25] ottomata: Totally ok to wait from my point of view. [18:35:42] k [18:35:45] I know that it is causing issues for me, but [18:35:51] I can take it for a few more days. [18:38:56] qchris, how do you know this? [18:39:02] when did the issues start? [18:39:23] I had jobs that have been waiting for memory. [18:39:31] But no other memory was free :-/ [18:39:50] hmm. Am I right that the memory usage is based on container usage? [18:40:22] i.e. processes use all the memory that a container has available regardless of whether it is needed [18:41:24] qchris, ^ [18:41:43] When running jobs by hand, there have been a few config settings where jobs could specify how much resources the job thinks that it needs. [18:41:50] But I never used that again :-/ [18:42:04] Let me see if I can find those setting again. [18:43:43] Meh. Right now, I can only find hadoop wide settings for the containers :-/ [18:44:19] Gotcha. The process I am running shouldn't need more than 100 megs of memory per reducer stream [18:44:32] BUT I suspected that the sort/shuffle would use memory if it could. [18:44:56] Ja. makes sense. [19:02:19] hiyo [19:02:21] meeting time!? [19:02:36] yup. [19:02:40] Dan and I are call [19:02:44] You can be call too [19:02:46] the dan-man! [19:03:40] hm? [19:03:45] i am in the meeting [19:03:47] where are you? [19:03:51] ping YuviPanda: Edward Galvez (Grantmaking) is using Quarry to get updated participation #s for State of the Wiki. He's pretty new to SQL, but getting more savvy every day. He's loving the tool. And I'm loving it too… because I can share examples and sanity-check his queries for him totes easily. [19:04:03] ottomata, we are all here. Come to us. :) [19:04:39] where? [19:54:01] hi J-Mo, got a minute? [19:54:10] sure, what's up? [20:25:50] where's ellery got to? [20:26:14] tnegrin, I'm now talking Ellery through writing UDFs and handling gerrit. [20:26:17] NOT A SOFTWARE ENGINEER. [20:26:37] on this channel you are [20:26:46] ;p [20:27:19] * Ironholds grumbles [20:53:24] Ironholds: do we have anything to talk today that was not already covered earlier this week? I could use some meeting-free time [20:54:40] I mean, Toby threw a load of pageviews work at me [20:54:50] I'd like to get your take on the priorities for the next quarter [20:54:53] that's it at my end [20:55:00] alright, cool [20:55:26] but we can totally cut it early; that shouldn't take a ton of time [20:58:33] Ironholds: kk, finding a room [20:58:56] also, halfak: https://twitter.com/peakscale/status/552985151794208769 [21:28:06] halfak around? [21:28:33] ToAruShiroiNeko, yes, but I'm just about to do my last meeting for the day. Looking to hack on some revscores? [21:28:49] Ironholds, lol. /me imagines cuts and crashes. [21:29:19] sure but also was curious if we can get some graphics before your presentation [21:29:36] yeah. that's a good idea. I forgot to point you at my notes too. [21:29:50] * halfak looks for the etherpad [21:30:42] http://etherpad.wikimedia.org/p/intro_to_revscores [21:36:19] ToAruShiroiNeko, into meeting. Done in 30 or so [21:36:32] kay [22:24:24] Ironholds: , halfak, leila, i should have mentioned this in the meeting today, but forgot. for now, the wmf.webrequest refined table only has mobile and next [22:24:28] no upload or bits [22:24:29] just fyi [22:24:33] YAY [22:25:20] ToAruShiroiNeko, running a little long. Will ping you when I get back. [22:27:44] sure [22:34:08] ottomata: thanks! (I guess next is text) [22:36:23] sorry [22:36:25] mobile and text [22:36:26] it has text. [22:36:26] yes [22:49:41] I booked a conf room in SF for rest of year for hour 2nd Thurs RD/DevOps meeting [23:08:01] hey ToAruShiroiNeko, Sorry for the delay. [23:08:07] Ready to chat now.