[00:14:01] yoyoooo, dschoon, any luck? [00:14:33] looked a little bit. [00:14:58] finishing up questions for diederik first, tho [00:15:07] will be on it in maybe 30? [00:15:59] that ok, ottomata? [00:16:19] ja [00:19:19] erosen: question [00:19:37] erosen: about matplotlib [00:20:11] erosen: I have 4G memory on this laptop [00:20:31] and while plotting with matplotlib I get 30% up to 70% gulped by matplotlib [00:20:55] erosen: I would be interested if you encountered this before [00:21:05] they're just regular barcharts [00:21:15] on stat1 I don't think it will be a problem since it has plenty of memory [00:21:29] erosen: https://github.com/wikimedia/metrics/commit/8c404e2bf9e132483d78fd973bb4c96ce6a70826 [00:21:45] drdee: did you get big memory consumption when using matplotlib ? [00:21:45] ottomata, new hue is cool and job browser works! [00:21:56] average_drifter: not that i recall [00:21:59] ok [00:22:02] new hue? [00:22:14] yes new hue [00:22:18] but shit i spoke too quickly [00:23:16] erosen: ok, moving to stat1 for a full run nov=>dec every day of it [00:23:17] did we upgrade? [00:23:23] i thought we were already on 2.2.0 [00:23:30] hm [00:23:34] yeah, definitely has new stuff. [00:23:38] impala! <3 [00:23:39] ottomata upgraded to cdh 4.2 [00:23:44] yes impala as well [00:23:47] ah, yes [00:23:58] didn't mean to upgrade! [00:24:05] somehow half of the nodes were already upgraded [00:24:09] so I just finished the job [00:24:13] boo, \u002D in names [00:24:15] but now i'm stuck on this hostname problem, which I didn't have before [00:24:19] http://localhost:8888/jobsub/ [00:24:25] job browser somewhat works but throws a 500 error if you dive into stuff [00:24:41] later guys [00:24:55] probalby for the same reason these other things aren't working, job browser would have to talk to resourcemanager [00:24:58] actually, drdee [00:25:01] how do you get a 500? [00:25:25] average_drifter: great [00:26:06] AH HA [00:26:10] dschoon, look at this! [00:26:10] Mapreduce.jobtracker.address [00:26:10] analytics1010.wikimedia.org:8032 [00:26:13] tsk tsk tsk [00:26:21] try http://localhost:8888/jobbrowser/jobs/job_1363208055308_0021 [00:26:25] what! [00:26:28] nope. [00:26:29] i think I need to set that explicitly [00:26:29] how! [00:26:34] i dunno, i'm not setting it [00:26:42] so probably the default is somehow picking that up [00:26:42] no idea how [00:26:49] foudn that ind a job conf [00:26:49] http://localhost:8888/jobbrowser/jobs/job_1363208055308_0001 [00:26:53] ahhh [00:26:53] yay for job browser! [00:26:56] i see why [00:26:58] oh? [00:27:09] erosen: http://bit.ly/ZAUssn [00:27:12] [00:27:12] mapreduce.jobtracker.address [00:27:12] localhost:8021 [00:27:14] mapred-site.xml [00:27:16] [00:27:20] *localhost* [00:27:29] erosen: have a look please . I tried to make a small barchart class [00:27:30] https://issues.apache.org/jira/browse/MAPREDUCE-3033 [00:27:40] erosen: then iterating through all features and making charts for them [00:27:43] average_drifter: in a meeting, but looking [00:27:55] erosen: we end up with around ~170 charts afterwards [00:27:56] change that to analytics1010.eqiad.wmnet? [00:28:05] erosen: oh, alright [00:28:11] ottomata: ^^ [00:28:16] also [00:28:20] i'm sure you've seen this [00:28:20] http://wiki.apache.org/hadoop/UnknownHost [00:28:45] but [00:28:47] yeah i've seen that [00:28:49] but [00:29:28] hm [00:29:49] hmm, that's weird though [00:29:51] that's 8021 [00:29:59] that's not 8032 [00:30:20] either way [00:30:26] change it to be the internal name? [00:30:28] see what happens? [00:30:33] but that isn't the value that is being picked up by the job [00:30:42] so, i actualy am explicitly setting that in mapred-site.xml [00:30:47] since the old version of hadoop required it [00:30:56] i'm going to first see what happesn if I remove it altogether [00:32:44] dschoon: i updated https://mingle.corp.wikimedia.org/projects/analytics/cards/92 [00:32:57] section Custom UserAgents [00:33:11] will look, drdee [00:33:16] ottomata: what were you setting it to? [00:33:40] before you start hunting for user agent strings, there is a wiki for that, and i added the link in the card [00:35:01] dschoon, i'm going to take it out of mapred-site.xml and leave it to whatever the mapred-default does [00:35:05] drdee: I have a suggestion to make related to custom user agents [00:35:32] drdee: it might be overkill, but I think it would be interesting(maybe good) to find a way to insert them in the dtree data structures [00:35:32] you might be missing the context in which i was using it [00:35:38] aiight [00:35:54] drdee: yeah, i just had that bookmarked from a while ago [00:35:57] figured i'd add it [00:35:59] ok [00:36:17] * drdee is off to the airport [00:40:18] !log restarting hadoop without mapred.job.tracker set to localhost:8021 [00:40:20] Logged the message, Master [00:41:35] can I use the logbot too ? [00:41:39] yup! [00:41:54] !log started charting for each feature on stat1 (500M bump related) [00:41:56] Logged the message, Master [00:42:14] ottomata: can I ammend a log message ? [00:43:27] i think you can edit the wiki page maybe [00:43:33] https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [00:43:35] ottomata: [00:43:50] [00:43:50] mapreduce.jobtracker.address [00:43:50] local [00:43:52] mapred-default.xml [00:43:53] [00:44:00] that seems... inauspicious [00:44:10] yup, same error without it set [00:44:15] ok going to set it now, but i'm not sure what to set it to [00:44:22] i'm starting to doubt this is related [00:45:35] well [00:45:40] !log upgraded to cdh4 4.2.0 :) :/ [00:45:41] it might not be related, but we have to fix it [00:45:42] Logged the message, Master [00:45:46] why? it might not matter [00:45:50] we aren't using a jobtracker [00:45:59] because jobs vomit errors, yes? [00:46:12] yeah but maybe not related to jobtracker address [00:46:12] ps: i vote analytics1010.eqiad.wmnet [00:46:15] there is no jobtracker [00:46:20] that doesn't even really make any sense [00:46:23] btw, are you setting this everywhere? [00:46:30] hmm, i guess it could [00:46:30] yes [00:46:35] but what port? [00:46:37] the resource manager port? [00:46:41] there's no jobtracker runnign [00:46:56] like, it requires updating conf/mapred-site.xml on every goddamn machine, doesn't it? [00:48:25] yes [00:48:29] sed :) [00:48:32] + dsh [00:48:42] (actually i'm doing for loop + ssh, dsh never works 100% for me) [00:49:26] heh [00:51:46] ottomata: hmmm [00:52:13] i think oozie might be the one interacting with the non-existent jobtracker [00:52:25] http://oozie.apache.org/docs/3.3.1/AG_HadoopConfiguration.html [00:52:27] ottomata: ^^ [00:54:29] ottomata: do we have an oozie-site.xml? [00:56:09] it probably wouldn't hurt to add oozie.service.HadoopAccessorService.hadoop.configurations=*=hadoop-conf [00:56:17] or rather: [00:56:50] yes we do [00:57:19] *=/etc/hadoop/conf [00:57:24] thats what it is right now [00:57:38] huh [00:57:48] it doesn't show up on http://localhost:8088/conf [00:58:14] that's in oozie-site.xml [00:58:41] :8088 is the ResourceManager, oozie doesn't run on that node [00:58:48] oozie runs on an27 [00:58:56] iiiinteresting [00:58:56] which has the same confs [00:59:00] same hadoop confs [00:59:04] but also has /etc/oozie/conf [01:00:06] hmmm. [01:00:21] i'm pretty sure this isn't an oozie problem though [01:00:31] i thougth it was at first [01:00:33] and to test [01:00:44] I manually added an /etc/hosts entry for analytcis1010.wikimedia.org on an27 [01:00:55] oozie submitted the job just fine then [01:01:09] but when hadoop tried to run the job, I got the same error from one of the job containers on one of the nodemanager/datanodes [01:01:18] so this is def a more global thing [01:02:57] lol [01:03:01] i have a terrible question [01:03:10] when confs are changed, what do you restart? [01:03:26] all hadoop services [01:03:37] namenode, resourcemanager, datanode, nodemanager [01:03:53] well there are more hadoop services, history server, don't thikn I restart those [01:04:12] ...do you restart oozie? [01:04:18] * dschoon ducks [01:04:22] https://github.com/wikimedia/kraken/blob/master/bin/khadoop [01:04:25] yes [01:04:36] yeah I restart oozie [01:04:41] not with the khadoop script [01:04:52] i'm 95% sure this is not an oozie problem [01:05:01] i guess I could run a pig script, entirely out of oozie [01:05:04] and see if I get the same problem [01:05:05] interesting. [01:05:06] okay. [01:05:16] so let's list theories: [01:05:25] - actual, honest-to-god DNS problem [01:05:34] average_drifter: hey, I'm back [01:05:45] (not totally crazy because it'd be outside our control, and it could appear without us doing anything) [01:05:46] bar chart looks good [01:05:47] oh shit! [01:06:04] ottomata: any chance you've heard anything recently from mark about the network ACLs? [01:06:27] like, say, that we're no longer in the eqiad DNS zone? [01:07:02] have not heard that, but getCanonicalHostName still resloves to .eqiad [01:07:13] hmm. [01:07:18] shouldn't it be .eqiad.wmnet? [01:07:35] yes yes [01:07:41] i just didn't type that out :) [01:07:56] kk [01:07:57] ok before I re-add mapreduce.jobtracker.address to the configs [01:08:02] i'm going to try running a pig script [01:08:03] hmm. [01:08:04] let's see.. [01:08:45] erosen: ah cool :) [01:09:07] average_drifter: do you have the charts sitting around somethwere? or did I miss that form earlier? [01:10:41] erosen: the script is running now, so they'll be available in the next hours [01:10:48] cool [01:11:42] question [01:12:38] ottomata: why is analytics1010.wikimedia.org hardcoded into your job conf? [01:12:39] https://github.com/wikimedia/kraken/blob/master/oozie/webrequest_mobile_country_device_vendor/coordinator.properties#L3 [01:12:42] ... [01:12:51] ......... [01:12:54] ….......>>>>>.... [01:12:57] ... [01:12:58] .. [01:12:59] ...... [01:12:59] . [01:13:03] [01:13:08] so! [01:13:17] this is the part where we all pretend this never happened [01:13:18] The program 'mobile_report.py' received an X Window System error. [01:13:19] This probably reflects a bug in the program. [01:13:19] The error was 'BadAlloc (insufficient resources for operation)'. [01:13:19] (Details: serial 279515 error_code 11 request_code 53 minor_code 0) [01:13:21] (Note to programmers: normally, X errors are reported asynchronously; [01:13:23] must be a copy paste error [01:13:24] AGGHGHGHGHGHG [01:13:24] that is, you will receive the error a while after causing it. [01:13:27] To debug your program, run it with the --sync command line [01:13:29] option to change this behavior. You can then get a meaningful [01:13:30] and the problem was actually really subtle and impressive [01:13:32] backtrace from your debugger if you break on the gdk_x_error() function.) [01:13:33] average_drifter: i've seen that [01:13:38] erosen: hmm [01:13:45] that is because you need to set the backend to something which doesn't try to display something [01:13:47] one sec [01:13:51] * erosen digs for code [01:13:55] erosen: ok [01:13:59] (i bet i know the origin -- when we switched an01 -> an10; an01 had a real .wikimedia.org address) [01:16:23] aghhhh eyah i copied this from died's stuffa ggghh [01:16:25] thank youuuuu [01:16:27] i think nobody wants to admit what % of "debugging" time is shit like this [01:16:36] i know i do this ALL THE TIME [01:16:39] average_drifter: http://matplotlib.org/users/customizing.html [01:16:39] god save pair programming [01:16:51] backend : PDF [01:18:15] on the plus side [01:18:19] you were totally right, ottomata [01:18:23] it wasn't oozie :) [01:18:49] haha [01:18:58] well it kinda was, it wasn't oozie configs [01:19:05] but, weiiiird that I didn't see this error utnil I upgraded cdh4 [01:23:05] interesting. [01:23:09] i have a theory [01:23:17] the typo was in coordinator.properties [01:23:38] but ${jobTracker} isn't used in coordinator.xml [01:23:46] only in workflow.xml [01:24:36] maybe jobTracker and nameNode have defaults, and coordinator.properties wasn't loaded for workflow.xml interpolation until the upgrade? [01:25:25] average_drifter: or import matplotlib [01:25:26] matplotlib.use('PDF') [01:25:35] [travis-ci] master/04b2381 (#86 by Andrew Otto): The build has errored. http://travis-ci.org/wikimedia/kraken/builds/5488226 [01:28:38] how's it look now, ottomata? [01:33:40] the workflow ran! [01:33:52] i'm trying to see if I can edit the device coordinator job properties [01:34:40] edit? [01:34:44] oh, via the UI? [01:37:32] via mysql :p [01:37:35] actually [01:37:59] i really wanted to know if workflow confs get recreated from coordinator properties when you rerun a job [01:38:00] they do! [01:38:01] so that's good [01:38:08] heh [01:38:13] and I edited the coordinator conf in mysql :p [01:38:14] so [01:38:16] ah, good [01:38:17] should be good [01:38:25] i don't have to go and edit all of the pending workflows [01:46:43] couldn't you just cancel them all? [01:46:47] and resubmit? [01:47:15] i think that would make it rerun all of the jobs, even for the ones that have alrady succeeded [01:49:30] grrr, I think I do have to update all the workflows, crap, they are coordinator actions [01:49:33] aghhhh [01:49:40] hm. what to do... [01:49:55] i could try to submit a new coordinator, many of the tasks will die due to existing data [01:49:57] i guess that is ok? [01:50:05] hm [01:50:42] i woudln't mind updating the individual workflow confs, except that the confi s stored as text xml [01:50:50] not individual fields [02:03:00] lmk what you decide to do [02:03:06] i realize i kinda skipped lunch [02:03:12] and i am getting light-headed, heh [02:04:56] i dunno, i'm trying to update them now, unngh [02:05:01] i need to peace out again real soon [02:05:04] going to give this 20 more mins [02:05:11] aiight [02:05:16] i'll probably get into it after dinner [02:05:37] could you send a progress update email to the group? [02:05:49] sure [02:05:53] this blocker was a Source Of Worry earlier, so it'd be nice to say we figured it out [02:06:07] (or at least, took another step toward figuring it out) [02:07:10] wtf, now all the coord actions are updated, and I didn't change them [02:07:18] hm! maybe oozie got smart after a while [02:07:18] HMM [02:09:54] ahh, no it didn't, doh [02:10:41] oh god now i'm kicking myself because ottomata said like a while back - why the hell is it hitting analytics1010.wikimedia.org, that setting's not showing up anywhere [02:10:56] i knowwww! [02:10:59] and I saw that and failed to say anything [02:11:00] ugh [02:11:02] so much slef kicking right now [02:11:04] awww [02:11:08] team self kick man [02:11:14] :) [02:11:18] yay for david [02:11:39] i'd do some sort of self-aggrandizing backpatting [02:11:40] also - dschoon- that's why you always get sick brother [02:11:43] gotta eat man! [02:11:45] EAT FOOD [02:11:52] but i've done that WAY too often myself [02:36:46] heading out [02:38:26] drdee: i'm almost done with some questions/clarifications for the mobile features [02:38:35] i'll send after eating :) [02:38:44] back in a bit [05:17:11] [travis-ci] master/d738fab (#87 by Diederik van Liere): The build has errored. http://travis-ci.org/wikimedia/kraken/builds/5491263 [13:34:20] morning [13:34:42] morning drdee [13:34:56] any news on the iOS user agent string for mobile apps?, maybe you can add it to http://www.mediawiki.org/wiki/Mobile/User_agents [13:35:15] drdee: I responded to you yesterday :) [13:35:20] drdee: but you disconnected righta fter, I think [13:35:21] let me find it [13:35:27] oh yes, I'll just add it there [13:35:27] oh ughh [13:35:31] thx! [13:36:45] drdee: Android is accurate, and I'm not sure if the claims about iOS are right or wrong [13:36:47] let me verify [13:41:43] drdee: I *think* iOS should also be setting it properly [13:41:44] (the UA) [13:41:53] drdee: I think that that bug is only for *master*, not the last released version [13:42:09] drdee: can you grep the logs for 'WikipediaMobile/3.2.1' and see if iOS UAs turn up? [13:42:10] they should [13:43:29] drdee: oh, nevermind. [13:43:35] drdee: so, iOS app is not setting any UA :| [13:43:43] I should shoot brion when I see him, I think? [13:44:14] drdee: I don't think there's an easy way to pick up iOS app stuff now :| [13:47:19] ok good to now [13:47:32] and old iOS stuff, how do we pick that up? [13:51:48] drdee: until 3.2.1, you can pick that up via 'WikipediaMobile/3.2.1' prefix [13:51:55] to the UA [13:52:16] drdee: that page says you can still detect it because it lacks the 'Safari' tag, but that's going to catch a few other apps too [13:52:20] ok [13:52:29] drdee: I'll poke brion into setting out a release for this [13:52:32] can you update the wiki accordingly [13:52:36] ? [13:52:38] ok [13:52:55] drdee: oh, it is up to date, I tihnk [13:53:17] just added exact version info [13:53:32] coffee shop's closing down, am out. bb [13:53:33] brb [13:53:41] cool [14:08:12] drdee: hi [14:13:00] hi [14:13:07] charts ready? [14:13:19] drdee: yes they are [14:13:37] show me :) [14:20:50] drdee: check e-mail please [14:21:52] ty [14:28:12] morning [14:28:20] monring [14:30:20] I have an idea [14:30:25] but it's crazy [14:30:31] can I write it even though it's really crazy ? [14:31:19] yes [14:31:31] ok, here goes [14:32:21] the english language is spoken in a few major parts of the world, which means most pageviews come from those places [14:32:43] * milimetric braces himself [14:32:44] U.S. , UK, australia, new zealand, [14:33:01] now, each of those countries have different timezones, let's take UK [14:33:06] it has only one(I think) [14:33:28] that means pageviews align in a periodic pattern on the time-series [14:33:35] day-night-day-night-day-night... [14:33:51] that means we have a small wave [14:34:09] when counting pageviews for the english language, they might come from many parts of the world [14:34:23] let's study the charts first :) [14:34:25] but all of these parts have different waves like this. and those waves are shifted depending on timezone [14:34:43] right, with you so far [14:34:48] but this is exactly about the charts :) [14:34:52] the mobile page views chart looks like an ocean [14:35:03] so here comes the weird idea => [14:35:04] (the one broken down by countries) [14:35:25] ... [14:35:30] we can do Fourier analysis on those things, and find out the different waves that compose the big time-series of pageviews [14:35:38] i object [14:35:38] * average_drifter shuts up [14:35:40] sorry [14:35:41] hehe [14:35:46] drdee: please object [14:35:55] (sheepishly) I actually did that a while back for fun [14:43:03] hahaha YES [14:43:13] dschoon: yes? [14:43:40] i am of the firm belief that if you have a hard problem, the application of an FFT can only improve things [14:43:50] drdee: Evan suggested that we now go for combinations of features that show spikes on 14-december-2012 [14:44:01] yes that makes sense [14:44:07] yes [14:44:16] and if that somehow doesn't help, blindly applying SVD comes next [14:44:29] finally, as a last resort, one might try to understand the problem [14:44:31]