[03:44:54] drdee: you shouldn't be here, but if you are, do you have any ideas for how to copy over the zero logs to kraken now that the /a mount on an01 is gone? [03:45:22] where are they now, erosen? [03:45:27] stat1 [03:46:06] i was copying them over, but I used up all the space on the root mount [03:46:11] do you need to do it tonight? [03:46:16] heh [03:46:17] ….sort of [03:46:18] right. [03:46:21] uh. [03:46:23] hm. [03:46:25] that complicates things. [03:46:30] i was going to say "rsync" [03:46:38] i can write a bash script to do them one by one and put them on hdfs [03:46:46] actually [03:46:55] i think i might be able to manage by only copy the latest files [03:47:03] I forgot that most of the files are already on hdfs [03:47:21] I'm hoping that solves things. thanks for the concern [03:47:25] i think the reason there's no space on / is because drdee/drdee_ saved many gigabytes of raw udp2log output to disk when debugging last week [03:47:32] ah. [03:47:51] but an01 got wiped i though? [03:47:57] well, if you end up with issues on that front, i recommend sending an email asking people to remove any temporary files they have on an01 [03:48:00] and i just discovered that pig is not installed... [03:48:01] i don't think it's been reimaged yet [03:48:05] oh. [03:48:08] maybe i'm wrong then [03:48:16] interesting [03:48:41] yeah, it has. [03:48:46] judging from the fact that its host key changed [03:48:47] i think yes [03:48:50] maybe my path is messed up [03:49:09] yeah definitely was reimaged or something like that [03:49:45] yes. [03:49:47] also: [03:49:55] Filesystem Size Used Avail Use% Mounted on [03:49:55] /dev/md3 19G 1.6G 16G 9% / [03:50:04] yeah [03:50:07] so either way, the problem should not exist. [03:50:09] i deleted everything [03:50:11] it was at 100% [03:50:15] ah. [03:50:17] well then. [03:50:25] but I should be able to manage with 16G [03:50:32] or 17.4 [03:50:43] but [03:50:48] the bigger issue is that I haven't found pig yet [03:50:49] if you need the hdfs CLI tools [03:50:55] use another node. [03:51:08] that was going to be my question [03:51:13] do they have public ips? [03:51:16] no. [03:51:27] use an01 as a bastion. it still works for that. [03:51:31] how do login from an01? [03:51:39] just ssh analytics1010 or something [03:51:42] yes. [03:51:57] hmm [03:51:57] if you've already got that set up in .ssh/config, you're fine [03:52:01] if not: [03:52:16] i think not in my case [03:53:20] grr. [03:53:24] it's not on our access page. [03:53:27] you want: [03:54:44] --> gchat, easier to paste big blocks [03:57:55] lmk if you have problems [03:59:19] erosen ssh bast1001.wikimedia.org [03:59:28] and then go to any analytics100* node [03:59:37] an01 is not fully back online as you noticed [03:59:46] gotcha [03:59:47] thanks [03:59:58] probably you will have to fix your known_hosts file [04:00:27] yeah did that for an01 [04:00:39] and used it as a bastion [04:00:48] but I'll use the official bastion instead [04:00:55] ah [04:00:57] good to know. [04:00:59] i'll switch also. [04:01:35] i'm getting public key denied on bast1001 [04:01:38] bast1001.wikimedia.org is the new and preferred way [04:01:43] not an issue at the moment, though [04:02:09] that sucks :( [04:02:57] i'll talk to andrew tomorro [04:02:58] pig runs on analytics1002 for sure [04:03:03] i used it today [04:03:05] thanks for help at this late hour [04:03:22] late is a relative term [04:03:23] great, yeah I just tested it on an10 [04:03:27] yeah [04:03:29] true [04:03:43] i just did the math and realized it's prime coding time [04:03:48] :) [04:03:48] maybe not even prime, yet [04:03:49] not for me [04:03:55] hehe [04:04:00] can't way for hue to come back [04:04:13] so we can finalize the zero script and have it run every 15 minutes :D [04:04:18] and then hook it up to Limn [04:04:22] and make Amit happy [04:04:25] milimetric: sorry to bug you at what is actually a late hour for you, but can you send me that see command for fixing the limn_data repo? [04:04:39] indeed [04:04:42] that will be awesome [04:04:57] i'm sadly working on getting the old zero method to work [04:05:10] did you look at the gist? [04:05:16] yeah [04:05:19] either maxmind db is broken [04:05:22] but that is for x-cs right? [04:05:24] yes [04:05:27] I'm like zombie [04:05:30] oh the sed thing? [04:05:52] erosen, no prob so you're working on a different dashboard or same? [04:05:56] same [04:06:08] i'd just like to update the data a bit tonight and see how it looks [04:06:13] it's not essential [04:06:40] sed -i 's/\/data\/datafiles/\/data\/datafiles\/gp/' *.yaml [04:06:51] awesome thanks [04:07:04] that's in the datasources directory [04:07:40] another way you could do it is make a patch from the uncommitted stuff on kripke/gp-dev and apply that [04:09:44] interesting idea [04:09:55] i'll keep it as a shell script for now, but that would be some git fun [04:10:06] i'm off to a house meeting! [04:10:14] see y'all tomorrow morn [04:10:26] thanks for the help [04:14:35] no prob - sorry - absorbed in stacking bars :) [14:56:05] hey guys - i've been around, just absorbed in metrics meeting last minute requests [14:56:18] ottomata: drdee not sure if there is a plan for hue to come back up soon, but let me know if it does--it would be handy [14:56:52] i'm working on it [14:56:57] have to get stuff through ops :/ [14:56:58] getting there [14:57:14] ottomata: i figured, not trying to add priority, just saying let me know [14:57:16] is mettics meeting today? [14:57:19] cool [14:57:19] ya [14:57:25] why no calendar say? [14:57:30] hmm [14:57:32] not sure [14:57:34] thon is very bad at calendar [14:57:54] thon not understand time [14:58:01] thon thinks there is no time [14:58:13] thin make own event at wrong time, but thin still find thon's own calendar event useful [14:59:01] somehow colloquy autocorrect seems to be more extreme than other programs [15:00:35] oh ottomata, another q: the contents of an01 home are really gone right? [15:00:41] oh [15:00:42] i have them [15:00:44] backed up [15:00:45] oooo [15:00:46] haven't restored them yet [15:00:50] i can do that now if you want [15:01:01] that would be INVALUABLE [15:01:22] turns out I had some non-checked in code there... [15:01:26] which I wrote again last night [15:01:43] but isn't fully debugged ;/ [15:01:44] yeah, on it now [15:01:51] thanks a bunch [15:17:03] morning guys [15:17:14] aye [15:17:15] milimetric the http://test-reportcard.wmflabs.org/graphs/active_uploaders_commons_target has two missing observations [15:19:29] erosen [15:19:32] /home restored [15:19:44] yay [15:19:45] thanks [15:59:30] drdee: what is the best way you've found to monitor pig jobs from the command line? [16:04:33] erosen [16:04:34] maybe try [16:04:35] mapred job [16:04:47] k [16:04:52] i've done that [16:04:54] hm [16:05:02] but it doesn't give details on number of mappers reducers [16:05:14] but that seemed like the best option from the internets [16:05:41] i think I've mentioned this before, but sometimes I just use lynx on an analytics machine to browse the status pages [16:05:50] but I can't seem to find a machine with lynx installed [16:07:04] oh I can install it if you like [16:07:06] also, erosen [16:07:12] you can use ssh tunnel to access a service [16:07:16] if you want hue [16:07:19] interesting... [16:07:31] ssh -v bast1001.wikimedia.org -L 8081:analytics1027.eqiad.wmnet:8888 [16:07:45] replace the last bit with whatever host:port you want [16:07:48] then navigate to [16:07:51] localhost:8081 [16:09:06] k, will try later today [16:09:13] i think things are working atm [16:25:36] btw, ottomata mapped job is what I needed, I just hadn't looked at the status option [16:25:42] mapred* [16:39:27] cool! [16:56:11] wrote simple sessionize pig udf, it reads the seed value from hdfs, time to test this baby! [16:56:25] yeehaw [16:56:47] i am cranking out pig udf's like crazy! [16:56:47] i'm at the waiting for review part of an01, going to see if i can get the concat_sort thing running for logloss [16:56:57] k [16:57:25] ottomata, can you also review my patch to filter ? [16:57:59] fiiiilter [16:57:59] where? [17:07:00] gerrit [17:07:05] 1 sec [17:07:18] https://gerrit.wikimedia.org/r/#/c/47827/ [17:11:35] the gerrit build fails because of the outdated jenkins config [17:13:19] looks fine to me drdee_, as longs as it works [17:13:28] but, replace_space is just more for filter to do [17:13:44] please don't beat me over this, say it to EZ [17:13:53] i think it's insane to fix in the first place [17:14:10] i agree, this might be break the whole thing [17:14:52] but apparently we care a lot about fixing literally a handful of URL's that are not even real page views as they come from Googlebot and are 301 redirects [17:15:34] what if we just made tab the separator in the filter output? [17:16:05] no we can't because that will break all the 3rd party scrips that are using dumps.wikimedia.org [17:18:02] hm, k [18:00:14] STANDUP! https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [18:02:48] that happening? [18:29:38] oh, drdee [18:29:49] aight [18:29:55] fwiw, i found the line in puppet where they define the nginx access_log format: [18:29:56] https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=blob;f=templates/nginx/nginx.conf.erb;h=75c55cde5e9168e57e6bd6f54aaa251ba75f9e86;hb=HEAD#l31 [18:32:35] you have a link to one of those busted lines again? [18:33:11] in the email [18:51:51] bb in 1h [20:16:01] aiight [20:16:04] lunch, then cubes [20:28:46] ung, pig error messages are so cryptic sometimes [20:42:36] what's up ottomata? [20:42:43] dschoon wanna talk about the reviews next week? [20:47:23] oh, pig says I'm casting something weird, but it won't tell me where, since the exception is thrown in java [20:47:43] kinda like dschoon debugging javascript error messages in coco code, except worse! [20:47:58] what helps [20:48:04] is going to grunt [20:48:11] and then do ILLUSTRATE [20:48:22] to inspect your data and datatypes [20:48:52] HMMMMmmm [20:52:11] drdee_, illustrate causes the exception too :/ [20:52:11] java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double [20:52:34] oh [20:52:36] maybehmm [20:52:45] but that's a clear error message [20:53:31] well, yeah [20:53:37] but it doesn't tell me where in my script i'm doing that [20:53:42] i can't see anywhere i'm doing it [20:53:48] maybe when I'm adding a literal + 1? [21:04:31] hangout and screen share? [21:04:42] meh, might not care [21:04:48] i was trying to import the sequence number as a double [21:04:56] cause I imagine it could get real high [21:05:04] if I leave it as a long no problems [21:05:15] i'd rather not deal with that more than I have already rigiht now [21:06:10] i think long is big enough [21:06:14] 2^63 [21:06:17] oh its 64? [21:06:22] yeah duh guess it is [21:06:37] iirc it's also a long in udp2log [21:06:39] you think int is 64 as well? [21:06:41] ok ok ok [21:06:41] cool [21:06:45] yeah i'm sure that is big enough [21:06:49] ok thanks goodbye double leave me alone [21:07:02] so, the import mostly works, i just don't have it working smoothly enough to try in oozie [21:07:18] i ask question [21:07:19] https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/ggDLy5P63Y0 [21:07:20] hope for answer [21:09:39] i asked https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/3oZPkZympW4 [21:10:27] hm [21:22:03] dschoon wanna talk about the reviews next week? [21:23:56] hellohello [21:23:58] i return! [21:24:13] lunch with Real People always takes so long :( [21:24:22] ottomata, drdee, milimetric -- cubez? [21:24:30] js meeting [21:24:33] you said you were gonna join [21:24:37] no cubez for me thank youuuu [21:24:42] i am game [21:24:46] orite [21:24:59] i think we should chat cubes while everybody is around tho [21:25:16] so didn't you wanna crash the JS party/ [21:25:18] i started putting together a wiki page with my notes on it earlier today [21:25:20] i *do* [21:25:29] but that will go on forever :) [21:25:47] i think this is more important [21:26:54] drdee_, milimetric -- how about in 10m? [21:27:10] i am game [21:27:17] (and ottomata, in case he was being facetious) [21:27:28] i don't think he cares [21:27:32] ja [21:27:42] gonna restart to see if it fixes my hangout woes. [21:27:43] brb [21:28:27] so drdee_, the reason I'm attending the JS meetings is to be less invisible to the rest of the org [21:28:35] i can talk cubes if you guys want [21:28:51] i thinkit's good that you attend [21:28:53] but I think we should review the user metrics API Very carefully before we talk [21:29:57] btw, i'm going to start hacking on the metrics api (which is going by UMAPI for user metrics api) [21:31:37] mk, posting notes. [21:37:49] erosen_ what are you planning on doing with UMAPI? [21:38:10] I feel like we should sort of coordinate efforts here - come up with some goals and dole out tasks [21:38:16] making it work with publicly provided user name lists and cohort definitions [21:38:39] basically merge it with userstats (github.com/embr/userstats) [21:39:00] i might also set it up so that it can run without a db and use the api [21:50:52] https://www.mediawiki.org/wiki/Analytics/Kraken/Query_Service [21:50:58] milimetric, drdee, ottomata ^^ [21:51:20] i'm eager to talk whenever you guys are free [21:51:33] cool, i'll hop off this in 5-10 min. [21:51:42] just finishing up reading Ryan's github page [21:51:53] focusing on this perspective: http://etherpad.wikimedia.org/a4NCijsjQp [21:53:55] kk [21:54:21] link to the github page? [21:54:35] https://github.com/rfaulkner/E3_analysis [21:54:36] i think [21:54:49] that is what dan is talking about [21:54:53] but not sure [21:55:43] but this is what you probably want: http://stat1.wikimedia.org/rfaulk/pydocs/_build/ [21:59:12] mk [22:00:06] (the link to the github was in the etherpad) [22:00:30] The docs are great, the source to the metrics is what I was paying close attention to [22:00:37] trying to figure how to best fall in line with this [22:00:41] ok, I'm ready totalk [22:00:44] *to talk [22:01:01] dschoon, drdee_, erosen_, at your convenience ^ [22:01:10] me too [22:01:13] woo [22:01:27] https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [22:02:22] lemee grab a room [22:51:41] :) [22:51:48] que pasa? [22:52:02] nuttin [22:52:06] just wanted to wave