[00:22:50] <average_1rifter>	 drdee: ready for new patch sewt
[00:22:53] <average_1rifter>	 *set
[00:29:50] <average_1rifter>	 drdee: please review https://gerrit.wikimedia.org/r/#/c/32137/
[00:29:56] <average_1rifter>	 when you have time
[00:31:19] <louisdang>	 average_1rifter, I think he's getting dinner now. He said he'll be back later.
[00:36:40] <average_1rifter>	 louisdang: no problemo, I'm going to be here for a while
[00:36:45] <average_1rifter>	 louisdang: thanks for telling me
[00:36:53] <louisdang>	 average_1rifter, np
[00:57:02] <drdee>	 getting some coffee now, then i;l do the review and then some groceries :)
[00:57:14] <average_1rifter>	 drdee: nice, thanks :)
[01:32:32] <drdee>	 average_1rifter, can you make me the maintainer in the change log for all versions so that we can sign it using PGP?
[01:35:43] <average_1rifter>	 yes
[01:35:48] <average_1rifter>	 drdee: yes
[02:38:32] <drdee>	 average_1rifter: that's the only comment I have, after fixing that ill merge it
[02:40:13] <average_1rifter>	 drdee: yes, just refactoring debianize.sh to accept parameters
[02:40:21] <average_1rifter>	 so I can have something like
[02:40:22] <average_1rifter>	 sorry, I meant git2deblogs
[02:40:23] <average_1rifter>	 so I can have something like
[02:40:36] <drdee>	 cool
[02:40:44] <average_1rifter>	 ./git2deblogs --generate --force-maintainer-name="..." --force-maintainer-email="..."
[02:43:14] <drdee>	 awesome!
[02:44:42] <drdee>	 i am still confused about the PLACEHOLDERS in configure
[02:45:04] <drdee>	 when are those replaced?
[02:45:08] <average_1rifter>	 so basically there are two directories
[02:45:10] <average_1rifter>	 udp-filters
[02:45:11] <average_1rifter>	 and
[02:45:19] <average_1rifter>	 udp-filters-0.3.etc
[02:45:21] <average_1rifter>	 depending on the version
[02:45:28] <average_1rifter>	 now, udp-filters is the one in which we develop
[02:45:37] <average_1rifter>	 and udp-filters-0.3...
[02:45:47] <average_1rifter>	 is the one in which dpkg-buildpackage is built
[02:46:02] <average_1rifter>	 all replacements are made in    udp-filters-0.3..
[02:46:58] <average_1rifter>	 so inside debianize.sh, it will change directory to udp-filters-0.3..
[02:47:05] <average_1rifter>	 and it will run some oneliners to replace versions
[02:47:12] <average_1rifter>	 afterwards
[02:47:26] <average_1rifter>	 so one of the versions it will replace is in configure.ac
[02:47:46] <average_1rifter>	 then dpkg-buildpackage will at some point run compile.sh
[02:47:53] <average_1rifter>	 and compile.sh will run autoconf
[02:47:58] <average_1rifter>	 now that autoconf takes stuff from configure.ac and creates configure
[02:48:10] <average_1rifter>	 so the version which was replaced by debianize.sh is now in configure too
[02:48:17] <average_1rifter>	 but only in the udp-filters-0.3..
[02:48:45] <average_1rifter>	 the same goes for src/udp-filters.c
[02:48:51] <average_1rifter>	 src/udp-filter.c I mean
[02:49:13] <average_1rifter>	 where the version is being replaced directly by debianize.sh
[02:49:51] <average_1rifter>	 that version in configure.ac is important because I think it will be passed to help2man and the manpage generated contains it
[02:50:43] <average_1rifter>	 the only reason you see PLACEHOLDERS inside configure in the repo right now, is because I ran compile.sh manually inside udp-filters
[02:50:57] <average_1rifter>	 which is the development directory
[02:52:05] <average_1rifter>	 so compile.sh from udp-filters development directory ===>  autoconf ====> generates configure with "PLACEHOLDER" string.. literally inside configure
[02:57:39] <drdee>	 thx :)
[13:47:47] <milimetric>	 good morning!!!
[13:58:01] <drdee>	 yoyoyoyoyo
[13:58:07] <drdee>	 good morning milimetric
[13:58:39] <milimetric>	 woooooo!
[13:58:41] <milimetric>	 :)
[13:58:54] <milimetric>	 two storms down, no basement flooding yet
[14:03:44] <drdee>	 you are on a roll!
[14:22:46] <drdee>	 yo ottomata
[14:23:21] <drdee>	 so i debugged with louisdang a sqoop job that runs using oozie inside hue :)
[14:23:31] <drdee>	 it works.
[14:28:40] <ottomata>	 yooooo
[14:28:48] <ottomata>	 yeah?
[14:28:57] <ottomata>	 any luck?
[14:29:10] <drdee>	 it works
[14:29:31] <drdee>	 it was non-trivial as there are a lot of parts that all need to be synced up
[14:29:52] <drdee>	 but we can now create a workflow that would import data from enwiki on a weekly basis
[14:30:05] <drdee>	 and run it automatically like a cronjob
[14:30:31] <drdee>	 you should take a look at http://incubator.apache.org/ambari/
[14:30:55] <drdee>	 might be useful, it's puppet stuff for deploying, monitoring and configuring hadoop clusters
[14:32:08] <ottomata>	 its puppet?
[14:32:41] <drdee>	 yes
[14:35:55] <ottomata>	 wow looks thourough
[14:36:02] <ottomata>	 i'm just skimming through some puppet manifests
[14:36:18] <ottomata>	 poorly documentd, geez, why don't people document things anymore? (or ever?)
[14:36:49] <drdee>	 :)
[14:37:01] <drdee>	 brb coffee and then oozie time!
[14:37:16] <ottomata>	 there's something similar that dschoon liked, can't remember its name though
[16:01:39] <sumanah>	 ottomata: http://lists.vireo.org/pipermail/debiannyc/2012-November/001794.html I just posted this and got about 4 responses offlist. Would you be interested in hearing about them? one of them is free: http://www.wixlounge.com/
[16:03:37] <ottomata>	 hmm, wix looks cool
[16:03:41] <ottomata>	 we should go check it out someday!
[16:04:27] <sumanah>	 yeah!
[16:04:37] <sumanah>	 if you can GET to Manhattan. can you?
[16:04:54] <sumanah>	 for all I know you live in a yurt right now with only a laptop strapped to your back and as much as you can carry on a bike
[16:06:24] <sumanah>	 oh one more thing ottomata would you like to see http://www.telecharge.com/BehindTheCurtain.aspx?prodid=9238 with me sometime?
[16:06:41] <sumanah>	 it has Aasif Mandvi from the Daily Show
[16:07:24] <ottomata>	 sounds kinda interesting and intense, hmmm, mayyyyybe
[16:07:36] <ottomata>	 i'm out of town til after thanksgiving
[16:07:37] <ottomata>	 HA!
[16:07:38] <ottomata>	 AND
[16:07:44] <ottomata>	 I might spend a night or two in a yurt!
[16:08:36] <sumanah>	 ottomata: cool! I spent a night in a Mongolian-style yurt while walking across England this summer and it was neat-o burrito.
[16:09:12] <drdee>	 ottomata, see also https://app.asana.com/0/828917834272/1055914021119 for storm
[16:12:59] <drdee>	 morning average_drifter, how is the debian package doing?
[17:29:18] <drdee>	 ottomata, quick question: why do the hive.stats.* properties from  http://analytics1001.wikimedia.org:8888/beeswax/configuration not match /etc/hive/conf/hive-site.xml ?
[17:30:25] <ottomata>	 hm, i dunno, i suppose they should huh?
[17:30:40] <drdee>	 i would think so :D
[17:31:01] <drdee>	 i am trying to get hive metrics written to msyql as you can imagine
[17:31:05] <drdee>	 and nothing is written atm
[17:31:22] <drdee>	 or do i need to restart hue?
[17:31:49] <ottomata>	 ich weiss nicht, mein freund!
[17:31:54] <drdee>	 i'll try
[17:31:58] <ottomata>	 but let us see
[17:31:59] <ottomata>	 ok
[17:32:24] <drdee>	 no restarting hue does not solve the issue either
[17:33:04] <ottomata>	 you don't have metrics thigns in hive-site.xml
[17:33:04] <ottomata>	 do you?
[17:33:45] <drdee>	 i am just looking as well
[17:33:54] * drdee  is flabbergasted

[17:34:09] <ottomata>	 is puppet running by default?!
[17:34:21] <drdee>	 at least that explains why it doesn't show the properties :)
[17:34:25] <drdee>	 dunno
[17:34:58] <ottomata>	 why yes I think it is!  It shoudl be running the wmf puppet, not kraken
[17:34:59] <ottomata>	 grrr
[17:35:31] <ottomata>	 ok stopped it
[17:35:36] <ottomata>	 edit away :)
[17:38:29] <drdee>	 ty
[17:47:23] <ottomata>	 haha, drdee:
[17:47:24] <ottomata>	 https://github.com/jordansissel/fpm
[17:48:26] <drdee>	 looks cool
[17:48:55] <ottomata>	 yeah, this is using it
[17:49:02] <ottomata>	 https://github.com/wmf-analytics/storm-deb-packaging
[17:49:05] <ottomata>	 (i forked)
[17:49:09] <ottomata>	 gonna chnage some versions and such
[17:49:10] <ottomata>	 looks nice!
[17:49:11] <drdee>	 k
[17:49:12] <ottomata>	 and easy!
[17:49:23] <ottomata>	 presentation too:
[17:49:23] <ottomata>	 https://docs.google.com/present/view?id=0Aa9liCTsAyzRZGNtd3dkOTRfMTdmczY2azlkcg
[17:49:23] <drdee>	 still no hive metrics in mysql btw
[17:49:37] <drdee>	 also no errors to be found
[17:50:54] <ottomata>	 yeah, that's how far I got when I tried it out
[17:56:38] <drdee>	 https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90
[18:01:47] <drdee>	 ottomata, milimetric ^^
[18:02:51] <ottomata>	 trying
[18:32:45] <louisdang>	 hi drdee, got anything for me?
[18:34:06] <drdee>	 yeah let's do oozie stuff!
[18:34:24] <drdee>	 how about creating a serious workflow that would import a number of tables from wiki?
[18:34:32] <louisdang>	 sure
[18:43:11] <drdee>	 so louisdang let's create an ether pad with what needs to happen:
[18:43:12] <drdee>	 http://etherpad.wikimedia.org/oozie
[18:43:26] <louisdang>	 ok
[18:53:55] <drdee>	 louisdang: i left you some question on the etherpad doc
[18:54:07] <louisdang>	 drdee, responded
[18:54:48] <drdee>	 thx
[18:57:24] <drdee>	 louisdang: do you have enough information to get started?
[18:57:33] <louisdang>	 drdee, one more question on etherpad
[18:57:38] <drdee>	 ok
[18:59:17] <drdee>	 otstormata is storm nice to you?
[19:04:16] <ottomata>	 yes at least packaging it is!
[19:04:25] <ottomata>	 i need to change a few of the storm-package-deb settings
[19:04:28] <ottomata>	 which is why I forked it
[19:04:34] <ottomata>	 that repo is so nice
[19:04:39] <ottomata>	 so much thanks to that guy
[19:04:39] <drdee>	 very cool
[19:04:50] <drdee>	 nathan?
[19:14:48] <ottomata>	 no
[19:15:01] <ottomata>	 https://github.com/phobos182/storm-deb-packaging
[19:15:03] <ottomata>	 phobos182?
[19:15:04] <ottomata>	 dunno
[19:18:12] <average_drifter>	 drdee: pong pong
[19:18:21] <drdee>	 hey!
[19:18:30] <drdee>	 you were so quiet today :)
[19:18:59] <average_drifter>	 drdee: oh sorry
[19:19:00] <average_drifter>	 drdee: https://gerrit.wikimedia.org/r/#/c/32137/
[19:19:02] <average_drifter>	 drdee: new patch set
[19:19:13] <drdee>	 that's what i am used to :)
[19:20:10] <drdee>	 merged
[19:22:44] <average_drifter>	 :)
[19:43:07] <drdee>	 you can ask ottomata to install the new udp-filters package on stat1
[19:43:25] <drdee>	 then we can geocode all the existing sampled traffic files
[19:44:23] <RobH>	 fyi: analytics1011-1022 are racked, im working on flashing the drac on them so they have remote access
[19:44:29] <RobH>	 its slow though, will take most of afternoon.
[19:46:12] <average_drifter>	 yep
[19:46:30] <drdee>	 WOOT WOOT, thanks so much RobH
[19:54:28] <RobH>	 drdee: so i would normally discuss the tech stuff with andrew, but he isnt about
[19:54:37] <RobH>	 so when he is, will update him as well
[19:54:46] <RobH>	 but im not partitioning these, just getting them remotely manageable for him
[19:55:01] <drdee>	 ok
[19:56:03] <RobH>	 since the old disk layout isnt going to work, you guys hvae 4 more 2tb disks than the old servers had =]
[20:04:16] <drdee>	 another 4? per server?
[20:05:02] * drdee  is thinking that the analytics should wear t-shirts of Cookie Monster eating hard-drives

[20:11:22] <average_drifter>	 I'm out to get some stuff, bb in ~1h
[20:11:55] <drdee>	 k
[20:12:14] <average_drifter>	 btw, is there any free/cheap sms sending service with a public API that I could use ? I need to write something to hook into irssi so I can get texted on my phone
[20:12:39] <drdee>	 maybe whatsapp
[20:12:54] <average_drifter>	 ok, thanks
[20:21:38] <drdee>	 AAARRGGHHHHHHH
[20:22:13] <drdee>	 in mysql giving rights to user@localhost != user@127.0.0.1
[20:23:00] <drdee>	 ottomata:
[20:23:07] <drdee>	 drdee: AAARRGGHHHHHHH
[20:23:07] <drdee>	 [3:22pm] drdee: in mysql giving rights to user@localhost != user@127.0.0.1
[20:23:18] <drdee>	 i got hive stats to work
[20:24:03] <ottomata>	 ?
[20:24:06] <ottomata>	 whatcha doing?
[20:25:51] <drdee>	 i got hive stats to work
[20:26:06] <drdee>	 i reassigned the asana task to you with the steps
[20:26:20] <drdee>	 in the end this was my problem:  in mysql giving rights to user@localhost != user@127.0.0.1
[20:26:40] <drdee>	 so mysql kept complaining about access denied
[20:26:50] <drdee>	 and i am like i am sure i granted access to the hive user
[20:26:53] <drdee>	 etc etc
[20:27:11] <ottomata>	 yeah, localhost != 127.0.0.1 according to mysql
[20:27:14] <ottomata>	 so it was a permission problem?
[20:27:44] <ottomata>	 lemme do that now, I just finished the storm packaging!
[20:27:47] <ottomata>	 so easy! so nice
[20:28:03] <ottomata>	 i made a few tweaks to some of the packaging stuff to make it not conflict with existing libzmq package, etc. etc.
[20:28:08] <ottomata>	 now its in the kraken apt repo
[20:28:15] <ottomata>	 installing storm should just be apt-get instal lstorm
[20:28:16] <ottomata>	 weeeee
[20:28:30] <drdee>	 https://app.asana.com/0/828917834272/2399816680078
[20:28:43] <ottomata>	 ja see it
[20:28:45] <drdee>	 oh you knew this  localhost != 127.0.0.1 according to mysql
[20:28:47] <drdee>	 ????
[20:29:14] <ottomata>	 that i knew, yeah
[20:29:23] <drdee>	 hive errors are in /tmp/<username/hive.log btw
[20:29:26] <ottomata>	 not only is the username different (usernames in mysql are the full user@host)
[20:29:31] <ottomata>	   /tmp!
[20:29:33] <ottomata>	 crazy
[20:29:34] <ottomata>	 but also
[20:29:43] <ottomata>	 mysql sees 'localhost' as connections coming through the mysql.sock file
[20:29:46] <drdee>	 we can fix that (i guess)
[20:29:50] <ottomata>	 and '127.0.0.1' as localhost netowrk connections
[20:30:48] <drdee>	 not sure if it makes sense to me but i figured it out the hard way (without google) ;)
[20:31:43] <ottomata>	 hmmm, weird, hmm
[20:31:53] <ottomata>	 so I did grant these perms on my local when I was trying to figure this out
[20:38:57] <ottomata>	 cool, drdee, done.  puppetized
[20:39:01] <ottomata>	 check to make sure it still works
[20:39:13] <drdee>	 aight, let me know when puppet has run
[20:39:29] <drdee>	 should we fix the log location of hive or just leave it at the default?
[20:40:14] <drdee>	 ottomata, RobH was looking for you, the Dell's are racked up and he was fiddling with them
[20:40:21] <ottomata>	 its run
[20:40:23] <ottomata>	 Whoaaaaaa
[20:40:30] <ottomata>	 yeah my adium crashes sometimes
[20:40:36] <ottomata>	 and when I reopen it
[20:40:39] <ottomata>	 i forget to join the chat rooms
[20:40:41] <ottomata>	 it doesn't auto join me
[20:40:48] <ottomata>	 RobH
[20:40:54] <ottomata>	 tell me about things!
[20:41:20] <ottomata>	 hmmm, well they are per-user logs
[20:41:23] <ottomata>	 which is good in /tmp
[20:41:41] <drdee>	 k
[20:42:48] <RobH>	 ottomata: 1011-1022 racked
[20:42:54] <RobH>	 setting up the drac stuff now
[20:42:57] <RobH>	 will be awhile longer
[20:43:07] <RobH>	 (but will have them accessible today for you)
[20:45:00] <ottomata>	 ok cooooooooool
[20:45:04] <ottomata>	 exciting!
[20:50:39] <drdee>	 ottomata, are you gonna play with the dell's now or is that too early?
[20:51:06] <ottomata>	 need drac set up, probably only working for an hour or so more today
[20:51:11] <ottomata>	 so sounds like a tomorrow project
[20:51:32] <RobH>	 yea i will be settin gup drac stuff for another 30 to 45 minutes atleast
[20:51:49] <RobH>	 these didnt come enabled, so have to boot off iso, enable trial drac port, then go back and upload the actual drac key
[20:51:54] <ottomata>	 cool, i might start playing with them for a min, just to get my bearings and see if the scripts we have currently will work
[20:51:57] <ottomata>	 and start doign them tomorrow
[20:52:04] <RobH>	 well, none are ready yet ;]
[20:52:06] <ottomata>	 ah annoying, but cool
[20:52:09] <ottomata>	 yeah yeah
[20:52:10] <ottomata>	 when they are
[20:52:15] <RobH>	 im batching, doing all the flashing, then all the programming, then all the uploads
[20:52:24] <ottomata>	 yayyyy
[20:52:26] <ottomata>	 thanks
[20:52:37] <drdee>	 you wanna spend your final hour on the jobhistory server stuff?
[20:52:46] <ottomata>	 sure, actually, aeyh thats a good idea
[20:52:52] <ottomata>	 reading some storm docs now, but lemme put that aside for now
[20:52:53] <ottomata>	 cool
[20:53:47] <drdee>	 so it seems that all the logs of finished jobs are stored on hdfs://var/log/hadoop-yarn/apps/<username>/logs/
[20:53:52] <ottomata>	 yeah
[20:57:44] <drdee>	 which log file shows the "Failed redirect for container" message?
[20:58:17] <ottomata>	 good q
[21:01:59] <drdee>	 hadoop has some many log files…….. :\
[21:02:51] <drdee>	 probably the node manager log files...
[21:05:40] <ottomata>	 tailing all logs now, don't see anything
[21:06:20] <ottomata>	 so periodically, i see this message in  /var/log/hadoop-mapreduce/yarn-mapred-historyserver-analytics1001.log
[21:06:26] <ottomata>	 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
[21:10:36] * drdee  is reading yarn-mapred-historyserver-analytics1001.log 

[21:10:50] <ottomata>	 (i just restarted historyserver, fyi)
[21:10:54] <drdee>	 but that's not periodically i would say, it's all the time
[21:11:09] <drdee>	 but there is not a single mention of redirect error
[21:11:16] <drdee>	 maybe that error is caught by jetty
[21:11:26] <drdee>	 as it is a web / http error
[21:11:49] <ottomata>	 eah
[21:12:32] <drdee>	 i am just generating ideas here :)
[21:14:56] <drdee>	 what is your suspicion?
[21:15:11] <ottomata>	 i think that the jobs are not being  moved properly
[21:15:26] <ottomata>	 from /user/history/done_intermediate to /user/history/done
[21:15:26] <ottomata>	 not sure why t hough
[21:15:40] <ottomata>	 i just set those values explicitly
[21:15:54] <ottomata>	 i'm gonig to restart all hadoop and see if it makes a difference
[21:15:59] <drdee>	 how about tweaking /etc/hadoop/conf/log4j.properties
[21:16:03] <drdee>	 and get more data?
[21:21:43] <ottomata>	 growl internet
[21:22:03] <drdee>	 not sure if you received this msg:
[21:22:07] <drdee>	 drdee: how about tweaking /etc/hadoop/conf/log4j.properties
[21:22:08] <drdee>	 [4:16pm] drdee: and get more data?
[21:22:19] <ottomata>	 i think that the jobs are not being  moved properly
[21:22:19] <ottomata>	 4:15
[21:28:19] <drdee>	 this is the relevant source code: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.0/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java
[21:34:13] <average_drifter>	 back
[21:35:05] <drdee>	 welcome
[21:35:10] <average_drifter>	 :)
[21:35:15] <average_drifter>	 ok so I'm going to do the editor configs now
[21:35:42] <drdee>	 ?
[21:35:52] <average_drifter>	 drdee: wikistats editor separte config .pm files
[21:36:01] <average_drifter>	 *separate
[21:37:55] <average_drifter>	 getting a new instance of wikistats ready that only consumes the sampled edits traffic
[21:38:16] <average_drifter>	 where do I get the edits traffic logs ?
[21:38:22] <average_drifter>	 are they somewhere on stat1 ?
[21:38:55] <average_drifter>	 or maybe I have to grep from existing log specific urls which are the ones from the edits
[21:40:24] <drdee>	 stat1:/a/squid/archive/edits/
[21:40:30] <average_drifter>	 alright
[21:40:31] <drdee>	 or something like that
[21:45:48] <ottomata>	 ok, restarting hadoop one more time
[21:48:11] <drdee>	 ottomata, are we on a roll today or what??
[21:48:25] <drdee>	 i am all PUMPED
[21:48:53] <ottomata>	 yeehaaaawwww
[21:49:20] <drdee>	 we put the history in the yarn in the server in the netty going backy to the hue weathering the storm racking up some boxes all back 2 back with
[21:50:27] <ottomata>	 hahaha
[21:50:29] <ottomata>	 drdee is freestyling!
[21:50:59] <ottomata>	 so the failed redirect thing still happens
[21:51:05] <drdee>	 cRAp
[21:51:14] <drdee>	 bout….. the great news is that we really are getting closer with configuring the entire stack
[21:51:40] <drdee>	 wait
[21:51:46] <drdee>	 what redirect?
[21:51:54] <drdee>	 i mean from where?
[21:52:05] <ottomata>	 so, for new jobs that have not yet rotated into job history
[21:52:13] <ottomata>	 i et the failed redirect for container
[21:52:16] <ottomata>	 when I try to view the logs
[21:52:22] <drdee>	 me too
[21:52:36] <ottomata>	 so i'm not sure what that is
[21:52:55] <ottomata>	 i think maybe you can't view completed job logs until the job has been moved into job history?
[21:53:33] <ottomata>	 yeah, my job just got rotated, and I can find it in history server
[21:53:35] <drdee>	 yeah…. or maybe it' still syncing up old logs?
[21:54:10] <ottomata>	 yeah, i can get logs while the job is running
[21:54:14] <ottomata>	 http://analytics1009:8042/node/containerlogs/container_1352411161525_0003_01_000001/otto/syslog/?start=-4096
[21:54:23] <ottomata>	 but once it finishes
[21:54:24] <ottomata>	 not
[21:54:32] <ottomata>	 and I have to wait til job is put into history
[21:54:35] <ottomata>	 which i guess is fine
[21:54:39] <ottomata>	 that's even what the error says
[21:54:42] <ottomata>	 Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.
[21:54:46] <ottomata>	 it has already completed
[21:55:02] <ottomata>	 so I guess that is normal
[21:55:51] <drdee>	 ok
[21:55:55] <drdee>	 this one is still empty: http://analytics1001.wikimedia.org:19888/metrics
[21:56:12] <drdee>	 i was hoping it would be fixed now as well
[21:56:56] <ottomata>	 wasszat?
[21:57:35] <ottomata>	 oh whoa
[21:57:38] <ottomata>	 this is useful:
[21:57:38] <ottomata>	 http://analytics1001.eqiad.wmnet:19888/conf
[21:57:43] <drdee>	 yup
[21:57:59] <drdee>	 this is super useful information
[21:58:11] <drdee>	 info on what conf was executed with a particular job
[21:58:19] <drdee>	 all kinds of instrumentation data
[21:58:30] <drdee>	 really useful for debugging
[21:58:57] <drdee>	 louisdang: the job history server is now also working so you should be able to debug stuff much faster
[22:01:06] <louisdang>	 drdee, cool thanks ottomata and drdee
[22:01:41] <average_drifter>	 ottomata: can we deploy the new version of udp-filters ?
[22:02:07] <ottomata>	 bwerrrrrrrr? new version!
[22:02:08] <ottomata>	 cool
[22:02:12] <drdee>	 just on stat1 for now
[22:02:12] <ottomata>	 ummmmmmm yes but not tonight!
[22:02:19] <average_drifter>	 alright
[22:02:35] <drdee>	 ottomata, pretty please :) can you just install it on stat1? no puppet
[22:03:10] <ottomata>	 oh on stat1 no probs,
[22:03:14] <drdee>	 thx!
[22:03:17] <ottomata>	 ummmm wait, can't you just use the bin?
[22:03:21] <ottomata>	 do you need it 'installed'?
[22:03:28] <drdee>	 mmmmm good point
[22:03:30] <ottomata>	 you can just compile it there, right?
[22:03:35] <ottomata>	 or even copy over the bin
[22:03:40] <drdee>	 average_drifter: can you just copy the bin to stat1?
[22:04:21] <average_drifter>	 yes
[22:05:24] <drdee>	 so just do that and go ahead with writing a bash script to geocode all existing files in /a/squid/archive/
[22:19:22] <average_drifter>	 got a package on build1, and binaries
[22:23:08] <RobH>	 ottomata: Ok, so the new servers are onlie and remotely accessible
[22:23:22] <RobH>	 their dhcp files are wrong and outdated
[22:23:48] <RobH>	 (they are com2 115200, so they will go in linux-host-entries.ttyS1-115200 file
[22:30:23] <drdee>	 brb
[23:59:50] <dschoon>	 as might be deduced from the hour, Internet Installation was a failure.
[23:59:56] <dschoon>	 i think we will tragically have to switch to comcast :(