[14:12:22] mooooooooooorning [14:27:51] morning! [14:32:22] righty [14:32:51] how's tricking sbt working? [14:32:59] morning milimetric? [14:37:20] its ok, it feels so wrong, every bit of it [14:37:29] apparently I need to update the kernel on all analytics nodes and reboot them [14:37:32] and i should do this asap [14:37:34] going to work on that now [14:39:27] ? [14:39:35] what's happening [14:39:48] hi drdee [14:39:55] yoooooooooo! [14:39:57] workin on limn iterator :) [14:40:04] http://arstechnica.com/security/2013/05/critical-linux-vulnerability-imperils-users-even-after-silent-fix/ [14:40:35] ottomata: what's faidon mean when he says "downloads random binary from personal github account" [14:41:01] don't we have only blessed packages in our nexus? [14:41:09] and if not, couldn't we easily do that? [14:41:25] milimetric: i am working on an email about that [14:41:52] milimetric: the kafka build does this [14:41:53] addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.2.0") [14:42:00] and is there a way to check signatures of packages with maven? surely there must be [14:42:18] oh :/ [14:42:18] yes there is [14:42:55] i think w don't need that though, it just does it by default [14:43:19] we can disable that [14:43:44] that's for intellij idea integrateion [14:44:51] yeah, we should definitely keep dev env specific stuff out [14:45:10] erosen, you there? [14:45:16] ya [14:45:19] sup [14:45:28] i need to reboot stat1 sometime soon [14:45:31] erosen: saw my email? [14:45:34] see you are doing stuff there [14:45:40] drdee: yes [14:45:44] ottomata: I am... [14:45:54] any chance you can hold off for another hour? [14:45:56] yeah, can do [14:45:59] or two? [14:46:05] let me get a concrete estimate [14:46:05] ja [14:47:27] ottomata: what is the timeline for the reboot? [14:47:53] i think realistically this might be the sort of thing which takes all day [14:47:59] but I can just restart the job after you restart stat1 [14:48:34] ottomata: ^^ [14:50:43] ops wants me to do it asap, especially for stat1 [14:52:34] k [14:52:39] i'll kill things now [14:53:29] ok [14:56:17] ottomata, shall we send an email to the analytic list about the unscheduled maintenance? how long do you think it will take? [14:56:20] (regarding stat1) [14:56:49] not long [14:56:51] long enough to reboot [14:56:55] 5 mins? [14:56:59] ottomata: go for it [14:57:00] oh ok [14:57:05] no email necessary [14:57:15] i have a few processes still, but they can all be killed in any manner [14:57:24] well, maybe, halfka has snuggle on there still :/ [14:58:07] definitely send an e-mail [14:58:21] but maybe it can be a notification not a RFC [14:59:13] send email :) [15:00:17] ottomata: https://plus.google.com/hangouts/_/bd7d4399631838019418ac7f6c21d0f7cd6328dd [15:00:25] meeting with alekandros [15:00:30] oh k [15:20:55] milimetric, erosen, shall we quickly test do a min-load test on metrics.wikimedia.org ? [15:21:01] sure [15:21:07] you mentioned that tool [15:21:12] apache something [15:23:15] ab [15:23:17] i tried that [15:23:25] but it was not working properly with https [15:23:33] maybe i did something wrong [15:28:16] we can curl with the htaccess credentials in the url [15:28:52] i suppose if we all had a script that ran a few jobs, we could all go 1,2,3 and launch them at the same time [15:29:26] k [15:31:28] ah, hm, no [15:31:32] the htaccess password is fine [15:31:39] but the login cookie will be harder to get [15:40:59] ok, drdee, I puzzled it out [15:41:04] cool [15:41:13] curl --data "username=dandreescu&password=****" https://e3:****@metrics.wikimedia.org/login -c ~/umapi.session [15:41:27] then you can issue requests with -b ~/umapi.session and it'll read the session cookie [15:45:37] k, tried my script and it works - let me know whenever you guys are ready we can all try it together. And we can decide how many requests to actually send. A lot of them will be served from cache so maybe we should delete the cache prior [15:47:17] !log rebooting analytics1027 (hue, oozie, etc.) [15:47:19] Logged the message, Master [15:55:04] ready [15:55:43] milimetric; i am ready to help you test [15:56:08] should we wait to have three? or just test harder :) [15:56:25] let's do it now [15:57:10] can you post the script on a gist/ [16:02:57] sure, one sec [16:03:49] drdee, ezachte and aaron H have things running on stat1 [16:03:58] i should try to get a hold of them, eh? [16:04:09] yup [16:04:11] ezachte for sure [16:05:08] https://gist.github.com/milimetric/5592871 [16:05:37] when you have it set up drdee, just ping me "START" [16:05:41] and we can both start :) [16:05:58] remember to replace the user and password :) [16:06:12] ottomata, i have pinged erik zachte [16:06:58] k, danke [16:07:07] start [16:20:48] ottomata: what is the status of stat1 [16:20:48] ? [16:22:41] i guess i need to here from ezachte before I can reboot it, sorry [16:22:48] ohp [16:22:51] and I have [16:22:54] ok, doing that now [16:23:15] milimetric, i'm rebooting stat1, i see you logged in :) [16:23:25] oh just needing to look at the slaves [16:23:27] you can boot [16:23:30] let us know when it's back up [16:23:37] k [16:36:34] p.s. re rebooting for kernel upgrades: can't wait til we use kafka instead of udp2log! we're going to lose ~5 minutes of logs per udp2log instance machine right now :/ [16:38:30] heh, i think that last script broke it :) [16:38:30] job queue went blank: https://metrics.wikimedia.org/job_queue/ [16:40:47] actually, right now we are pretty set with data loss (at least for mobile streams) due to downtime, because we are essentially importing the stream twice (once raw, and again anonymized) [16:41:31] https://gist.github.com/milimetric/5592871 [16:44:00] milimetric: how is the parallelism test going? [16:44:12] that gist will let you partake in the fun [16:44:19] maybe after scrum we can all run it together [16:44:40] hehe [16:44:41] k [16:52:18] !log rebooting stat1 for kernel upgrade [16:52:20] Logged the message, Master [18:38:02] hey erosen, wanna load test? [18:38:29] yeah [18:38:34] chatting with drdee right now about x-cs [18:38:36] but in 15? [18:38:37] k [18:38:41] sorry to keep pushing it back [18:38:45] np [18:53:00] milimetric: hangout? [18:53:04] milimietric; batcave [18:53:31] yep,going [19:14:15] hey i'm looking at x-cs right now [19:14:17] should I join? [19:15:16] we're in umapi mode atm [19:16:40] otto join [19:16:43] us pleae [19:45:23] ottomata: any ideas on why I might be having trouble logging into hue? [19:51:27] hm, no [19:51:28] what's up? [19:51:52] just hanging? [19:51:53] hm [19:52:23] hey [19:52:28] ottomata: sorry, didn't see you responses [19:52:34] yeah, it's just hanging [19:52:37] i have the ktunnel up [19:52:38] yeah for me tooo... [19:52:46] i eventually get: Error! Invalid username or password. [19:52:48] just restarted hue, looks better now [19:52:51] k [19:52:57] at least for me it is [19:53:28] hmm [19:53:35] hm, or maybe not [19:53:38] now i'm not getting anything--not even a login prompt [19:56:52] ottomata: I'm gonna grab lunch [19:57:00] let me know if you make progress on hue [19:57:06] it's not urgent for me, though [19:59:50] erosen, now? [20:16:30] ottomata: still no [20:25:39] whaaa [20:25:40] its all fine for me [20:25:42] totally