[00:45:26] (PS1) Jdlrobson: Fix localurl and remoteurl for mac osx [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76050 [00:45:28] (PS1) Jdlrobson: Add --graph options to allow generation of single graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76051 [00:45:29] (PS1) Jdlrobson: Give a meaningful message about generation [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76052 [00:45:30] (PS1) Jdlrobson: Add graph for all successful edits [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76053 [00:45:31] (PS1) Jdlrobson: Add all the editing graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76054 [00:47:45] (PS2) Jdlrobson: Add all the editing graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76054 [13:19:20] morning guys [13:19:26] average aroudn? [13:19:31] qchris around? [13:19:39] Hi drdee :-) [13:19:43] yo [13:19:50] Good morning [13:19:57] what's the progress with getting the zero dashboards up-to-date? [13:20:34] I am working on getting the zero config into kraken in an automated way [13:20:43] k [13:20:46] So we operate on up-to-date data. [13:20:55] That means adding date ranges, checks etc [13:20:56] but i think we should first get the data into the dashboards [13:21:04] and then tweak the workflow [13:21:15] amit really needs his dashboards :( [13:21:26] The data we now have is wrong, as the config in kraken is wrong. [13:21:42] Would Amit want to see data based on wrong Kraken configs? [13:21:59] i don't think it's order of magnitude wrong [13:22:16] but we could also first to manually tweak the config [13:22:19] We do not have all carriers. [13:22:30] And have wrong ones etc. [13:22:50] but those can be manually fixed [13:23:02] (using ZeroConfig) [13:23:15] That trades a broken system for a system that is broken by tomorrow :-/ [13:23:25] true and that's fine [13:23:30] ? Really. [13:23:36] for now it is [13:23:40] Ok. [13:23:42] So it be. [13:23:44] next week we can fix it for real [13:24:01] but now we just need to give some dashboards to amit [13:24:13] Ok. [13:24:27] don't get me wrong, i agree we need a permanent fix for this [13:24:44] i just would like to punt that to next week [13:35:03] morning everyone [13:35:05] morning milimetric [13:35:09] I'm gonna be kind of spotty today [13:35:13] my brother's over [13:35:21] (from Tennessee) [13:35:23] aight that's ok [13:38:48] (PS1) Erik Zachte: Only show incomplete projects in 'progress last week' [analytics/wikistats] - https://gerrit.wikimedia.org/r/76095 [13:40:01] (CR) Erik Zachte: [C: 2 V: 2] Only show incomplete projects in 'progress last week' [analytics/wikistats] - https://gerrit.wikimedia.org/r/76095 (owner: Erik Zachte) [14:45:48] hi [14:46:03] hiiii [14:46:11] qchris , drdee , milimetric , ottomata hi :) [14:46:21] Hi average. [14:53:16] qchris: wanna talk some dclass ? [14:53:37] average: Sure. [14:53:49] I'm in the hangout [14:54:05] average: Oh ... ok. [14:54:15] average: Gimme a few minutes to beet the google machine. [14:54:18] ok [16:30:10] drdee: erosen ran the basic preparation on stat1, to which I am lacking access. [16:30:34] ottomata: can you give access to qchris to stat1? [16:31:12] qchris, curious, whatcha need on stat1? [16:31:18] i thikn the answer is yes, but you might not need it [16:31:28] https://rt.wikimedia.org/Ticket/Display.html?id=5474 is already approved [16:31:40] erosen told me he ran data preparation jobs for wp_zero on stat1 [16:31:56] yeah we can get you stat1, [16:31:57] hm [16:31:59] ottomata: So I wanted to run them on the same machine. [16:32:01] stat1002 has all the raw data though [16:32:08] we shoudl move that kind of stuff over to stat1002 [16:32:12] I have access to stat1002. [16:32:13] we'll eventually delete that data from stat` [16:32:14] stat1 [16:32:22] So I'll try on stat1002 directly. [16:32:31] ok, if you need something on stat1 copied to stat1002, let me konw [16:32:37] i'll give you access to stat1 too [16:32:42] Ok. Thanks. [16:32:44] but this stuff should be on stat1002 [16:33:42] i agree with ottomata (for what it's worth :) ) [16:44:08] k, gonna change locations and get lunchy, back on in a bit [18:01:52] drdee [18:01:55] i'm not sure yet [18:02:01] at your service! [18:02:12] but I think we have to be on cdh 4.3.0 in order to turn a non HA cluster into an HA cluster [18:02:22] it seems like an important step is missing [18:02:26] it isn't in the 4.2.1 docs [18:02:32] and the command isn't available [18:02:34] but it is in the 4.3 docs [18:03:55] HMMmm, no [18:03:57] it should be there [18:04:00] hmmmmmm [18:04:01] not sure yet [18:04:30] i see JIRAs that do things with this command (hdfs namenode -initializeSharedEdits) back in Aug 2012 for 4.2.0 in changelog [18:04:31] hmmmm [18:08:43] bwa but I really don't have that command [18:08:47] grhh [18:11:33] try it in labs first? [18:12:14] i'm trying in in my vm first [18:12:22] labs happens after it all works here :) [18:12:31] i'm installing 4.3.0 to see if the command exists there [18:15:43] k [18:15:45] http://www.youtube.com/watch?v=QW_aJDPPGMg [18:16:12] Captain Jack: ^^ [18:21:14] (CR) JGonera: [ V: 2] List the files you need to create when adding a graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76037 (owner: JGonera) [18:23:24] (CR) JGonera: [ C: -1] "Unfortunately, this breaks it for Linux..." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76050 (owner: Jdlrobson) [18:23:26] ha [18:23:56] hmmmMMm ok i think I go tit [18:24:01] i can't run that command while the namenode is running [18:24:04] which is kinda weiiird [18:24:17] but in our case ok as long as we don't run the journalnode on the namenode [18:24:21] which I don't think i want to do anyway [18:24:29] but we *should* be able to if we wanna [18:44:06] (PS2) JGonera: Fix localurl and remoteurl for mac osx [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76050 (owner: Jdlrobson) [18:45:06] (PS3) JGonera: Fix localurl and remoteurl for mac osx [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76050 (owner: Jdlrobson) [18:45:18] (CR) JGonera: [ C: 2] Fix localurl and remoteurl for mac osx [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76050 (owner: Jdlrobson) [18:45:23] (CR) JGonera: [ V: 2] Fix localurl and remoteurl for mac osx [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76050 (owner: Jdlrobson) [18:45:34] (PS2) JGonera: Add --graph options to allow generation of single graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76051 (owner: Jdlrobson) [18:50:01] (CR) JGonera: [ C: 2 V: 2] Add --graph options to allow generation of single graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76051 (owner: Jdlrobson) [18:50:17] (PS2) JGonera: Give a meaningful message about generation [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76052 (owner: Jdlrobson) [18:50:23] (CR) JGonera: [ C: 2 V: 2] Give a meaningful message about generation [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76052 (owner: Jdlrobson) [18:51:06] (PS2) JGonera: Add graph for all successful edits [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76053 (owner: Jdlrobson) [18:56:20] drdee, fyi [18:56:21] (CR) JGonera: [ C: 2 V: 2] Add graph for all successful edits [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76053 (owner: Jdlrobson) [18:56:34] ahhh i'll email [18:56:36] (PS3) JGonera: Add all the editing graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76054 (owner: Jdlrobson) [19:03:08] heya Snaps, you there? [19:03:26] (CR) JGonera: [ C: 2 V: 2] Add all the editing graphs [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/76054 (owner: Jdlrobson) [19:08:11] milimetric, has something changed in production server for mobile limn? fab is asking me for password... [19:08:14] [limn0.pmtpa.wmflabs] Login password for 'jgonera': [19:08:31] hey jgonera [19:08:36] yeah, unfortunately [19:08:38] I have no idea what [19:08:40] or why [19:08:52] uh, so how do I deploy new graphs? [19:08:54] I haven't been able to deploy any fixes or changes for over a month [19:09:01] grumble [19:09:03] well [19:09:12] for now, you can just ssh to limn0 [19:09:21] cd to /var/lib/limn/your-folder [19:09:30] and sudo git pull on your repository [19:09:40] or set up a cron to do that once in a while [19:09:46] other people (Evan) have done that [19:10:01] but I think something changed in Labs that broke fabric [19:10:04] or something changed in Fabric [19:10:14] I looked into it for a few hours but couldn't figure it out [19:11:18] ottomata, could you shed your light on this problem maybe ^^ [19:11:31] jgonera: let me know if you can't do that and I'll do it for you [19:11:39] sure, gimme some mins, or maybe tomorrow [19:11:42] my head is in this thannng [19:11:46] milimetric, how do I ssh to limn0? $ ssh limn0.pmtpa.wmflabs [19:11:46] ssh_exchange_identification: Connection closed by remote host [19:11:48] or i mean, monday [19:11:52] milimetric: want to bug me about this monday? [19:11:54] mabye we can solve it [19:11:55] ? [19:12:04] ok ottomata [19:12:15] if it takes too long I'll have to abandon it again [19:12:25] jgonera can you do -vvv and paste the output to a gist? [19:12:26] wikimetrics needs a bit of love this first week post-launch [19:12:27] ha, ok, well, maybe you can just get me into the fab stuff, and I can figure it out [19:12:41] ssh -vvv? [19:12:42] yeah, jgonera, should be limn0.pmtpa.wmflabs [19:12:45] yup [19:13:10] in the meantime jgonera, I'll deploy the limn-mobile [19:13:16] would you like limn-mobile-dev too? [19:13:34] try the dev first and see if it works, I tried it locally, but just to keep it safe [19:14:26] drdee_, https://gist.github.com/jgonera/0c45e90e21e00bf5e940 [19:15:23] k, jgonera, dev is up: http://mobile-reportcard-dev.wmflabs.org/ [19:15:25] looks ok [19:15:33] oh [19:15:33] nvm [19:15:44] jgonera: this looks suspicious: [19:15:51] debug3: Incorrect RSA1 identifier [19:15:52] debug3: Could not load "/home/jgonera/.ssh/id_rsa" as a RSA1 public key [19:15:53] throws 404s on this: http://stat1001.wikimedia.org/limn-public-data/mobile/datafiles/successful-edits-other.csv [19:16:11] milimetric, yeah, I guess the generator script runs every hour [19:16:20] so it will be empty until it generates them [19:16:31] gotcha [19:16:31] but that's OK I think [19:16:37] if dev looks ok [19:16:41] i can do the main one [19:16:42] so I think we can deploy to the normal one too [19:16:49] yes, please [19:17:04] k, done [19:17:13] /var/lib/limn/mobile-reportcard/data-repository$ sudo git pull [19:17:16] that's the command [19:17:48] if we can't resolve this and you have a problem the next time, just let me know and I'll set up a cron job for the dev one. Then it'll always deploy and you can ping me if you want the normal one [19:17:59] drdee_, I did not change anything in my SSH key, and it works for gerrit and stat1 [19:18:18] yeah, I think everyone had problems with limn0, and had to change their bastion or something [19:18:22] Evan and I at least [19:18:34] but even after that, fabric still won't work [19:18:42] I see, what should I change it to? [19:18:44] the crazy world of ssh... [19:18:49] yeah, but I could at least ssh I guess [19:18:50] let me tell you what i have [19:19:11] /var/lib/limn/mobile-reportcard/data-repository$ sudo git pull [19:19:15] oops :) [19:19:23] ;) [19:19:26] Host limn0 [19:19:26] HostName limn0.pmtpa.wmflabs [19:19:28] jgonera can you connect to bastion.wmflabs.org ? [19:19:36] Host limn0 wikimetrics puppet1 [19:19:36] ProxyCommand ssh -e none bastion.pmtpa.wmflabs exec nc -w 3600 %h %p [19:19:36] User milimetric [19:19:36] ForwardAgent no [19:19:57] drdee_, yes [19:20:01] mmmmmm [19:20:06] Host bastion.pmtpa.wmflabs [19:20:06] HostName bastion.wmflabs.org [19:20:13] I have [19:20:13] Host bastion.pmtpa.wmflabs [19:20:13] HostName bastion2.wmflabs.org [19:20:14] #HostName bastion.wmflabs.org [19:20:14] can you connect to limn0 from bastion [19:20:14] those three sections define the access to limn0 [19:20:17] yeah! [19:20:19] maybe that's the problem [19:20:21] I had bastion2 as well [19:20:30] that was back when bastion went down for a while [19:20:33] I think they killed 2 though [19:20:53] ok, changed it and it works, cool, thanks milimetric and drdee_ [19:21:04] no prob [19:21:07] np [20:24:50] ottomata: On limn0 there should be a cron job to pull from https://github.com/embr/wp-zero-data. (Maybe that's hidden in a script) Probably running ever 15 minutes. Any chance we could redirect to pull from https://gerrit.wikimedia.org/r/#/admin/projects/analytics/wp-zero/data instead? [20:25:04] ottomata: (For user erosen) [20:25:14] i betcha it is under erosen's user [20:25:20] Yep. [20:25:21] you should ahve sudo there [20:25:25] i thnink [20:25:28] oh. [20:25:29] yes [20:25:32] That would make things easier :-) [20:25:47] i gave qchris sudo on all our labs instances [20:26:05] * qchris does some evil grinning. Hehehe. [20:26:07] may wisdom be your travel partner, and havoc stay waway [20:26:16] :-) [20:27:01] Wisdom already ends: [20:27:06] Sorry, user qchris is not allowed to execute '/bin/ls /home/erosen' as erosen on limn0.pmtpa.wmflabs. [20:28:22] Oh ... I found a way that should work. [20:28:26] Sorry for the noise. [20:38:40] milimetric; check https://mingle.corp.wikimedia.org/projects/analytics/cards/797 [21:52:44] drdee: When looking at the current dashboards as for example http://gp.wmflabs.org/graphs/free_mobile_traffic_by_version , the data seems to less shaky for now [21:53:12] drdee: Since we are lacking country numbers, do you think we could use the country numbers of those graphs? [21:54:12] drdee_ ^ [21:54:37] * qchris gets confused by the different dees in here. [21:54:45] we can try [21:54:53] and see if it works [21:54:54] or not [21:55:16] But at leaste we have no reason to not trust the data? [21:55:31] not in particular [21:55:41] Ok, that's good. [21:55:42] i am sure there are some edge cases that are not covered [21:56:01] Do we know where that data is coming from. It's the sampled webrequests? [21:56:30] Coming from kraken? [21:56:35] hadoop? [21:56:39] nothing comes from kraken/hadoop [21:56:42] it goes into kraken [21:56:50] so the data always comes from udp2log [21:56:57] ok :-) [21:57:13] yeah we could use the sampled web request stream to get country numberse [21:57:28] and use the pig udf just like you did for X-cs