[10:07:40] mooorning [10:07:43] qchris [10:07:50] talk dashboards? [10:07:52] good morning drdee [10:07:59] Sure :-) [10:08:07] hangout? [10:08:17] Gimme time to boot the google machine. [10:09:37] k [10:09:40] average around? [10:16:09] hey [10:16:19] drdee , qchris morning [10:16:26] I'm gonna go get a bike, I'll be back in 1h [10:16:36] less actually since I'm coming back on it [10:29:32] hey average [10:29:35] happy biking! [11:33:25] drdee: I found a host in the sampled logs that's sending space separated logs. It's only ~1 line per day, but as that's sampled data, it might actually be much more lines. Has this been brought up for "maerlant" before? [11:34:43] (all log lines of maerlant cover IPv6 addresses) [11:50:56] i think that's the machine hosting the blog sit [11:51:13] poke ottomata about it, i think it has it's own config separate of varnish [11:51:17] good catch! [11:51:34] Ok, I'll ping ottomata. [13:09:19] hey ottomata [13:10:54] hiya morning [13:11:12] isn't maerlant hosting the blog? [13:11:20] that was marmontel [13:11:25] stupid names [13:11:29] yeah you are right [13:11:42] ok then i don't know what's going on either [13:18:25] good monday morning everyone :) [13:18:35] good mornign! [13:18:41] Good morning milimetric :) [13:19:09] qchris, i'm talking to faidon about maerlant, looks like it is an old unused ipv6 proxy that probably should be decomissioned more completely [13:19:27] :) [13:19:31] I just saw it in the ops channel. Thanks. [13:19:32] morning milmietric [13:20:01] milimetric, could you update User Metrics Database Model.mwb in Wikimetrics [13:20:24] sure [13:20:40] (i mean with the current state of the db) [13:20:40] htx! [13:57:56] restarting (mysql workbench died without saving a couple times) [14:06:53] drdee: I am having a hard time bringing the old and new country numbers for zero in relation. [14:06:57] drdee: Do you know how the numbers of [14:07:01] drdee: http://stats.wikimedia.org/kraken-public/webrequest/mobile/zero/country/zero_country-daily.tsv [14:07:07] drdee: got/get computed? [14:07:22] using the zero_country.pig script IIRC [14:07:40] drdee: On what input? [14:08:01] on kraken on the mobile stream [14:08:20] so you might need to adjust the zero_country script when running it against the sampled stream [14:08:28] and only consider .m. and .zero. traffic [14:08:35] not sure if zero_country.pig does that [14:09:31] * qchris facepalms himself :-) [14:09:39] Thanks. [14:14:25] np [14:31:12] hm, i'm pretty sure zero country doesn't know antyihng about zero traffic, but it was only running on the mobile stream [14:31:22] qchris, you should just filter for a list of mobile hosts, just like we are doing: [14:31:43] http://noc.wikimedia.org/pybal/eqiad/mobile [14:31:43] http://noc.wikimedia.org/pybal/esams/mobile [14:32:16] hey dudes, qchris maybe, i need a little brain bounce [14:32:20] but I think via IRC is fine [14:32:22] so. [14:32:32] i'm working on this hadoop HA namenode stuff [14:32:37] ottomata: Mhmm. So filtering on the domains is not what we want? [14:32:43] not for country [14:32:51] country didn't filter domains (aside from using the is_pageview udf) [14:33:09] we imported any requests generated from those hosts [14:33:15] and then ran zero_country.pig on that data [14:33:16] that's it [14:33:28] Do we know how the list of hosts changed over time? [14:33:36] a little bit, ja, hmm [14:34:05] (Since mid of July would suffice ) [14:34:32] oh, since mid of july [14:34:37] it should be the same as what you see there [14:34:42] Ok. Great! [14:34:46] Thanks. I'll try that. [14:34:55] Now about brainbouncing around Hadoop :-) [14:35:08] this is the regex we are using to imoport [14:35:09] '(cp1046.eqiad.wmnet|cp1047.eqiad.wmnet|cp1059.eqiad.wmnet|cp1060.eqiad.wmnet|cp3011.esams.wikimedia.org|cp3012.esams.wikimedia.org|cp3013.esams.wikimedia.org|cp3014.esams.wikimedia.org)' [14:35:18] ok about hadoop [14:35:27] so, there's a new service called JournalNode [14:35:38] this service has a directory that needs to be initialized with the shared edits [14:35:43] journalnode can run anywhere [14:35:52] any machine that can talk to the namenode [14:36:00] it could run on a namenode, or on a datanode, or anywhere [14:36:03] k [14:36:06] now, to initialize the journalnode [14:36:10] i need to run a command on the namenod [14:36:12] e [14:36:15] -initializeSharedEdits [14:36:35] this copies the namenode data node over to the JournalNode quorum, and has each of them initialize their edits directory [14:36:45] this is a difficult thing to achieve in puppet [14:37:06] I need to run a command on a host, and then check the result other host(s) [14:37:10] But I take it, it's not a one off command, or is it? [14:37:14] it is [14:37:17] Ok. [14:37:45] you'd need to run it anytime you add new journalnodes, including when you are building the cluster [14:37:58] i'm not sure how I can do this in puppet, hmm [14:38:10] i need a way to check the result of the -initializeSharedEdits command from the namenode [14:39:12] I am not sure but puppet is mostly considering static things (except services), while that seems like a dynamic property ... [14:39:25] Could one puppetize a script to carry out those tasks .. [14:39:38] Then we could run the script upon need. [14:39:53] naw, that kinda goes against what i'm trying to do here [14:40:02] Ok. [14:40:05] sure, general maintainance (like adding a new journalnode) won't be puppetized [14:40:08] you can do that manually [14:40:08] but [14:40:20] i'd like this to be able to almost 100% init a new cluster from scratch [14:40:30] Ah. I see. [14:40:32] its been like that so far [14:40:38] this new thing is making it more difficult [14:43:09] Mhmm ... so you are trying to run a command on the one host and check the result on the other host... [14:43:28] Can you invoke the "command on the one host" from the other host? [14:43:47] Or can you bring the result from the other host to the one host? [14:44:10] i could, but it would require complicate ssh key setup [14:44:42] ok, i just realized something, hmmm [14:44:56] i only need to run this command when i'm adding HA to a cluster that didn't have it before [14:44:59] so maybe I don't need to puppetize it... [14:45:00] hmmm [14:45:46] Having to run it only once would make sense to me. [14:47:13] HMmmm [14:47:50] hmmm, yeha ok, i think this is fine [14:48:00] when i'm creating a brand new cluster [14:48:13] as long as the journalnodes are running before I initially format the namenode, then i dont' need to run this command [14:48:27] but, if I am adding HA to an existing cluster [14:48:33] I will have to run this command from a namenode [14:48:43] which i will not puppetize, just document [14:48:47] ok cool, thank you [14:49:10] Sounds good to me :-) Yippie HA is coming :D [15:06:50] drdee, ottomata: I am thinking about filtering sampled-1000 for mobile hosts ... couldn't I just pick /a/squid/archive/mobile/mobile-sampled-100 ? [15:08:25] yes that should work as well (i think) [15:08:37] yup [15:08:37] :) [15:08:56] Great. Then we can go on using the vanilla zero_country.pig scripts. Thanks. [15:08:57] i only fixed that a couple of weeks ago though, so you can't go back far with that [15:09:18] I'll start on Juli 12. That should work. [15:20:42] (PS1) Milimetric: db design update, forms renamed, tests fixed [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/76514 [15:21:04] (CR) Milimetric: [C: 2 V: 2] db design update, forms renamed, tests fixed [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/76514 (owner: Milimetric) [15:21:18] drdee, ^^ adds the updated db file [15:21:24] ty [15:21:32] could you edit it and remove the old diagram? [15:21:38] sure [15:21:39] it crashes every time I try for me [15:22:01] so you can leave the original umapi design and the new one which is "diagram 2" [15:22:28] I used the reverse engineer method after I fixed all the tests, pretty easy so I'll try to keep it up to date now [15:25:33] aight [15:25:34] check https://mingle.corp.wikimedia.org/projects/analytics/cards/814 [15:25:50] let me know what you think [15:33:52] milimetric: why does the cohort_user table contain a link to user_id? [15:36:51] milimetric_ ^^ [15:38:17] the user that owns the cohort drdee [15:38:33] ohh duuhh [15:38:39] sorry i misread that table [15:38:59] yea, it's that whole "user" / "wikiuser" / "mediawiki user" thing [15:39:02] soo confusing [15:39:29] but shouldnt role migrate to to the 'user' table [15:39:38] that seems redundant now [15:39:47] no there are two roles we're talking about here [15:39:56] the CohortUser role says what rights the User has on the Cohort [15:40:09] the User.role says what rights the User has in WikiMetrics overall [15:42:51] aigt [15:42:52] ty [15:57:41] milimetric, have a look at design/WikiMetrics_Database_Model_Tags.mwb [15:57:53] in the wikimetrics repo [15:58:39] milimetric, can you also have a look at https://docs.google.com/a/wikimedia.org/document/d/1GBckuYOeFFsISNOsgTFskOcRM00SHR4X5Wo4mVt7RGo/edit# ,there are some questions for you [16:02:18] (PS1) Erik Zachte: final tweaks on new udp msg loss report, before publishing [analytics/wikistats] - https://gerrit.wikimedia.org/r/76518 [16:12:23] milimetric; ^^ [16:13:37] (CR) Erik Zachte: [C: 2 V: 2] final tweaks on new udp msg loss report, before publishing [analytics/wikistats] - https://gerrit.wikimedia.org/r/76518 (owner: Erik Zachte) [16:31:14] hey qchris, what's the progress with the country job? [16:31:59] Generation of the data is mostly done. [16:32:07] 2 more jobs to run. [16:32:38] Then copying the things around, regenerating the dashboards, and pushing them. [16:33:04] how much more time would that take? [16:33:26] I'll finish that today. [16:33:38] But we've got the standup meeting, I've got to eat something, ... [16:33:47] So give it a few hours. [16:33:58] ok [16:36:59] going to get some food, ttys [17:00:45] tnegrin: scrum [17:21:18] milimetric: like your idea with the card [17:21:20] let's do that [17:53:00] drdee: Could you look at the zero dashboards, if they are as expected? [18:35:54] qchris: can you send me some URL's? [18:40:57] hi tnegrin [18:41:21] hey -- on a call [18:42:17] :) [19:29:19] (PS1) Milimetric: moving report nodes in their own directory [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/76559 [19:29:29] (CR) Milimetric: [C: 2 V: 2] moving report nodes in their own directory [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/76559 (owner: Milimetric) [20:03:40] (PS1) Milimetric: changing to 90 character lines to make editing easier [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/76617 [20:03:49] (CR) Milimetric: [C: 2 V: 2] changing to 90 character lines to make editing easier [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/76617 (owner: Milimetric) [20:14:36] hey ottomata [20:15:30] hiya [21:57:10] milimetric: can you look at https://docs.google.com/a/wikimedia.org/document/d/1GBckuYOeFFsISNOsgTFskOcRM00SHR4X5Wo4mVt7RGo/edit# and address those issues were your name mentioned i s? [21:58:55] yes... but i'm kind of in the middle of a bunch of problems [21:58:57] (see my email) [21:59:04] so I look to you for guidance on priority [22:00:33] the docs are very quick questions, and i need that for my email to jaime [22:00:49] for the rest i agree with your email and you should focus first on the snags that you identified [22:04:02] ok boys i'm out for the eve i think [22:04:03] latesr all [22:04:29] tnegrin: hi there, got a moment? [22:04:31] laterz ottomata [22:04:34] I am following ottomata [22:04:40] good night everybody :-) [22:04:55] yes -- hi sumana [22:05:06] tnegrin: I can talk now about the Wikivoyage stuff