[11:15:05] (PS1) QChris: Added data for Orange Madagascar [analytics/wp-zero/data] - https://gerrit.wikimedia.org/r/88709 [11:15:32] (PS2) QChris: Add data for Orange Madagascar [analytics/wp-zero/data] - https://gerrit.wikimedia.org/r/88709 [13:18:59] morning guys [13:19:08] qchris around? [13:19:14] Morning. [13:19:25] Here I am. Tataaaa! [13:19:41] What's up? [13:22:10] the draft wikipage for the page view api [13:22:18] any thoughts / comments / suggestions? [13:22:33] Yes. It's the next but one item on my todo. [13:22:47] awesome,ty [13:23:03] can i help you with something ? [13:23:45] No. Just taking care of dependencies on others for my cards... [13:28:12] all cards in good shape :) sure we should not spend 10 minutes of writing the ua criteria for #1168 [13:28:12] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1168 [13:34:42] drdee: Was that "ua criteria for #1168" directed at me? :-) [13:35:25] About the Wikipage for the API ... is that meant as specification or as request to rope in the community? [13:38:56] drdee: ^ [13:42:25] api: both: a proposal to invite feedback [13:42:45] yes ua for #1168 was directed at you but basically to both of us [13:42:45] drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1168 [13:43:28] about 1168 ... you you want to overthrow what we previously decided on? [13:43:40] s/you you/so you/ [13:44:50] about the api: Doing two completely different things in one document typically makes both of them hard. [13:44:51] overthrow sounds almost revolutionary [13:45:42] api: it's a design document based on the understanding of the analytics team and what we think needs to happen and we ask for feedback [13:45:58] that should be possible with one document [13:46:02] api: I cannot read that off of the document. [13:46:47] To me the document is too mixed up. [13:47:42] what's mixed up about it? [13:48:49] Do we want to bikeshed that in a publicly logged channel? [13:48:57] I'd rather take it offline. [13:49:20] batcave? [13:49:30] Ok. Gimme 5 minutes. [13:54:01] Hey guys. I saw there there's a "Staff Edit-A-Thon" on the staff calendar today. What's that? [13:54:09] drdee: I am in the batcave [13:54:19] that's not 5 minutes :) [13:54:29] actually is it [13:54:33] It is! 15:49 + 5 = 15:53 [13:54:35] halfak -- [13:54:44] It is! 15:49 + 5 = 15:54 :-))) [13:54:59] halfak -- i think it's a training session for staff to learn wikitext [13:55:05] not sure if you would need it ;) [13:55:07] Oh. haha [15:26:43] FYI: "Can't connect to MySQL server on 'db1047.eqiad.wmnet'" [15:26:55] I'm in #wikimedia-operations right now trying to get a hand. [15:30:58] Looks like db1047 is back online. [15:47:56] hellllOOOo [16:03:26] qchris: hiii [16:03:36] Hi ottomata :-) [16:03:46] so let's see! [16:03:48] questions! [16:03:50] um, first [16:03:53] stat1 geowiki cron [16:03:55] Is wy spam firiing back at me? [16:04:00] Yes. [16:04:16] this class misc::statistics::geowiki::jobs::data [16:04:19] is included on stat1 [16:04:29] it puts in place cron { 'geowiki-process-data': [16:04:33] should it not? [16:04:55] 1 sec. Opening puppet files [16:06:02] Ok. [16:06:06] Yes. Agreed. [16:06:28] That cron job should be there and it is there. [16:07:23] But the problem is the cronjob 'geowiki-process-db-to-limn' [16:07:34] oh [16:07:55] We set that one to ensure => absent and it is still there [16:07:59] hm it does ahve ensure => absent [16:08:19] Yes :-)) [16:08:32] So I was curious why puppet likes it so much and keeps it around. [16:08:33] There must be something strange in your data [16:08:44] Thanks CommanderData :-D [16:08:48] running puppet [16:09:35] why [16:09:36] I didn't receive a ticket for it [16:09:39] why [16:09:40] I couldn't find any examples of how that can be done anywhere online [16:09:53] why not/ [16:09:55] You're doing it wrong [16:12:38] How long does a puppet run roughly take? [16:12:55] depends [16:12:59] on how busy the puppet master is [16:13:08] and right now it is really overloaded [16:13:14] so next question! [16:13:15] limn0 [16:13:15] hmm [16:13:21] Ok. I'll recheck in an hour. [16:13:28] Yes limn0 it is :-) [16:13:42] The changes look harmless. [16:13:46] i think dan knows more about limn0.pp than me [16:13:47] so [16:13:51] limn0 is a self hosted puppetmaster [16:14:05] Ok, then I'll ping him about it. [16:14:06] in general, we don't push labs instance puppetization changes [16:14:22] so yes, it is ok to have uncommitted changes on self hosted puppet masters in labs [16:14:26] Oh I was not talking about pushing ... just committing. [16:14:39] committing i don't know, its however dan manages that [16:14:40] So we can move in time on che checkout itself [16:14:42] i don't usually commit [16:14:43] but [16:14:46] Ok. Point taken. [16:14:46] that makes sense [16:14:49] especially on a branch [16:15:29] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [16:15:36] ok and stat1 <-> limn0 questions [16:15:41] no, they do not share a connection [16:15:42] Cool. [16:15:46] labs is supposed to be isolated from production [16:15:52] the only way that I know of that data has been shared [16:16:02] is to use a gerrit repo to push and pull from [16:16:23] limn should be moved into production [16:16:26] it just hasn't been done yet [16:16:33] So it bites us that limn0 sits on labs although it's serving production data. [16:16:37] yes [16:16:44] Ok. Then I work around the problem. [16:17:01] you have a geowiki data repo, right? [16:17:05] can you push/pull with that? [16:17:13] And remove the work-around once limn0 went into production. [16:17:24] You mean on gerrit? [16:17:29] yes [16:17:42] We do not have the private data on gerrit, as gerrit is not good at keeping things hidden. [16:17:47] oh priiivate date [16:17:48] data [16:17:49] I do not trust gerrit to do that. [16:17:55] no, it won't [16:17:58] And ^de mon does not trust it either. [16:18:13] Yes private data :-) [16:18:51] Exec[git_pull_geowiki-data]/returns: change from notrun to 0 failed: git pull --quiet returned 1 instead of one of [0] at /etc/puppet/modules/git/manifests/clone.pp:102 [16:18:52] notice: /Stage[main]/Misc::Statistics::Geowiki::Jobs::Limn/Cron[geowiki-process-db-to-limn]: Dependency Exec[git_pull_geowiki-data] has failures: true [16:18:53] warning: /Stage[main]/Misc::Statistics::Geowiki::Jobs::Limn/Cron[geowiki-process-db-to-limn]: Skipping because of failed dependencies [16:20:00] Oh ja. That repo has gone. [16:20:01] root@stat1:/a/geowiki-data# sudo -u stats git pull [16:20:01] fatal: Project not found: analytics/geowiki-data [16:20:18] Is the error fatal for puppet? [16:20:27] not fatal for pupppet, but for the dep chain [16:20:30] the cron job depends on the clone [16:20:33] We could just remove the ensure latest I guess. [16:20:37] and the pull exec fails [16:20:41] yeah that would do it [16:20:44] I'll provide a change for that later today. [16:20:48] Thanks. [16:20:55] orrrr, we could just remove the cron now and delete it manually [16:21:00] do we want to keep this class around at all? [16:21:07] Yes. [16:21:14] The repo will come back. [16:21:19] And it will only hold the public data then. [16:21:23] ohhh ok [16:21:36] but you do not want the job to come back, right [16:21:37] ? [16:22:52] At the end of the sprint, the job should come back. [16:23:06] It will only push the public data to gerrit [16:23:08] ok cool [16:23:17] ok i'll just remove the cron job for now [16:23:18] manually [16:26:58] ok qchris, q for you [16:26:58] what's up with the kraken etl target? [16:27:27] I tried over the weekend, and I failed to to the nexus issues and accessing the kafka instances. [16:27:42] But ... [16:27:59] I saw that it's not just the camus-example target. [16:28:11] Really, we need a custom camus build. [16:28:17] +1 [16:28:18] ? [16:28:26] As we changed the hadoop version underneath camus. [16:28:36] yes, but we can use our fork of camus, no? [16:28:37] So I split up your changes into: [16:28:55] * Parts that are just camus (adjusting hadoop version) [16:29:11] * Parts that are specific to us (your separate classes) [16:29:35] In kraken repo, you'd just go mvn package, and you'll get the shaded jar. [16:29:49] oh its ready now? [16:29:56] No. [16:30:10] Just go on using the jar that you built before. [16:30:22] We'll just change it underneath afterwards. [16:30:23] oh ok [16:30:24] yeah [16:30:46] oh because we need our version of camus in our maven/nexus repo? [16:30:48] Does that approach work for you? [16:30:54] Yes. [16:31:02] yeah its fine, i'm just starting to look into the git deploy bit [16:31:11] can still do that without hte camus part [16:31:12] Oh. Ok. [16:31:47] Just pick whatever name you want for the shaded jar. [16:31:55] We'll make sure, kraken produces that. [16:32:09] Just make sure that it allows for proper versioning. [16:32:14] hm [16:32:26] So kraken-etl.jar would be bad. kraken-etl-0.2.3.jar would be better. [16:33:22] To test, you can just 'cp camus-example-0.x.y-SNAPSHOT.jar kraken-etl-0.2.3.jar' [16:52:35] aye ok [16:52:37] drdee: [16:52:43] yo [16:52:48] what was your vision for kraken / .jar deployment using jenkins? [16:54:40] have jenkins build the jars; have maven manage the version numbers; get jars onto hdfs somehow; symlink versioned jar to unversioned jar; keep track somewhere mapping of report generated and jar version (this is an open question) so it's not like a vision more some thoughts [16:55:10] yeah [16:55:20] the q i'm curious about it get onto hdfs somehow part [16:55:33] i'm also thinking about deploying the whole kraken repository to hadoop nodes local fs [16:55:45] but we don't want to build the jars on those nodes [16:55:49] right? [16:55:50] so [16:56:23] maybe jenkins can rsync jars over to the deployment server [16:56:24] (tin) [16:56:33] and we can use git deploy to deploy the jars with the rest of the kraken repo [16:56:34] ? [16:56:47] we can use some git hooks with git deploy to automatically sync them into hdfs then [18:30:18] phew back, sorry guys [19:48:56] ok, drdee, let's fix kraken jenkins, eh? [19:49:08] sure [19:49:49] do we know what is broken? [19:53:49] i think some of the dclass unit-tests [19:53:50] that's all [19:53:51] hmm, ok [19:54:08] we could disable unit-testing for now [19:54:14] so you can tweak building jars etc [19:54:18] have new dclass debs been deployed to jenkins ? [19:54:24] yes [19:54:38] and unit tests still fail for them, that's a bit odd [19:55:00] qchris and me had a gerrit patchset some time ago updating the dclass code and the tests were passing [19:55:23] drdee_: do the tests pass for you locally ? [19:55:52] https://twitter.com/harej/status/388029808875417601 [19:55:53] <3 [19:55:55] it's because the new openddr lib has changed some fof the outputs [19:56:14] ah, alright [19:56:19] milimetric: ^^ [19:56:20] :D [19:56:25] so they are easily fixable then [19:57:13] :) awesome [19:57:18] :) [19:57:33] so this twitter thing is really catching on huh? [19:57:45] ;) [19:58:15] we've had great success with talking' evaluation via social media, it's quite strange [19:58:42] wikimedians who are all NOTFACEBOOK don't want to participate in our fancy meta portal. they just post on twitter and our Facebook group about this stuff. [19:58:47] it's quite funny [19:59:06] it's really up-side-down world [19:59:27] yeah what started off an experiment has become one of our best modes of communication. if i email them about something, they "never got the email" [20:00:00] but if I Facebook them about logic models they're all on it. [20:02:45] ok drdee, so yeah i don't really want to work on unit tests right now [20:03:02] i want to get jenkins building a jar, that I can somehow hook up with git-deploy [20:03:05] give me a sec -- i can disable that [20:04:13] drdee, how did the kraken project get into jenkins? [20:04:20] is that through the jenkins web gui? [20:04:25] i created it [20:04:25] yes [20:05:48] can I change the jdk on it? [20:05:49] to 7? [20:05:55] openjdk 7? [20:06:09] i am working on it ;) [20:06:12] please wait 1 sec [20:06:13] https://integration.wikimedia.org/ci/view/Analytics/job/Kraken/configure [20:06:46] ok [20:06:49] yeah i'm there [20:06:53] can/should I change it or are you? [20:07:25] consider using zuul [20:07:42] hashar seems to be online [20:08:11] i am working on it right now [20:08:19] k [20:08:49] (CR) Milimetric: [C: 2 V: 2] Add data for Orange Madagascar [analytics/wp-zero/data] - https://gerrit.wikimedia.org/r/88709 (owner: QChris) [20:09:31] ottomata: https://integration.wikimedia.org/ci/view/Analytics/job/Kraken/21/ [20:09:44] it builds the jar while skipping the unit-tests [20:09:53] using openjdk 7 [20:09:57] awesome [20:09:58] ok [20:10:02] i think you are good to go [20:10:06] so, where do those jars go? on gallium somewhere? [20:10:14] can I get them through jenkins or should I just rsync or something? [20:10:31] i guess that's for you to figure out :) [20:10:36] haha ok [20:10:53] oh look! [20:11:01] they are here [20:11:01] https://integration.wikimedia.org/ci/view/Analytics/job/Kraken/21/org.wikimedia.analytics.kraken$kraken-generic/ [20:11:10] i can just dl them maybe, hm [20:11:11] scroll down at https://integration.wikimedia.org/ci/view/Analytics/job/Kraken/configure [20:11:19] there are post build actions [20:11:23] hmm [20:12:56] oo ArtifactDeployer [20:13:50] that sounds like it :) [20:28:22] milimetric: batcave? [20:28:28] sure [20:29:02] drdee_: there [20:32:33] * SarahStierch mutters something about how using twitter to teach someone how to use wiki metrics is very weird. [20:33:04] thank god he moved to IRC. gahhh [20:59:16] have we already killed the former internal analytics list? I accidentally posted to it, as expected :( [21:00:20] I don't think we killed it DarTar, but I think it might just work as a side-effect of the lists.wikimedia.org one [21:01:01] either way, just follow my instructions to delete it from your contacts in gmail [21:01:16] hey ottomata, how am I set up to access the anXX machines? [21:01:39] Ironholds is sayin that having my key stored on bastion/s1/whatever is frowned upon? [21:01:50] I replied: "I don't understand ssh" [21:02:22] yeah you shouldn't manually install any keys [21:02:26] puppet does it when you request accounts [21:03:11] i did never install any key anywhere [21:03:20] so either you or dschoon set it up for me [21:03:42] but where would I check to make sure I'm accessing it "properly" [21:04:12] ottomata: ^ [21:04:32] Ironholds: andrew said this so far: [21:04:33] [17:02:22] yeah you shouldn't manually install any keys [21:04:33] [17:02:26] puppet does it when you request accounts [21:05:15] milimetric: Since otto mata is busy in ops, maybe I can help you? You are asking howe to get across the bastion hosts, without installing your keys there? [21:05:15] https://wikitech.wikimedia.org/wiki/Server_access_responsibilities#SSH [21:05:47] thanks qchris, I have access to all the analytics10xx machines [21:06:06] but the question is, how I got that access and is it violating the policy that ottomata just pasted [21:06:35] aha, thanks for the link, ottomata [21:06:39] because I personally never manipulated my keys in any way other than sending them to andrew / david and attaching them on tickets [21:06:41] that clarifies things somewhat [21:06:50] Ok. [21:07:47] yeah so [21:07:49] they are added in puppet [21:07:55] and puppet puts them in place wherever it is told [21:08:14] http://git.wikimedia.org/blob/operations%2Fpuppet.git/222d1834b08d4f4f935b27fecbfa48798c58782d/manifests%2Fadmins.pp#L2579 [21:09:03] okay, this is confusing as hell [21:09:30] so here's the situation; I have access to stat1. I access this by just sshing in [21:09:43] I have now been given access to the analytics machines. How the heck do I get to them? [21:10:02] noting that I don't currently go through bastion and, moreover, bast1001 is rejecting my key. [21:11:07] Ironholds: bast1001 is bast1001.wikimedia.org? [21:11:19] yip [21:11:22] Ok. [21:11:52] yeah ok [21:12:03] I've been in that situation before. And the team requests access to this bastion host for me spearatedly. [21:12:22] Ironholds: yeah, if you don't have bastion access yet [21:12:23] s/requests/requested/ == They did it for me. [21:12:25] we need to get it for you [21:12:29] you shouild have been already given it [21:12:32] i think probably it was just overlooked [21:12:36] aha [21:12:42] on it... [21:12:45] thankee :) [21:19:06] first comment: https://www.mediawiki.org/wiki/Talk:Analytics/Hypercube [21:24:42] shoot -- I just edited the page -- should I have put comments in talk? [21:24:54] but yeah -- that's really cool [21:26:20] tnegrin: want some hilarity? [21:27:24] I indeed have hive/hadoop access. I know where the machine lives. But in a year and a bit of data-monkey work at no point have I needed bastion access, and so I don't have it, and so I can't get to hive ;p [21:27:33] (it's being resolved, but I thought it might amuse) [21:28:16] almost! [21:28:37] tnegrin: i don't thikn we have any pagecount data in hive yet [21:28:39] should I do that? [21:28:51] wouldn't be automated, but I could load a bunch in and make some tables [21:28:58] I think milimetric does [21:29:07] or Ironholds could do it i guess to [21:29:09] oh ja? [21:29:38] https://gist.github.com/milimetric/6830990 [21:29:51] ottomata: potentially, but I won't be able to until next wednesday (my assigned analytics day) [21:29:51] HDFS_DIR=/user/milimetric/pagecounts/$YEAR.$MONTH.${DAY}_${HOUR}.00.00 [21:30:53] dan got all that stuff working I think -- we just might want to move it out of his homedir [21:30:58] hmm cool! [21:30:59] yeah [21:31:07] Ironholds: it doesn't have referer tho [21:31:08] ok i thikn leaving it in his homedir is good for now though [21:31:17] since we are going to figure out all the job dependency stuff [21:31:18] people have RO access right? [21:31:19] drake or whatever [21:31:22] totally [21:31:58] RO to hadoop? they can write whatever to their homedir, and I think also read webrequest data [21:33:48] do people run as the hadoop user? [21:34:23] I was just wondering if someone other than milimetric can mess with the data [21:39:59] no [21:40:05] they run with their own user [21:40:15] but yes, they should be able to query milimetric [21:40:18] milimetric's tables [21:40:21] tnegrin: ^ [21:44:58] cool -- that's perfect [21:45:05] we can turn Ironholds loose [21:45:33] once bastion access is through, da [21:46:22] it's like editing wikipedia -- if you can't figure out the syntax you shouldn't be editing [21:46:38] ok Ironholds [21:46:44] you have an account on bast1001.wikimedia.org now [21:46:45] try that [21:46:46] can we hook him up today? [21:46:49] awesome!! [21:47:18] ottomata: okay, lemme test this [21:49:25] right, I can get into bastion but now don't have a key available for the analytics machines [21:50:18] ech. I'll muck around with key forwarding and remote agents later [21:50:22] for the time being, I have bastion :) [22:01:37] Ironholds: which machine are you trying to log into? [22:01:42] how did you log into bastion? [22:02:00] you should add a proxy command to your .ssh/config file [22:02:01] probably [22:02:04] ottamata: analytics 1011.eqiad.wmnet; it's just that I haven't set up a config file to proxycommand in [22:02:07] yeah, exactly [22:02:29] but given that I can't work on this stuff since next wednesday, it seems like something that can wait [22:02:31] https://gist.github.com/ottomata/6909260 [22:02:32] mabye [22:04:46] yo drdee [22:04:52] hi [22:05:03] i can't make it -- i cancelled [22:05:06] it's 6pm est [22:05:10] bedtime [22:05:12] oh snap [22:05:16] i told this :) [22:05:25] when did we reached? [22:05:34] ? [22:05:36] reschedule [22:05:49] we did not -- i spoke with kenan about a week ago [22:05:55] ok -- we'll fix it [22:05:59] i don't think things changed a lot :) [22:06:04] so please go ahead without me [22:06:07] ok