[11:30:06] (PS1) Nuria: Adding test for cohort uploading for cohort with cyrilic and arabic usernames. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129672 [11:30:08] (CR) jenkins-bot: [V: -1] Adding test for cohort uploading for cohort with cyrilic and arabic usernames. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129672 (owner: Nuria) [12:43:02] (CR) Gilles: "That value gets set as an environment variable in jenkins: https://www.dropbox.com/s/d5xp14qnulcer4s/Screenshot%202014-04-25%2014.41.41.pn" [analytics/multimedia] - https://gerrit.wikimedia.org/r/129566 (owner: MarkTraceur) [12:46:49] springle: checking in [12:46:54] and wondering about the migration [12:53:05] milimetric: db1048 is finishing up pt-table-sync. i'm about to upgrade db1047 [12:53:46] cool springle, that sounds good, did it give you any trouble so far? [12:53:57] i think in restropspect we lost time doing the consumer fancy footwork, but at least we should have avoided data gaps [12:54:19] db1046, an m2 slave, is playing up a bit. but nothing earth shattering [12:56:58] (PS1) Gilles: Add .gitreview file for easier gerrit setup of that repo [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/129679 [12:57:22] yeah, interesting about the consumer fanciness. I think it's great that we didn't lose any data though [13:04:38] (PS1) Gilles: Add .gitreview file for easier gerrit setup of that repo [analytics/multimedia] - https://gerrit.wikimedia.org/r/129680 [13:54:50] wait qchris, you minus 1-ed yourself?! [13:55:38] milimetric: Your stalking me? [13:55:41] :-P [13:55:52] Yes, I -1 myself. [13:56:02] I thought I overlooked a comment from the reviewer. [13:56:16] :D that's actually perfectly sensible [13:56:20] I just never saw anyone do that [13:56:30] qchris: you ok for me to merge that? looks good to me [13:56:45] ottomata: Think so. Yes. [13:57:40] ottomata: about the @ in templates. Do we do that everywhere now? [13:57:45] yes [13:57:58] Ok. [13:58:00] if its from puppet scope, puppet makes it a ruby instance variable inthe ERb [13:58:16] if it is a variable you made in the ERb, then it does not need the @ [13:58:34] I just copy/pasted that part over from the old template. [13:58:38] aye, makes sense [13:58:45] yeah, older versions of puppet didn't do that [13:58:49] we try to do that everywhere now [13:58:54] Coolio. [13:59:06] So are we expecting a puppet upgrade soon? [13:59:15] cool! it works! [13:59:23] \o/ [13:59:25] Yay! [13:59:32] oh, i dunno, i think we are working on being puppet 3 compatible [13:59:39] not sure if we willactually upgrade anytime soon [14:32:34] (CR) Milimetric: "A few nitpicks, you need to rebase, and the flake8 seems to be failing too." (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129672 (owner: Nuria) [14:52:34] (PS1) Milimetric: Remove runTest method which was useless [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129695 [14:56:40] ottomata: would you mind reviewing https://gerrit.wikimedia.org/r/129542 ? [14:56:48] we don't have +2 :) [14:57:06] looking at it now :) [15:00:01] milimetric: that will work, but kinda inelegant...maybe we shoudl abstact the database creation into a define [15:00:03] then you could do [15:00:25] db_name_mediawiki_testing => [ 'A', 'B', 'C'] [15:00:26] ... [15:00:50] wikimetrics::database::create { $db_name_mediawiki_testing: user => $db_root_user, pass => $db_root_pass } [15:00:53] and it would create all of them [15:01:14] shall I have a go at it? :) [15:02:14] ottomata: be my guest if it's not distracting to you [15:02:34] but if you're busy we can do it [15:03:14] but you saw that each grant requires the creation of different databases right? [15:03:20] yes [15:03:33] hang on, 1:1 [15:04:23] sorry! no rush [15:20:24] (PS1) Gilles: Generate matrices for additional pilot sites [analytics/multimedia] - https://gerrit.wikimedia.org/r/129699 [15:20:58] (PS2) Gilles: Generate metrics for additional pilot sites [analytics/multimedia] - https://gerrit.wikimedia.org/r/129699 [15:31:09] (PS3) Gilles: Generate metrics for additional pilot sites [analytics/multimedia] - https://gerrit.wikimedia.org/r/129699 [15:34:51] (PS1) Gilles: Generate dashboards for additional pilot sites [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/129702 [15:36:29] (PS4) Gilles: Generate metrics for additional pilot sites [analytics/multimedia] - https://gerrit.wikimedia.org/r/129699 [16:03:50] nuria: I gotta run grab some lunch [16:04:01] but I made that new dev server, don't know if you saw [16:04:02] me too! [16:04:10] I'll join you. [16:04:15] should be fully functional at https://metrics-dev.wmflabs.org [16:04:16] ok [16:04:27] alredy? wow that was really fast! [16:04:29] csalvia_: you around? [16:04:33] fast?! [16:04:38] one of these days [16:04:40] here's what we're gonna do [16:04:46] you guys spin up a new server using puppet [16:04:57] and I'm gonna set one up from scratch using nothing but my scripts [16:05:03] we'll see who does it faster ;) [16:05:46] 100 times? cause that is the test you know [16:08:47] (PS2) Nuria: Adding test for cohort uploading for cohort with cyrilic and arabic usernames. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129672 [16:09:00] just saying .... [16:41:46] (CR) Nuria: [C: 2] "Indeed now test runs report correct numbers. Tested on vagrant." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129695 (owner: Milimetric) [16:47:23] springle, milimetric: hey, is the migration done? [16:47:29] * ori is just catching up [16:47:43] milimetric: Taking over shinny new sandbox environment for wikimetrics to test some changes [16:48:57] ori: pt-table-sync is still running [16:49:08] rest is done, db1047 upgrade, etc [16:49:53] springle: isn't it a holiday weekend in aus? [16:51:05] yep :) [16:52:10] eep! we owe you one or three [16:52:25] heh [16:52:53] * springle will steal some days to make up [17:00:19] (PS3) Nuria: [WIP]Adding test for cohort uploading for cohort with cyrilic and arabic usernames. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129672 [17:19:12] hey milimetric [17:19:27] why do you only wrap the GRANT execs in the if ($debug) block? [17:19:32] why not the create database statements too? [17:22:42] ottomata: i thought the create database statements would only run if required, like I did from the grants [17:22:51] but if that's not the case, then yeah, they should also be in the debug block [17:23:14] i think if you want a non-debug instance to not create those databases [17:23:18] then they shoudl be in the if block [17:23:18] also - there's concern that the debug block doesn't work at all, because $::wikimetrics::debug is never changed from the hard-coded true [17:23:25] at least on staging that seems the case [17:31:20] on staging it is debug, isn't it? [17:31:24] you have to set it in the labs insterface? [17:31:26] or is that not working [17:36:12] ok, nuria, milimetric, I amended that patch to DRY some stuff up :) [17:36:27] buuuut, i haven't tested it, because my vagrant isn't set up right now [17:36:30] can you test for me :D [17:36:32] ? [17:37:03] i'll test for you ottomata [17:37:07] sorry we're in a meeting :) [17:37:16] np [17:39:34] (CR) Ottomata: [C: 2 V: 2] Updating to version 0.1.2 [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/128952 (owner: Ottomata) [18:02:14] i can test it too, let me know [18:05:17] ottomata cries inside when he sees our puppet changes i'm sure [18:06:01] haha [18:06:03] not crying [18:06:19] but who in any language wants a variable named "variable_2" [18:06:20] ?? [18:06:26] :) [18:07:21] Mr Not Valid, that's who [18:07:52] ottomata: It could be worse. [18:07:59] I once saw a codebase that used a string for multi-line comments. [18:08:14] Ironholds: that's a common pattern in python [18:08:14] assigned to a variable and everything. Just loose variables, hanging around, filled with "so THIS bit of code..." [18:08:22] haha [18:08:25] whoa [18:08:26] ori, as an object and everything?! [18:08:33] not just printed out, but actually assigned? [18:08:35] no, not saved in a variable [18:08:38] phew. [18:09:10] javascript uses them for interpreter pragmas ;) [18:09:18] i.e. 'use strict;' [18:10:21] ottomata: btw, i dunno if you followed the thread w/multimedia about how they're generating percentile figures [18:10:28] it's some awful sql query [18:10:35] it's a perfect use-case for a hadoop job [18:10:45] so that's a good case for the EL data in hadoop [18:10:46] i did notfollow thread [18:10:51] ayyye ok! [18:11:17] I wonder at what point it's easier to use positive rather than negative looking up for excluding API requests. [18:11:25] ori, oh god, that thing? [18:11:32] with the 8 nested subqueries? [18:11:37] it made me sad. [18:13:28] http://opentsdb.net/ looks cool [18:13:35] it's a time-series database built on top of hadoop [18:13:38] opentsdb does look cool! [18:14:42] Ironholds: see http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html to whet your appetite [18:14:58] oh, fun [18:15:22] ooh, it handles multiple time series with non-overlapping points. [18:17:30] https://ja.wikipedia.org/wiki/%E7%89%B9%E5%88%A5:%E3%83%87%E3%83%BC%E3%82%BF%E6%9B%B8%E3%81%8D%E5%87%BA%E3%81%97/cJM3vA5CAq0 what in gods name is this. [18:17:41] the namespace aliases/names? [18:20:29] * ori has no idea [18:22:13] our request logs are a cthulian nightmare. [18:22:41] the fact that Christian and Erik are not currently wearing jackets with extra long sleeves shows they're stronger people than I ;p [19:14:17] awesome dry-ness ottomata :) [19:14:18] works great [19:14:22] awesome! [19:14:47] so the last thing i'm not sure about is the debug thing [19:15:04] if i set debug to false in init.pp, then it seems to work properly [19:15:21] you think its not coming through in the labsconsole interface? [19:15:21] but on staging, where debug is false in the wikitech configure page, it still creates the testing databases [19:15:25] right [19:19:41] ori: from sean's update it sounds like people can use db1047 and be none the wiser about how it's set up at the moment [19:24:02] uhhh, milimetric I don't seeany instance in the analytics project in my wikitech [19:24:16] oh ok, just re-logged in [19:24:18] they are there now [19:24:19] that was weird [19:24:33] that's what happened to me before ottomata [19:24:40] I tried to tell you but you thought I was crazy probably [19:24:42] :) [19:38:24] milimetric: staging doesn't have this change merged [19:38:38] debug = false is coming through to the module though [19:41:05] ok great ottomata, then merge away! [19:41:21] and I'll submit the proper patch to mediawiki-vagrant while you do the operations/puppet one? [19:42:25] ok [19:44:22] ottomata: https://gerrit.wikimedia.org/r/129774 [19:45:38] thank you! [19:45:38] :) [19:46:22] oh, milimetric [19:46:27] i see you modified passwords.pp on staging [19:46:31] so, to keep that sane [19:46:33] what I have been doing [19:46:51] is maintaining a local commit on the wikimetrics self hosted puppet instances that has the passwords change [19:46:59] when you need to update the puppet repo there [19:47:01] git pull [19:47:05] git rebase origin/production [19:47:08] that keeps the local commit at HEAD [19:47:15] then, if you need to make changes [19:47:20] you can --amend them to the same commit [19:47:32] mind if I do that now? [19:47:48] I see the meta_mw keys have changed [19:48:40] yeah, the keys changed, different host needed a different google consumer [19:48:53] but ... uh... i don't get what you're saying yet [19:48:57] ok so [19:49:13] we have to edit passwords.pp and site.pp [19:49:20] directly on the instance [19:49:32] because the passwords shoudln't exist in the public repo, andthere is no private labs repo [19:49:33] right? [19:50:10] i'm in batcave if you want... [19:50:35] coming [19:59:54] (PS15) Terrrydactyl: Add ability to remove cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 [20:00:02] (CR) jenkins-bot: [V: -1] Add ability to remove cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 (owner: Terrrydactyl) [20:13:28] (PS16) Terrrydactyl: Add ability to remove cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 [20:15:23] well, i'm off for the weekend. In case you missed it in -dev, ori, https://bugzilla.wikimedia.org/show_bug.cgi?id=64445 is a thing right now. So long everyone, have a great one. [20:15:39] llaateres! [20:29:16] (CR) MarkTraceur: [C: 2 V: 2] Add .gitreview file for easier gerrit setup of that repo [analytics/multimedia] - https://gerrit.wikimedia.org/r/129680 (owner: Gilles) [20:29:40] (CR) MarkTraceur: [C: 2 V: 2] Add .gitreview file for easier gerrit setup of that repo [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/129679 (owner: Gilles) [20:32:11] (CR) MarkTraceur: [C: 2 V: 2] Generate metrics for additional pilot sites [analytics/multimedia] - https://gerrit.wikimedia.org/r/129699 (owner: Gilles) [20:34:33] (CR) MarkTraceur: [C: 2 V: 2] Generate dashboards for additional pilot sites [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/129702 (owner: Gilles) [20:41:45] Hey, if anyone wantsto take a glance at the analytics/multimedia repo...the scripts are starting to take a long time to run [20:47:39] Partly it looks like it's just volume but I'm not sure if we're doing things we shouldn't or what [22:24:31] tnegrin, DarTar, I just discovered something wonderful. [22:24:39] We still have an entire year of 1:1000 sampled logs. [22:24:53] so...yeah. you want me to maybe do the first 90 days, and then go backwards by 90 day intervals from there? [22:25:04] I actually knew that [22:25:11] then why not mention it? :P [22:25:12] well, not 1 year but isnt that 6 months? [22:25:23] starts on 1 May 2013. [22:25:29] I was not sure it was the same sampled log data you were working witj [22:25:34] yeah, same data. [22:25:45] I just checked the directory. [22:41:56] * ori catches up [23:55:54] (PS1) Terrrydactyl: [WIP] Add ability to global query a user's wikis [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/129858