[01:46:35] New patchset: Milimetric; "more validation and handling of special characters" [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/68041 [01:50:50] New review: Milimetric; "self-merging because of security fix" [analytics/user-metrics] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/68041 [01:50:50] Change merged: Milimetric; [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/68041 [13:31:01] New patchset: Milimetric; "turning off second upload validation to help performance while trusted users are the sole users of the site" [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/68174 [13:32:41] New review: Milimetric; "self-merging a controversial change - feel free to yell at me about this. This turns off the second..." [analytics/user-metrics] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/68174 [13:32:41] Change merged: Milimetric; [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/68174 [13:35:12] mooooooorning [13:40:38] mornig! [13:40:42] I locked myself out of my house! [13:40:44] aahhh! [13:41:12] yikes! [13:41:22] average: around? [13:41:55] average, milimetric, ottomata, erosen: can you please update the mingle WIP wall? i am working on the slidedeck [13:46:40] I put 757 back in analysis drdee, and I made 759 and marked it done [13:46:47] the spaces + underscores thing is hairy [13:47:00] it always is :) but thank you! [13:47:04] but there were three other issues which I've fixed, so I estimated that to about 2 points and marked it Showcasing [13:47:19] it's pretty awesome, we now handle crazy names like: [13:47:45] Chase me ladies, I'm the Cavalry [13:47:51] Xudognik,,999 [13:47:59] Swim Team "Prishtina" [13:48:04] yeah, those are usernames :) [13:48:18] think about that and think about handling them in CSVs :) [13:48:40] (not to mention JSON which is where there were three other bugs) [13:49:31] drdee: the other thing I'll do is finish commenting on robust job queue and namespace_edits, and then it's ready for showcasing [13:50:26] great! thanks man [13:54:37] oh drdee, I just noticed that puppetize kafka is on the same card as debianize kafka [13:54:49] ROFLOL [13:54:58] that's cool, i guess, the originally puppetization is there and was mostly approved, but it will need to be reworked a little bit [13:55:03] for the new package [13:55:10] so that's work to do as well [13:55:18] ?? [13:55:22] we should really try to keep debianization and puppetization as separate cards [13:55:45] not following, how can kafka already been puppetized without the debian package? [13:56:11] it was puppetized for the debian I built originally [13:56:20] well if you look a bit more often at mingle then you would have noticed this a long time ago :D but i am fine splitting it up in separate cards [14:00:01] drdee, i moved dclass debinization stuff into coding and testing [14:00:15] and added status reports to 131 adn 731 [14:00:30] no success running erosen's thing yet for the dupcliate loglines check [14:00:36] will have to tlak to him when he gets online later [14:01:15] cool, thanks for the update [14:01:38] ottomata: would like you to showcase something? [14:01:43] zookeeper? [14:02:32] sure,can you emphasize during your demo why the 'business' should care about this? [14:03:06] milimetric: what would you like to showcase? [14:03:13] erosen: anything you want to demo? [14:03:50] erosen's away for the showcase [14:04:16] drdee: yeah, sorry about the late notice; the grantmaking team is having a team retreat today and tomorrow [14:04:18] we're just about to hangout and coordinate on the plan for today though [14:04:29] erosen:oh yeah i forgot about that [14:04:35] haha, why business should care [14:04:37] ungh [14:04:44] i take it back i don't want to demo anything! [14:04:45] heheh [14:05:04] milimetric: I'm in hangout if you are ready to join [14:05:16] ottomata: too late! i have immutable slidedecks [14:05:35] i'm not sure what I'm going to say. should I justify puppetization? [14:05:56] not puppetization, but really brief explain what zookeeper is and why we need it [14:06:04] hmmmmmMMmMmmMok [14:06:23] and basically that it will improve quality of service [14:06:25] erosen: got 15 mins to help me to dashboard duplicate verificaiton thing? [14:06:25] or something like that [14:06:36] ottomata: yup yup [14:06:44] geez, i will talk about how it is needed for kafka [14:06:45] ottomata: in 15? [14:06:57] i gotta run in 15 to pick up spare keys from friend [14:07:00] iiin 45? [14:07:04] or more like [14:07:05] 1:15? [14:21:18] back in a bit [14:35:18] ottomata are you around [14:56:59] back [14:57:02] erosen, you around? [16:42:21] milimetric: i pushed a first draft of docs for the models module [16:42:35] saw that, good stuff [16:42:41] I just pushed too [16:42:50] moved the queue setup to the __init__ of the tests package [16:42:57] runs in 6 seconds instead of 34 now [16:43:18] though I suppose the demo would've been more dramatic and cool if it was 34 :) [16:43:57] setUpClass vs. setUp is 25 sec. vs. 34 sec. so that's interesting too maybe [16:44:06] erosen! [16:44:10] helpmeh! [16:44:24] I cleaned up everything flake8 was complaining about and pushed, except I avoided the models directory [16:44:41] let me know when the models dir is mine to mess with until you're back [16:44:57] ottomata: hey, sorry can't be more responsive [16:45:02] np [16:45:05] ottomata: i'm in the GLEE team retreat [16:45:09] GLEE!@ [16:45:13] are you singing? [16:45:17] i'm figuring out the schedule now [16:45:25] Grantmaking Learning and Evaluation ... [16:45:45] nope [16:45:45] singing [16:46:40] ottomata: can you give me a gist of hte status / problems? [16:46:48] and then I can maybe attempt to solve things now [16:47:32] i think it can't talk to google? [16:47:56] https://gist.github.com/ottomata/5767060 [16:47:58] erosen^ [16:49:25] ottomata: i think you are right about that [16:49:27] there are two solutions [16:50:05] either we can copy over the cache as a temporary fix (gcat has a caching feature), or we can write a script to scrape the wiki table [16:50:11] which i [16:50:14] ve been meaning to do [16:51:01] i'm fine with temp fixes [16:53:41] k [16:55:30] ottomata: working on a fix now [16:55:33] eta 20 min [17:02:45] k danke [17:02:45] no standup today, right? [17:03:34] oh, i see adrian's email, s'ok [17:31:18] ottomata: try pulling global-dev/dashboard [17:31:21] and rerunning [17:31:37] it just hard codes the gcat part to use the local cache [17:32:15] k [17:32:15] ImportError: No module named openpyxl.reader.excel [17:32:48] pip install —user openpyxl [17:32:49] done [17:32:59] ValueError: carrier Mobilink Pakistan is not present in carrier counts [17:33:03] getting better! [17:33:11] yea! [17:33:12] hmm [17:33:13] oh lots of those [17:33:17] maybe I am doing it wrong [17:33:22] that is okay though, maybe [17:33:24] oh I do have some outputs [17:33:28] it should still run [17:33:36] it just means that some of the carriers it is expecting didn't show up [17:33:49] ah cool! [17:33:50] i have limn files [17:33:59] where can I use them? [17:34:04] yeah, they should all be in /data [17:34:08] or ./data [17:34:15] or can we just compare them or something? [17:34:23] to what? [17:34:27] to existing ones? [17:34:32] or we can graph [17:34:39] eh, probably best to graph… but not sure [17:34:48] i mean if you want to just inspect [17:34:51] oh I don't have carrier [17:34:52] just country [17:34:59] ja lets graph [17:35:01] the _versions.csv should be a good start [17:35:12] ottomata: no carrier, really? [17:35:16] i'm not sure that will work [17:35:32] i mean, i have the carrier log files [17:35:32] just no output from dashboard [17:35:38] in datafiles [17:35:47] or datasources [17:36:02] i think i got errors for all of the carrier names [17:36:21] ah [17:36:21] File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime [17:36:22] (data_string, format)) [17:36:22] ValueError: time data '2013-05-01' does not match format '%Y-%m-%d_%H' [17:36:23] hm [17:36:28] [ERROR][__main__ ][MainProcess ][ load_counts:0162] exception caught while loading cache file: carrier.05.log [17:36:52] use the --carrier_date_fmt command to make it fit [17:36:55] ja [17:36:58] sorry there are so many steps [17:36:58] reruning [17:37:31] i htink it worked, but [17:37:35] File "dashboard/mobile/run.py", line 383, in make_version_graph [17:37:35] "Wikipedia mobile sites." % carrier.name [17:37:35] AttributeError: 'Graph' object has no attribute 'graph' [17:37:51] hm [17:37:53] hmm, maybe pull on limnpy? [17:37:54] but i have carrier stuff now [17:38:00] interesting [17:38:02] limpy up to date [17:38:08] maybe i need to push [17:38:32] can you try now? [17:39:08] i think that should fix it [17:39:50] looks good! [17:39:51] ok! [17:39:53] now what? :) [17:40:16] easiest way to test is to use a local limn install [17:40:22] and put the output in the appropriate location [17:40:25] var/data [17:42:15] ergh local limn install you say [17:42:23] yeah [17:42:37] i mean you can push to a live limn-instance somewhere [17:42:52] npm install [17:42:52] npm ERR! Error: No compatible version found: coco@'=0.9.0' [17:42:57] hmm [17:43:00] oh i guess I can use one of the dev instances [17:43:04] yeah [17:43:27] if you add all the data to reportcard-dev or something, i don't htink it will overwrite anything [17:43:43] it will just create /dashboard/ urls [17:44:09] do I need the mnn_mnc.csv file? [17:44:10] ottomata: is hyphen-delimited, so orange-kenya, digi-malaysia [17:44:26] ottomata: just to generate the dashboard [17:44:32] hi drdee [17:44:34] hi ottomata [17:44:37] hiya! [17:44:44] ottomata: I sent an e-mail and published some debs [17:44:54] ottomata: could you have a look at them please ? [17:45:02] debian tag was also made for 2.0.14 [17:45:06] sure, gimme just a few, will do [17:45:17] ok [17:45:19] thank you [17:45:53] ottomata: can you really not install limn on your mac? [17:47:49] npm install didn't work [17:47:50] that's all I konw [17:47:52] i'm in master branch [17:47:56] k [17:48:12] ottomata: do we have a jdk package inside apt.wikimedia.org ? [17:48:24] ottomata: my question is actually what jdk deb is being used for the kraken machines ? [17:48:41] we'll be using sun oracle jdk 6 [17:48:45] only because hadoop requires it [17:48:51] we'd much prefer to use openjdk 7 [17:49:26] ottomata: how/when will the decision between openjdk7 / oracle_jdk6 will be made ? [17:50:57] as soon as CDH is openjdk compatible then we will migrate [17:50:59] we don't have a choice rigiht now [17:51:47] erosen: do I have to do something to the limn instance to make it see those files? [17:51:48] restart it? [17:51:58] nope [17:52:00] as long as they are in the right place [17:52:03] which can be tricky [17:52:31] hmmm looks not right :) [17:52:31] http://dev-reportcard.wmflabs.org/dashboards/tata-india [17:53:00] ottomata: is jdk installed on machines through the .tar.gz from oracle ? http://goo.gl/TfUlf [17:53:18] ottomata http://dev-reportcard.wmflabs.org/data/datafiles/gp/tata_india_monthly_wp_view_by_version.csv [17:53:27] no average_drifter from a .deb [17:53:30] i think it is in our apt [17:53:31] ok [17:53:47] http://apt.wikimedia.org/wikimedia/pool/universe/s/sun-j2sdk1.6/ [17:54:07] wazzat [17:54:07] ? [17:54:09] gp/ [17:54:10] ? [17:54:14] erosen: ? [17:54:15] yeah [17:54:22] it needs to be a in sub dir [17:54:33] oh, ha, hm [17:54:33] within each of hte datafiles, datasources, graphs dashboards dir [17:54:35] k drdee, I moved the cards into showcasing, commenting is done for the most part [17:54:38] datasources too? [17:54:46] thankyou! [17:54:46] but the data dir created by run.py should do that [17:54:51] no [17:54:55] its data/datasources/...csv [17:55:01] hrm [17:58:28] ottomata: i've always been kinda confused about this level of organization [17:58:50] maybe milimetric can help? [17:58:52] i dunno [17:58:57] yeah, [17:59:01] i can figure it out [17:59:09] but I'm not able to focus fully right now [17:59:11] average_drifter: i think libdclass0-jni should just depend on default-java or default-jdk or whatever [17:59:16] i think that should be fine, no? [17:59:23] yeah and we have showcase starting now [17:59:50] ottomata, average: sprint demo [18:34:38] ottomata: ok, I'm going to change that [18:34:45] have to change places soon, heading out [18:35:00] I can attend the demo muted [18:36:48] http://dev-reportcard.wmflabs.org/dashboards/orange-kenya [19:23:36] back [19:23:50] updating and closing 716 and 738 [19:24:00] drdee , rounce123 looking for other tickets to work on [19:24:28] ottomata: or we probably have to pass this through Faidon right ? [19:24:43] joining #wikimedia-operations [19:25:57] oh yeah totally will [19:26:03] i'm about to start looking at it too [19:26:04] ummm [19:26:09] oh [19:26:12] average_drifter: [19:26:14] in the .dirs files [19:26:24] you should leave off the first / in the path [19:26:29] ok [19:26:30] to make it relative to the builddir [19:26:35] oh, alright [19:26:41] same for in .install [19:26:44] src/*.h /usr/include/libdclass [19:26:45] shoudl be [19:26:47] src/*.h usr/include/libdclass [19:27:03] also, I betcha faidon will say we need to explicitly list all of the .h files, but i'm not sure [19:42:15] also [19:42:16] average [19:42:27] uhhh, we should be pushing the debian/ folder for review [19:42:34] not directly to the repo [19:42:43] average_drifter: ^ [19:45:54] oh also [19:45:59] average_drifter: i recently learned this [19:46:10] you only need to put empty directories in .dirs files [19:46:25] those that won't have anyting installed in them by dh_install [19:47:06] and I don't think you have any empty dirs [19:48:42] ahhhhhh, also you need to push the tags to the repo [19:48:44] I don't have them [19:48:49] thats why I was buiding 2.1.1 or whateer [19:48:59] average_drifter: ohh wo has du gegangen? [19:49:06] hast* [19:54:31] ottomata: ich bin hier [19:55:27] ottomata: ok, then I will backup everything debian-related and send it all as a gerrit patchset [19:58:24] ja ees good [19:58:44] oh and we need the .shlib file [19:58:58] that one I don't have [19:59:05] I should make it [19:59:28] was counting on dpkg-shlibdeps to make it, but I think I have to do it manually [19:59:39] mine says this [19:59:41] libdclass 0 libdclass0 (>= 2.2.1-1) [19:59:46] you shoudl change version number [19:59:52] oh also [19:59:56] .links files [19:59:57] right? [20:00:04] i have a libdclass0.links file [20:00:05] that has [20:00:06] usr/lib/libdclassjni.so.0.0.0 usr/lib/libdclassjni.so [20:00:06] usr/lib/libdclassjni.so.0.0.0 usr/lib/libdclassjni.so.0 [20:00:08] oh [20:00:16] but htat would be for libdclass0-jni.links [20:00:51] oh yeah [20:00:53] the package name [20:00:57] for the jni package [20:00:58] should be [20:01:02] libdclass0-jni [20:01:10] i'm about 90% sure on that [20:01:15] hmmmm [20:01:17] hmmmm [20:01:18] maybe [20:01:34] actually [20:01:42] the package names on other -jni lbs don't even have the 0 [20:01:43] soname [20:01:47] libtritonus-jni [20:01:47] etc [20:01:59] libcephfs-jni [20:32:22] I'll be out for a bit, but back later [21:40:29] ottomata: LGTM on Kafka debian package!!! [21:45:54] yupyupyup [21:50:12] so who is going to give a +2? [21:50:55] we want alex to look over the final stuff [21:51:04] once he +1s or +2s we are good [21:51:26] YAYAYAYAAYAYAYAYAAYY! [22:03:06] hey milimetric [22:03:07] yt? [22:18:51] whatsup DarTar? [22:19:00] hey [22:19:13] I had question about Limn [22:19:33] just wanted to double check if I can only add but not modify datasources from the UI [22:19:55] it looks like I can directly modify the file the datasource form is writing into [22:20:57] yeah I guess milimetric has a definitive answer, sorry [22:21:18] hey DarTar [22:21:25] aloha [22:21:44] > DarTar: just wanted to double check if I can only add but not modify datasources from the UI [22:21:52] > DarTar: it looks like I can directly modify the file the datasource form is writing into [22:21:53] yes, you can only add right now [22:21:56] ok [22:22:06] though modify is really really easy [22:22:19] although, the trouble with all the editing features is [22:22:23] ok I guess I'll do that then [22:22:30] once you edit stuff, you'd have to commit it to the repository somehow [22:23:01] because it just creates files on the limn instance, but doesn't talk to git at all [22:23:20] so this is why we wanted to do the "save limn graphs/datasources into mediawiki" story [22:23:23] but it got deprioritized [22:23:27] so is the file kaldari is referring to here revision controlled? https://wikitech.wikimedia.org/wiki/EE_Dashboard [22:23:41] either that or "save limn graphs/datasources into a database" would make editing a lot more intuitive [22:23:52] yes [22:23:54] "Note: If you need to edit a data source after you create it, you can edit the raw file at /srv/ee-dashboard.wmflabs.org/limn/var/data/datasources/ on the kripke labs server." [22:24:10] right, so now it's not there anymore, we moved to a new labs server [22:24:12] let me update that [22:24:24] yeah, we could;t find kripke any more [22:25:06] ryan lane blew up kripke for us :) [22:25:30] where do those files live now? [22:25:36] poor saul [22:27:07] ok DarTar & kaldari, I've updated the location [22:27:16] it's on limn0 at [22:27:24] /var/lib/limn/ee-dashboard/data-repository/datasources/ [22:27:30] fixed this section: https://wikitech.wikimedia.org/wiki/EE_Dashboard#2._Add_new_data_source [22:27:36] cool, thanks! [22:28:00] great thx [22:28:08] yep, lemme know if you have any more questions [22:28:16] i'm making dinner but it pings loudly :) [22:28:53] the only q is if we can manage the instance from E2's side, I think last time I checked my permissions were broken [22:29:07] ee-dashboard, that is [22:29:40] manage? [22:29:41] basically just to be able to pull and deploy changes to the json config files [22:29:58] oh, yeah, anyone that has access to limn0 should be able to do that [22:30:08] and we can give anyone access [22:30:44] ok, do you mind adding the following: rkaldari, bsitu, mmullie, dtaraborelli ? [22:30:50] it's just "fab ee_dashboard deploy.only_data" [22:31:03] kaldari ^^ [22:31:06] sure, i'll add you guys right now [22:31:14] obrigado [22:31:54] milimetric: I'm actually kaldari for ssh [22:31:58] rather than rkaldari [22:32:49] hm, kaldari and bsitu are already added [22:32:50] oh right, and I'm dartar [22:32:55] :) [22:33:02] milimetric: probably because they were on kripke [22:33:06] but I can't add anyone, so maybe only ottomata has rights to do that [22:33:17] i can try doing it [22:33:18] hold on [22:33:45] https://wikitech.wikimedia.org/wiki/Special:NovaProject drdee [22:33:57] we'll also need ebernhardson for Erik [22:35:19] ebernhardson has been added [22:35:42] DarTar what is your labs username? [22:35:52] dartar [22:36:09] "Failed to add dartar to analytics." [22:36:26]