[08:21:33] (CR) Nuria: "The problem with undeterministic test in wikimetrics was present much before this change. Do grep for sleeps in test setup in the code, th" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124751 (owner: QChris) [09:05:14] (CR) Nemo bis: "Thanks for this and the other merges, Chris! The rest can wait for Erik, nothing's on fire. :)" [analytics/wikistats] - https://gerrit.wikimedia.org/r/100368 (owner: Nemo bis) [12:40:07] [travis-ci] master/f0e0a8f (#170 by Dan Andreescu): The build passed. http://travis-ci.org/wikimedia/limn/builds/22606147 [12:42:44] (PS1) Nuria: [WIP] Runs of reports that are public should create corresponding files on disk. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124829 [14:02:10] csalvia: Standup :-D [15:17:43] (CR) Milimetric: [WIP] Runs of reports that are public should create corresponding files on disk. (13 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124829 (owner: Nuria) [15:36:00] sooo qchris_away [15:36:00] ah away [15:36:13] wasn't sure where we left of settings vs pom repo stuff [16:00:04] Hey ottomata, what would it take to get gnumake installed on stat1003? [16:01:40] hmm, not sure, whatcha need it for? [16:02:57] Make'n things. [16:03:03] But seriously. I use make all the time. [16:03:18] It's part of my analysis code. I use it to manage dataset dependencies. [16:04:22] yeah i think that should be fine [16:07:59] halfak: stat1003 working ok for you then? [16:08:30] Oh yeah. :) I'm 100% converted save for a couple little issues (e.g. make). [16:08:38] oh make was on stat1? [16:08:41] I haven't logged into stat1 in a couple of days. [16:08:43] Yup [16:08:46] ok [16:08:46] make was on stat1 [16:08:50] welp, now you have make on stat1003 [16:08:51] :) [16:08:55] Thanks! [16:08:55] puppetized this time :) [16:09:38] halfak: are you going through bastion to get to stat1003? [16:10:22] I am with ssh, but I'm still using direct sftp through my editor [16:12:22] (CR) Nuria: [WIP] Runs of reports that are public should create corresponding files on disk. (9 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124829 (owner: Nuria) [16:12:47] halfak: wanna try to figure out that editor thing now? [16:13:19] On a call right now and I have to prep for the next meeting. How's 11AM PDT? [16:14:06] ottomata, ^ [16:15:13] ja shoudl be fine halfak, i have a meeting at 11:30 pdt [16:15:21] but we can probably get it in 30 mins i bet [16:15:26] or at least get somewhere [16:15:33] gotcha. If we can't then we should give up. [16:17:00] (PS1) Milimetric: Address parts of Bug 63680 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124872 [16:24:45] (PS8) Terrrydactyl: Removing cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 [16:24:53] Heya ottomata. About the pom stuff. I think we left at agreeing that all the release settings sholud go into settings.xml, but [16:25:06] for the repository setting, we didn't really agree where they should go. [16:25:19] manybubbles: wanted to put everything into settings.xml. [16:25:32] I prefer to have it in pom.xml. [16:26:02] manybubbles linked to a blog post ... which was a good read. But the arguments it brings do not apply too much [16:26:12] for the projects we have (i.e.: kraken, camus) [16:26:18] aye [16:26:18] hm [16:26:37] I think distributionManagement stuff has to go in pom.xml [16:26:39] for deployment, no? [16:26:44] mvn deploy? [16:26:54] but username, password stuff etc. [16:26:59] aye that goes in settings [16:27:36] hm, i but mg [16:27:41] that is what we are not sure about? [16:27:42] ? [16:27:46] The only thing that I really care about is: A fresh developer should be able to build the project without messing with settings.xml. [16:28:13] does that mean it should use our repo? [16:28:17] always? [16:28:21] Yes. That's repositories in the pom.xml. [16:28:22] yeah [16:28:26] oh [16:28:27] hm [16:28:30] Yes. Always our archiva. [16:28:33] Shouldn't it? [16:28:35] ah hm [16:28:38] yeah i think so too [16:28:40] I mean ... it's our software. [16:28:40] hmmm [16:29:02] If others use it, they can read our archiva as well. So they can build it. [16:29:18] And they can override the mirrors section in their settings.xml if they want to. [16:29:32] So they can force to not use our archiva. [16:29:50] But per default, they should be able to reproduce our builds. [16:30:14] yeah [16:30:16] ok [16:30:21] trying it with camus now [16:30:47] To build straight with only using archiva? [16:31:58] yes, i can already do that [16:32:08] but i'm editing pom.xml and removing some of my settings.xml stuff [16:32:48] Cool. [16:35:41] hm wait but, qchris, if we add just stuff to pom.xml [16:35:50] it won't only use that [16:35:50] we need to add mirror stuff too, right? [16:35:53] to pom.xml? [16:36:00] (can we?) [16:36:22] trying i guess, hmm [16:36:38] By using in pom.xml to override "using central" by "using our archiva" [16:37:02] We achieve what we want without adding mirrors in pom.xml. [16:37:09] Does pom.xml have mirrors? [16:37:14] Downloading: http://repo.maven.apache.org [16:37:22] dunno, but when I removed mirrors from my settings [16:37:22] it does that [16:37:25] (PS1) Terrrydactyl: Added delete wiki user functionality. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124878 [16:37:26] I thought mirrors of settings.xml are repositories of pom.xml. [16:37:31] I have repository stuff in pom.xml [16:37:34] (CR) jenkins-bot: [V: -1] Added delete wiki user functionality. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124878 (owner: Terrrydactyl) [16:37:48] i think maven has a bunch of default repos it knows how to look for [16:37:58] and if you don't override them somehow it will use them [16:37:58] ottomata: And you have central disabled there? [16:38:04] oh, no [16:38:46] trying that... [16:39:01] central [16:39:06] ^ repository id [16:39:30] Override that and set releases and snapshots to false [16:40:04] Should I push that to gerrit so you can just fetch it? [16:43:17] oh, hmm hang on am trying now [16:51:41] hello analytics team. Unrelated, but I just setup jenkins jobs for style checking my java code (with checkstyle). If you guys want to have a checkstyle for your code, it should be easy enough to do now (unlike before). Just a headsup :) [16:54:55] zz_yuvipanda: Thanks :-D There is a long standing change around kraken for that in gerrit. Awesome, if doing checkstyle with Jenkins got simpler now \o/ [16:55:17] qchris: yeah. {name}-maven-checkstyle should just work now [16:55:30] YuviPanda: You rock! [16:56:20] hm, ok qchris [16:56:24] [WARNING] The POM for org.apache.maven.plugins:maven-install-plugin:jar:2.3.1 is missing, no dependency information available [16:56:39] ? [16:56:52] I guess I'll upload my change to gerrit then :-D [16:56:54] http://codeshare.io/yIA48 [16:56:57] ok ok [16:57:10] oh i probably have to add pluginRepo for ours too [16:57:10] ok [16:58:00] (PS1) QChris: Force maven to only use WMF's Archiva [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124882 [16:58:19] ok ees good [16:58:54] Your code does not set a pluginRepository [16:58:57] (CR) Ottomata: [C: 2 V: 2] Force maven to only use WMF's Archiva [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124882 (owner: QChris) [16:59:02] So maven does not know where to get them. [16:59:40] Ottomata ... that change still depends on the unmerged "Adding camus-wmf subproject and setting wmf version to -wmf3" [16:59:56] (Because that was what I thought you were working on) [17:00:12] Should I rebase it on wmf's head so you can merge? [17:00:12] ohoho [17:00:23] i was about to do that, didn't realize it depended [17:00:29] will submit as is [17:00:36] (PS3) Ottomata: Adding camus-wmf subproject and setting wmf version to -wmf3 [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/122563 [17:00:40] Ok. [17:00:56] i think the only remaining issue there was the -coders dep, right? [17:01:00] in camus-wmf [17:01:01] i removed thta [17:01:02] that [17:01:08] I think so too. [17:01:10] Cool. [17:02:36] (PS2) QChris: Force maven to only use WMF's Archiva [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124882 [17:03:00] Had to rebase on master ... so your votes are gone. Sorry. [17:03:01] qchris: you can merge https://gerrit.wikimedia.org/r/#/c/122563 if you like [17:03:08] on master? hm [17:03:15] s/master/wmf/ [17:03:17] Sorry [17:03:21] k [17:03:37] that's fine, i think they can both be merged [17:03:39] in whatever order :p [17:04:07] qchris: , want to do the same for kraken then? [17:04:08] (CR) QChris: [C: 2] Adding camus-wmf subproject and setting wmf version to -wmf3 [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/122563 (owner: Ottomata) [17:04:24] (CR) Ottomata: [C: 2 V: 2] Force maven to only use WMF's Archiva [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124882 (owner: QChris) [17:04:51] bah [17:04:52] Oh no Jenkins. Right. I'll have to test "mvn clean package" [17:04:53] you just +2ed! [17:04:54] ha [17:04:55] ok ok [17:05:02] making some llunch... [17:05:03] :) [17:06:26] (CR) QChris: [V: 2] "Fails to find resolve dependencies for project com.linkedin.camus:camus-api:jar:0.1.0-wmf3-SNAPSHOT." [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/122563 (owner: Ottomata) [17:17:24] milimetric: I hear you merged a patch that gi11es needs on limn0, any chance you're doing that already? [17:17:50] limn1 is the place marktraceur, since the tampa migration [17:17:57] and it's merged and deployed as of this morning [17:18:16] (so 4 hours ago-ish) [17:18:24] ottomata: rm -rf ~/.m2 ; mvn clean package 2>&1 | grep -c 'maven\.org' [17:18:24] ottomata: 0 [17:18:34] ottomata: And build succeeded. [17:19:58] (CR) Milimetric: Removing cohorts from database. (20 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 (owner: Terrrydactyl) [17:25:05] (PS1) QChris: Force maven to only use WMF's Archiva [analytics/kraken] - https://gerrit.wikimedia.org/r/124885 [17:26:47] (Abandoned) QChris: Add WMF's repository for mirrored jars [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/122782 (owner: QChris) [17:28:02] 'kay [17:28:04] Ta, milimetric [17:37:56] (CR) Ottomata: [C: 2 V: 2] Force maven to only use WMF's Archiva [analytics/kraken] - https://gerrit.wikimedia.org/r/124885 (owner: QChris) [17:49:33] halfak: ready whenever [17:52:34] (CR) Milimetric: [WIP] Runs of reports that are public should create corresponding files on disk. (5 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124829 (owner: Nuria) [18:01:54] (PS1) Ottomata: Using 'wikimedia.mirrored' as repository name, like it is elsewhere [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124889 [18:02:00] Hey ottomata! [18:02:09] batcave? [18:02:17] (CR) Ottomata: [C: 2 V: 2] Using 'wikimedia.mirrored' as repository name, like it is elsewhere [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124889 (owner: Ottomata) [18:03:43] heya k [18:31:02] halfak: also, this is working crazy well for me for some other stuff: [18:31:02] https://github.com/apenwarr/sshuttle [18:31:06] it might work for you too [18:32:57] I'll check that out too. Thanks. [18:35:12] (PS1) QChris: Reformat README.md to meet line length limit [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124900 [18:36:56] (CR) QChris: "Original rendering is at" (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124900 (owner: QChris) [19:18:25] (CR) Milimetric: [C: 2] Reformat README.md to meet line length limit (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124900 (owner: QChris) [19:23:54] qchris [19:23:59] I updated the usage docs at https://wikitech.wikimedia.org/wiki/Archiva [19:24:09] wanna look them over and see if they are correct? [19:24:27] Sure. Gotta finish some code discussion. Then I'll look over them. [19:24:36] "correct" ... hahaha :-P [19:24:40] I know nothing about correct. [19:24:52] ha, k [19:24:55] well, what we want [19:24:56] at least :) [19:25:03] I can try that. yes. [19:25:31] k [19:25:32] :) [19:32:40] ok qchris, mind if I make a camus release? [19:32:50] Not at all, sir! [19:32:58] great, 0.1.0-wmf3 comin atcha [19:33:04] * qchris ducks. [19:33:17] we decided to tag, right? [19:33:26] I hope so. [19:33:29] ha, k [19:33:29] (CR) Milimetric: [WIP] Runs of reports that are public should create corresponding files on disk. (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124829 (owner: Nuria) [19:35:28] qchris, not entirely sure of best practice here to tag [19:35:34] commit to wmf branch wihtout -SNAPSHOT [19:35:40] create tag from that commit [19:35:42] checkout the tag [19:35:43] build [19:35:44] deploy? [19:35:49] sound right? [19:36:44] Sounds good. [19:37:42] Maven helps you with part of that IIRC. [19:38:27] mvn release, yeah, i thikn we didn't set that up [19:38:57] Should we set it up and make releases less painfull? [19:39:16] (PS1) Ottomata: Releaseing version 0.1.0-wmf3 [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124959 [19:39:19] mayyyybe [19:39:23] don't wanna right now though [19:39:29] k. [19:39:44] ottomata, FYI: No luck with forwarding via ssh. I'm working from an sshfs now. Ran a couple tests with list connections. I think I've reduced the pain. I'll test this out for a while. [19:39:53] ok [19:40:02] I'm ok with firewalling stat1003 effective immediately. [19:40:03] (CR) Ottomata: [C: 2 V: 2] Releaseing version 0.1.0-wmf3 [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/124959 (owner: Ottomata) [19:51:18] (PS1) Ottomata: Adding artifacts/camus-wmf-0.1.0-wmf3.jar [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/124990 [19:51:39] (CR) Ottomata: [C: 2 V: 2] Adding artifacts/camus-wmf-0.1.0-wmf3.jar [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/124990 (owner: Ottomata) [19:52:50] (PS1) Ottomata: Adding symlink from artifacts/camus.jar -> artifacts/camus-wmf-0.1.0.wmf3.jar [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/124991 [19:53:04] (CR) Ottomata: [C: 2 V: 2] Adding symlink from artifacts/camus.jar -> artifacts/camus-wmf-0.1.0.wmf3.jar [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/124991 (owner: Ottomata) [20:17:11] (PS1) Ottomata: Changing path to camus.jar file [analytics/kraken] - https://gerrit.wikimedia.org/r/124999 [20:26:27] (CR) Ottomata: [C: 2 V: 2] Changing path to camus.jar file [analytics/kraken] - https://gerrit.wikimedia.org/r/124999 (owner: Ottomata) [20:26:44] (PS1) Ottomata: Updating kraken submodule to latest [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/125004 [20:27:34] (CR) Ottomata: [C: 2 V: 2] Updating kraken submodule to latest [analytics/kraken/deploy] - https://gerrit.wikimedia.org/r/125004 (owner: Ottomata) [20:48:42] milimetric: Maybe I should have been paying more attention, but does stat1's public dataset directory still get pushed out to stat1001? [20:49:13] stat1 is/will be no more soon [20:49:22] not sure on the timing of that, ottomata is working on the migration [20:49:34] stat1003 is taking its place [20:49:44] but in theory, yes, that push to stat1001 is puppetized [20:49:56] as are any jobs that used to be running on stat1 [20:50:22] and in theory anyone running any un-puppetized work has migrated their stuff in accordance with the many warnings about that server being migrated out of Tampa [20:50:45] but in practice we're here to help out :) [20:51:04] and also in practice I apologize that data collection is so ad-hoc, we're working on ways to fix that [20:51:49] 's OK [20:51:59] milimetric: We're seeing either no push or delayed push of multimedia data [20:52:15] marktraceur, milimetric: IIRC, I saw a commit fly by that switched that rsync job to stat1003 already. [20:52:25] See http://stat1001.wikimedia.org/public-datasets/all/multimedia/ - I updated these about 50 minutes ago to no effect [20:52:31] marktraceur: it is beling pulled from stat1003 now [20:52:33] The weird thing is they got updated earlier today [20:52:35] ok, marktraceur so if that's true, are your jobs running on stat1003? [20:52:38] really? [20:52:39] I doubt it. [20:52:50] unless I reverted it... [20:52:51] checking [20:53:06] naw, it stat1001 is pulling from stat1003 [20:53:14] OK so I need to go and set that up [20:53:15] so if you want your datasets to be public, you should use stat1003 [20:53:21] But I may or may not have access to stat1003? [20:53:24] milimetric: remind me of the name of the js library you were planning on using to represent visualizations in a JSON file or somesuch? [20:53:30] haha, maybe you should have been paying attention! :p [20:53:31] vega js YuviPanda [20:53:36] this has been on all the mailling lists for weeks now [20:53:41] milimetric: ah, thank you :) [20:53:43] let's see [20:53:54] I saw something go by that was like "THESE ACCOUNTS ARE INACTIVE AND WILL NOT BE COPIED OVER" but my name wasn't on the list so I assumed it was cool [20:53:59] oh ok [20:54:00] milimetric: Maybe the new app visualizations can use the new stuff :) We have very accurate funnels setup [20:54:01] then it is probably cool! [20:54:11] ottomata: The weird thing is I cannot ssh stat1003 from bastion2 [20:54:20] what's your shell username? [20:54:23] Because DNS is cocked up I guess [20:54:28] ottomata: mholmquist [20:54:32] Or...what [20:54:33] marktraceur [20:54:34] Apparently [20:54:37] mholmquist [20:54:49] what is bastion2? [20:55:02] ....maybe I'm mixing up everything [20:55:07] That must be the labs bastion [20:55:13] use bast1001.wikimedia.org [20:55:41] That worked. [20:55:50] \o/ ty [20:56:02] cool [20:56:04] Oh, hm, but they're all running there too [20:56:09] My crontab is working anyway [20:56:12] yup [20:56:13] So I'll just run the jobs manually [20:56:17] all crons have been ported over [20:56:40] Running now [20:56:44] Thanks ottomata, milimetric [20:56:51] yup, thanks [20:59:21] exit [21:09:46] milimetric: gi11es is wondering whether the map update was actually applied to the multimedia-metrics server or what [21:10:03] marktraceur / gi11es, yes, the update was applied [21:10:07] http://superdevoluy.dubuc.fr:5000/dashboards/mmv#geographical_network_performance-graphs-tab on my machine [21:10:09] this morning [21:10:11] http://multimedia-metrics.wmflabs.org/dashboards/mmv#geographical_network_performance-graphs-tab [21:10:12] live [21:10:25] one sec gi11es, production issue, I'll help in a moment [21:10:32] np [21:13:58] and now the map stopped working on my machine O_O [21:16:59] ah, found the issue I think. the new version of the tsv has an empty column on the first row [21:17:20] or rather it's space-separated instead of being tab-separated [21:17:38] just that one row, very weird [21:19:20] marktraceur: I see that in http://stat1001.wikimedia.org/public-datasets/all/multimedia/media-viewer-perf-country-api.tsv [21:20:26] Huh, weird [21:20:44] Wait, no there's not [21:20:47] production issue - solved! [21:21:00] ok guys, have we consensus on what the issue is? [21:21:03] gi11es: The first row has 20 sample size [21:21:06] marktraceur / gi11es ^ [21:21:14] But the std is so small you can't tell [21:22:02] marktraceur: I mean that the "20" is space-separated instead of tab-separated [21:22:39] mmm actually I'm not sure anymore, I need to display named invisible characters [21:23:01] at any rate that file isn't being read correctly by limn, even in "view data as table" [21:23:08] No, it's tab-separated [21:23:32] Oh [21:23:34] Wait. [21:23:36] yeah, I see that now, so it's something else [21:23:43] No...never mind [21:23:49] milimetric: Long story short, no, no consensus [21:23:54] lol you guys are nutty [21:24:03] but the symptoms [21:24:13] the graph works locally but not remotely or is broken on both sides? [21:24:36] a few minutes ago it was still working locally, not anymore [21:24:42] it fails slightly differently though [21:24:51] gi11es: Probably the cache got invalidated, then, and the new version is the issue? [21:24:52] in production you don't even see the map backdrop [21:24:58] marktraceur: I guess so [21:26:10] ok, checking it out [21:26:41] maybe a new country appeared, which introduced a new 2-letter code [21:28:22] one that may be missing from the hacked list and thus cause a bug? I'm just speculating, given that the tsv data looks normal [21:31:34] ok gi11es so the production instance is not updated somehow or is holding onto some cached files [21:31:53] it's thus not splitting on tabs properly using the fix I sent [21:31:58] so that's one issue, fixing [21:33:21] I've spotted a small typo in another tab, I'll send a merge request to mark, but that still doesn't fix things locally for me [21:34:23] marktraceur: https://gitorious.org/analytics/multimedia-limn/merge_requests/4 [21:36:36] ah - my fault was not merging this to the develop branch which is what the limn1 instance is running on right now [21:37:27] ok gi11es, the data is loading fine in production but the graph's still not showing up, looking into that now [21:38:28] milimetric: when I do "view data as table" I notice that it's not filtering the column headings [21:38:40] "country" shows up on its own row [21:38:57] oh yeah that's ok [21:39:23] it should be anyway, the map I got to work this morning had a datafile with that first row as well [21:39:36] Fixed. [21:42:02] milimetric: I mention that because I recall having to skip that first row in my ghetto adapter, otherwise things were broken [21:42:11] and I noticed that the mobile adapter had the same issue [21:42:27] (trying to read the headings as if it were data) [21:42:46] right, both of our ghetto adapters hack in at a lower level so they have to do that [21:42:52] but the rest of the pipeline is fine with it, just double checked [21:43:06] I used your new graph definition instead of my old one from this morning and that broke things, so investigating that now [21:43:16] ah, I know why it stopped working locally, I switched to the wrong limn repo, duh [21:44:16] :) [21:44:17] that'll do it [21:44:28] but there's something up with this new graph definition, just a moment [21:44:42] ...however switching back to milimtric/master didn't fix it... [21:45:32] ok, finally reset everything and it's back: http://superdevoluy.dubuc.fr:5000/dashboards/mmv#geographical_network_performance-graphs-tab [21:45:51] [travis-ci] develop/da2c45c (#171 by milimetric): The build passed. http://travis-ci.org/wikimedia/limn/builds/22645842 [21:45:54] running milimetric/master from github [21:46:58] you can run wikimedia/master or wikimedia/develop, both are fine too [21:49:02] I really don't see what's different with production, I'm running the latest of both limn and multimedia-limn locally [21:49:21] yep, checking as well, datasource and graph both are fine [21:50:45] the js isn't minified locally [21:50:52] I wanted to diff it [21:51:45] you can do coke build and it'll generate minified files [21:51:58] in var/js [21:52:02] "coke build" [21:54:00] that builds the js from the .co files, but doesn't minify it, as far as I can see [21:54:48] right gi11es, I think "coke bundle" is necessary too [21:55:20] but you're right... i copied exactly the setup from prod to local and i don't see the problem [21:55:37] map looks very nice though :) [21:59:57] and is there a way to make the local limn run with the minified files? [22:00:14] yes, you have to put it in "production" mode, one sec [22:00:39] the diff shows no difference [22:00:59] but the minification might still be the source of the problem, once the js is run [22:01:08] yeah, that would suck :( [22:04:13] gi11es: so you have to set the NODE_ENV variable to "production" [22:04:18] it defaults to "development" [22:04:38] but I'm an idiot with bash so I can't find a command to do it - you'll probably figure it out before I bumble my way to it [22:05:40] still works locally [22:05:54] Limn Server (port=5000, env=production [22:06:06] right [22:06:07] and I see the minified files being loaded [22:06:08] :/ [22:08:55] found the problem [22:08:58] http://multimedia-metrics.wmflabs.org/data/geo/example/maps/world-countries.json?1430148102# [22:09:13] versus: http://superdevoluy.dubuc.fr:5000/data/geo/example/maps/world-countries.json?1991191242 [22:14:00] right! gi11es, sorry, that's totally my fault [22:14:27] I sent you that example and didn't realize you'd have to change "example" to "multimedia" or whatever the magic word is in your world [22:15:36] oh... yeah got it [22:15:50] without "example" (limn-data) "mounted" it doesn't work, right? [22:16:54] right gi11es, it's probably better to just add the world country files to your repository [22:17:05] so in case someone messes up the example repo, it doesn't affect you [22:17:17] yep, I'll do that [22:17:30] so you'll need the datasource and the datafile [22:17:55] so datasources/map-world_countries.json [22:18:18] and maps/world-countries.json [22:18:34] and *most* annoying you'll have to change the url in the datasource to replace "example" with "multimedia" [22:18:42] gi11es: ^ [22:18:58] yep, I'm trying that now [22:19:13] heh, god, sorry for this, you guys are the first people to ever use maps actually :) [22:21:59] marktraceur: https://gitorious.org/analytics/multimedia-limn/merge_requests/5 [22:22:42] (PS2) Milimetric: Address parts of Bug 63680 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124872 [22:23:20] Merged, pushing [22:23:50] Deployed [22:24:02] Unless I need to restart something... [22:25:42] still doesn't work, I guess you haven't mounted it under "mmv" [22:25:57] can you check what limn's var folder contains? [22:26:24] oh, no gi11es, it's mounted under "multimedia" I believe, let me double check [22:26:36] heh [22:27:38] yep, multimedia gi11es [22:27:57] and *most* annoying you'll have to change the url in the datasource to replace "example" with "multimedia" [22:28:37] sorry, I saw mmv locally and I didn't recall setting it to that [22:28:50] I went by the dashboard name, I guess [22:29:01] marktraceur: https://gitorious.org/analytics/multimedia-limn/merge_requests/6 third time's the charm [23:00:18] gi11es: Tadaaaa [23:01:48] http://multimedia-metrics.wmflabs.org/dashboards/mmv#geographical_network_performance-graphs-tab great success [23:01:53] thanks for the help milimetric [23:02:09] no problem, nice [23:02:49] wow! france has crazy bad performance [23:03:44] outlier, see the standard deviation [23:04:08] I'll probably need to update the sql query to exclude the extreme percentiles [23:07:48] :) cool, well glad you guys got maps, let me know if you need anything else [23:20:47] Ironholds: you're using the rPython package? [23:21:18] (PS3) Milimetric: Address parts of Bug 63680 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124872 [23:22:39] awight, yup [23:22:53] k... you're right that the python session is not reset between calls. [23:23:10] Which means that any temporary objects will swiftly, ah...expand. [23:23:11] The strange error you're getting is definitely consistent with memory issues. [23:23:18] they should just be bloody overwritten [23:23:21] *waves stick* [23:23:34] Yeah my current guess is that the UAParser is horrible. [23:23:55] what do you mean? [23:24:08] I've never had memory issues in python, even processing truly huge files. [23:24:27] Perhaps there's something stupid in UAParser or your py harness to call it. [23:24:42] awight: uaparser gives you memory issues ? paste a stacktrace somewhere with some details, maybe someone might have a look [23:24:55] average, naw, it's fine for me, just funny through rPython. [23:25:10] I suspect the problem might actually be the repeated file loading. I'll test and debug [23:25:15] but first I need to fill out HSA forms *spits* [23:25:26] haha you'll never see that money again. [23:26:25] Ironholds: can you link me your source for UAParser? [23:27:28] https://github.com/tobie/ua-parser [23:27:32] pretty sure the problem is rPython [23:27:42] if the ua parser wasn't able to deal with 50,000 entries, we'd have an issue ;p [23:27:49] (PS4) Milimetric: Address parts of Bug 63680 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/124872 [23:38:38] Ironholds: try running R under gdb or valgrind... [23:41:00] ooh, that's interesting [23:41:07] Ironholds: actually... I think your error is caused by the python.code string getting free'd. U should try copying the trying before the .C call. [23:41:18] sorry, try copying the python.code string [23:41:23] yep [23:41:30] well, possibly. [23:41:38] so, I've noticed it never happens with the first chunk of input [23:41:45] it *always* happens with the second [23:42:29] doesn't mean that much... [23:42:49] but if it's a rogue free(), valgrind will find it immediately. [23:43:07] I understood some of those words [23:43:51] awight: wouldn't he also need python & R compiled with debugging symbols to be able to find where the leak occured ? [23:44:03] y'know what, fuckit [23:44:10] I'll just re-implement rPython for this script. [23:44:23] that's insane. Just... intermediate filess..... [23:44:24] turn into JSON, stream to file, call python file through system, read through stdin [23:44:29] awight, that's what I'm saying ;p. [23:44:35] I meant the "JSON-to-JSON" bit. [23:44:39] ah [23:44:41] not the "I will build a generalised conduit!" [23:44:44] that way lies madness. [23:44:50] But I will build a conduit for this purpose. [23:44:52] yeah the rPython C is really scary. [23:44:55] it is [23:46:04] Ironholds: fwiw, valgrind does not require debugging symbols in yr objects. [23:46:18] I didn't say it did, average did ;p [23:57:19] I might be wrong, dunno