[08:59:10] qchris, hola [08:59:16] hi nuria [08:59:28] regarding the symlinks [08:59:43] are you working on that? or should i grab it? [08:59:47] There were two things [09:00:01] 1. enhance backaup to deal with symlinks [09:00:14] 2. make admin script create relative symlinks [09:00:34] I am working on oozie stuff, so feel free to grab it. [09:01:21] ok, the actual backup is deployed to production right? [09:01:28] Yes. [09:02:18] ok, will start working on that after i look at e-mail. thank you [10:56:46] (PS1) Nuria: Making pretty report symlinks relative [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/144439 (https://bugzilla.wikimedia.org/66087) [11:26:00] qchris, as far as i can see the backup logic tars with fcz so relative symlinks are working with backup ... [11:26:08] am i missing something ? [11:26:34] Symlinks will make it into the backup. Right. [11:26:57] But we should guard against symlinks that point outside of the public-files directory being present. [11:27:17] As for those, only the symlinks, not the content at the symlink target would get backed up. [11:27:49] For example, before adapting the absolute symlinks to relative ones, [11:28:29] The backup would contain symlinks to /srv/wikimetrics/..., but not the files in /srv/wikimetrics. [11:28:57] Hence the symlink underneath /srv/wikimetrics/ that points back to the /var/lib/wikimetrics would be missing. [11:29:11] And the same thing may happen with other symlinks that we add over time. [11:29:38] That was why we added "Backup's symlink behaviour is undefined" when we merged the backup scripts. [11:37:00] I see, but , provided that symlinks link inside var/lib/wikimetrics [11:37:05] all works right now [11:37:20] ^qchris [11:38:49] Do you want to dereference the symlinks before taring? [11:56:51] I'd not dereferencing symlinks, as it would remove the structure we're having. [11:57:09] And restoring such backups would make things harder. [11:57:38] Logging the symlinks that point outside of the public-files directory would be sufficient, [11:57:50] because then we'd at least get to know when symlinks sneak it. [11:58:31] Thinking about how to proceed if symlinks pointing outside of public-files occur. [11:58:54] I guess I'd not abort backup, but go on backing up. Just in case :-) [11:59:01] But YMMV. [11:59:34] Maybe we could dereference only links that point outside of the public-files directory and log this dereferencing. [12:00:11] Thereby, we'd have a proper backup even in case of outward pointing symlinks, and would be able to reconstruct things. [12:03:17] I actually do not think we should make provisions to back up functionality we do not have [12:03:39] we do not have any functionality making links out side /var/wikimetrics/lib [12:04:56] You made the same argument two (?) weeks ago that we do not have symlinks at all :-) [12:05:03] And that we'll never get them. [12:05:09] Now we have them. [12:05:18] I am fine if we do not dereference anything. [12:05:20] hey! good morning [12:05:25] what are we arguing about? [12:05:31] But we should log a warning, if we encounter outward pointing symlinks. [12:05:40] milimetric: symlink handling around backup. [12:05:55] right I saw you re-opened the bug [12:05:58] Otherwise, we'd not have working backups, and we'd find only when we have to recover from a backup. [12:06:52] qchris: could you review nuria's patch that changes our symlinks to relative? [12:07:16] I'm knee-deep in oozie stuff and have no clue how to run the admin script. [12:07:45] But yes, certainly. [12:08:04] you can just +1 it if it satisfies the relative thing, I can test it thoroughly locally [12:08:17] Qchris is very right that symlinks should be relative, i did changed that . [12:08:27] sorry, qchris [12:08:35] yep, I agree, I'll +2 the change once it works fully [12:08:41] milimetric: Ok. Will do. [12:08:44] but I just wanted a +1 from him since he brought up the issue [12:08:53] k, fantastic, (testing now) [12:17:43] (CR) Milimetric: [C: 2] "nice, symlinks are now relative and they find the report target once it exists." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/144439 (https://bugzilla.wikimedia.org/66087) (owner: Nuria) [12:17:53] (Merged) jenkins-bot: Making pretty report symlinks relative [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/144439 (https://bugzilla.wikimedia.org/66087) (owner: Nuria) [12:18:08] doh, forgot jenkins merges things [12:18:21] well, anyway, qchris you can -1 it if it's no good - but it works as designed [12:19:02] It does? Coolio. I was just looking at it, but ... well ... if it does work, let's merge it :-) [12:19:33] Is there a instance that has such links already? [12:22:49] Oh. Now I get it. Both parts of the "relative" computations happen through the symlink at /srv/wikimetrics/wikimetrics/static/public/ [12:23:07] Great! [12:23:18] both parts? [12:24:00] Yup. "path" and "symlink_folder". Aren't they? [12:24:44] (The file system layout fooled me into thinking one of them starts in /var/lib/ ... Stupid qchris) [12:33:22] Analytics / Tech community metrics: Bugzilla response time: "Longest time without comment" is actually "Longest time without any comment by non-reporter"? - https://bugzilla.wikimedia.org/67589 (Andre Klapper) p:Unprio>Lowest [12:33:24] Analytics / Tech community metrics: Bugzilla response time: "Longest time without comment" is actually "Longest time without any comment by non-reporter"? - https://bugzilla.wikimedia.org/67589 (Andre Klapper) NEW p:Unprio s:enhanc a:None I might be overly picky and complicating things, so... [12:34:24] oh btw, qchris / nuria, in case you didn't notice wikibugs is pinging our channel now for any analytics component bugs. The people in -dev got sick of our greatly increased bugzilla activity :) [12:34:46] so Yuvi changed the bot, and said we could tone it down anytime we needed (the change is very simple) [12:34:46] Yes, I noticed. I like it. [12:34:53] yeah, I'm not offended so far :) [12:35:09] No, it's fine. I'd vote for keeping it. [12:35:24] (PS4) Milimetric: Added projects to csv output [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138474 (owner: Terrrydactyl) [12:37:22] (CR) Milimetric: [C: 2] Added projects to csv output [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138474 (owner: Terrrydactyl) [12:51:14] (PS7) Milimetric: Remove limit on recurrent, add throttling [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142007 (https://bugzilla.wikimedia.org/66841) [13:28:41] nuria, so anything left with this: https://gerrit.wikimedia.org/r/#/c/142007/? [13:29:13] no, I do not think so , just let me check [13:29:15] also, this: https://gerrit.wikimedia.org/r/#/c/144159/ ? [13:29:27] (PS2) Milimetric: Fix test that adds existing tag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/144159 (https://bugzilla.wikimedia.org/66671) [13:30:14] i'm trying to clean up 'cause we should review / merge the growth team's stuff this next sprint [13:37:55] (CR) Nuria: [C: 2] Fix test that adds existing tag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/144159 (https://bugzilla.wikimedia.org/66671) (owner: Milimetric) [13:38:04] (Merged) jenkins-bot: Fix test that adds existing tag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/144159 (https://bugzilla.wikimedia.org/66671) (owner: Milimetric) [13:46:05] milimetric: Let's talk about this one after stand up: https://gerrit.wikimedia.org/r/#/c/142007/7/wikimetrics/schedules/daily.py [13:46:48] as the throttle to be effective has to be overall, not per report.. but maybe those are changes we can do on a subsequent change [13:51:57] anybody around? [13:53:00] DarTar, could you add me to the analytics labs? I would like to look at how zero graphs are set up, and possibly copy a few things [13:53:21] hey yurikR [13:53:27] hi [13:53:50] you should ask milimetric or ottomata to set you up [13:54:03] thx [13:54:09] np [13:54:32] DarTar, have you guys figured out which graphing platform we want to move to? [13:54:40] hey yurikR, the zero stuff is set up in a customized repository by qchris [13:54:44] limn is supposedly heading out... [13:54:47] I'll add you, but be careful :) [13:54:56] I think the dev team is still evaluating a few options [13:54:56] milimetric, thx, won't break it... i hope :) [13:55:01] * DarTar waves at milimetric [13:55:25] milimetric, i saw both repos, but i didn't see the actuall limn files - are they in a separate repo? [13:55:32] ok, yurikR, you're added [13:55:36] thanks! [13:55:50] I have standup now, but I'm glad to explain a bit after [13:55:58] do you know if qchris configured it by hand? [13:55:59] it's somewhat complicated [13:56:07] np, i actually should catch a few hours of sleep [13:56:14] * milimetric waves at DarTar [13:56:20] :) [13:56:23] be back around 10 pacific [13:56:25] am [13:57:02] yurikR: The limn files are in the analytics/wp-zero/data repository [13:57:17] nuria: speaking of glamour, I’ll save you the details of traveling to Milan on a packed regional train with no air conditioning and stuck for 30 minutes in the middle of the countryside ;) [13:57:21] Setup was not done by me, but I took it over when I joined the team. [13:57:21] I love my country [13:57:41] qchris, yes, i saw both of the repos, just wasn't sure where the actual limn files live and how it all ties together [13:57:45] So I am not sure how it was done initially. But there is no puppet or such. [13:58:14] DarTar: as long as the "countryside" does not include nuclean power plants to look at ...it is really no so bad .. [13:58:18] yurikR: Glad to explain after standup. [13:58:24] qchris, no rush :) [13:58:43] we have a big q-review, wanted to show a few good graphs :) [13:58:54] yeah, we fixed that problem a few decades ago [14:00:12] and I get to eat veggies locally grown by my father (and use 4 extra hands to take care of the girls), so life isn’t that bad [14:39:02] (PS8) Milimetric: Remove limit on recurrent, add throttling [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142007 (https://bugzilla.wikimedia.org/66841) [14:41:28] yurikR: so currently it is: (Varnish TSVs) -> ($GLUE_AND_PREPARATIONS) -> (processing through analytics/wp-zero) -> (analytics/wp-zero/data) -> (limn1 labs instance) [14:41:41] Which part are you interested in? [14:42:21] qchris, i have some data and a python script that can massage it into whatever is needed. I need to put it online :) [14:43:21] Ah. Ok. milimetric and nuria are currently working to building a new visualization system. [14:44:07] yurikR: I'm around now to help explain anything, so feel free to ask [14:45:08] qchris, milimetric, so here's my problem: the raw data is very sensitive - its the actual search queries performed by SMS [14:45:17] oh i see you want to sleep :) i'm around all day, anytime's fine [14:45:24] right [14:45:31] i have a script that downloads data from S3, and parses it [14:45:42] if you can suggest where i can run it [14:46:02] and what format to output data to have the quickest path to market, lots of ppl will be very happ [14:46:04] y [14:47:03] ok, i see. [14:47:18] most of my graphs are simple multi-line ones [14:47:39] downloading and parsing S3 data -> that should go into a new instance on labs for fastest results I think. We wouldn't want to put it on limn1 since that runs people's dashboards and we don't want to make it do other stuff [14:48:30] so, if the result of the parsing can be a limn datafile with multiple columns, there's a really simple way to build a dashboard on top of that [14:48:30] milimetric, what about locking it down? [14:48:51] locking limn1? [14:49:05] not sure I follow [14:50:45] milimetric, the data is fairly sensitive, i would want the fewest possible ppl exposed [14:50:55] oh, I see [14:51:10] limn graphs are going to expose the data [14:51:19] even though we encrypt the phone numbers (with salt that we don't know) [14:51:23] because you can just click on the datafile download link [14:51:47] oh, wait, so is the source S3 dataset sensitive or the output after parsing into a datafile? [14:51:54] the source [14:51:57] oh, ok [14:52:14] so that's up to you, you can even do that locally on your machine if you want [14:52:18] Analytics / Wikimetrics: Wikimetrics backup should warn about symlinks pointing out of /var/lib/wikimetrics - https://bugzilla.wikimedia.org/67597 (nuria) NEW p:Unprio s:normal a:None Wikimetrics backup should warn about symlinks pointing out of '/var/lib/wikimetrics'. The '/var/lib/wikime... [14:52:19] limn doesn't care where the data is coming from [14:52:34] no thanks, already did that, wasn't too happy ) [14:52:44] and it can read datafiles from one of two places: a github repository it pulls from when you have changes, or a remote datafile [14:53:20] milimetric, if i set up a labs instance that will serve up the data files, how hard is it to configure a few limn pages? [14:53:42] easy, as long as the datafiles are in the proper format [14:53:59] (csv with dates as the first column) and pretty labels on each column [14:54:13] the datafile name -> graph name [14:54:20] the column header -> legend label [14:54:41] if that is so, limn dashboard looks like this: [14:54:51] milimetric, yes, understood, will create data, no prob. [14:55:17] another q - can the url be of form /.../ [14:55:23] without those numbers published anywhere? [14:55:37] ok, yurikR, just ping me when you have data and I'll set up the dashboard for you [14:56:23] not really yurikR, because limn will have the link to download the datafile [14:56:29] * qchris throws in https://github.com/wikimedia/limnpy which helps a lot around wp-zero to create the needed limn structure. [14:56:29] that's why i was asking if the parsed data is sensitive [14:56:31] the idea being is that we want to be able to give those links to partners, but they should not be abe to browse each-other's daata [14:56:49] qchris: metadata shouldn't be required here [14:57:05] security through obscurity type of deal - you have to know url to get the data [14:57:29] oh, yurikR, you could name your dashboards something non-discoverable [14:57:47] milimetric, the problem is that at this point all the dashboards are in git [14:57:55] which makes them somewhat easily discoverable :) [14:58:12] we leave the dashboards only on limn1, we don't need to put them in git [14:58:27] and we back them up separately so we don't lose them [14:58:59] (sorry that sounds like that's what we do with everything - I mean, that's what we can do for your case here to help obscure) [14:59:09] ok, milimetric, how about this - i will generate a few sample files, and if you can set up a few dashboards with them in a place where i can further tweak them? [14:59:21] yes, let's start that way yurikR [14:59:52] milimetric, thx, will produce something shortly... hoping to have some rudimentary stuff for tomorrow's q review meeting [15:00:01] (hence haven't slept yet :( [15:00:34] yeah, I understand you're rushed, I'll try to help as I can [15:00:45] note: sleep good [15:02:34] qchris [15:27:08] hi nuria, pong. [15:28:42] I though i'd ask 1st .. which is the best shell utility to get absolute paths from relative ones? [15:28:58] realpath? [15:29:43] no clue. [15:31:21] nuria, realpath isn't installed everywhere by default [15:31:23] its a special package [15:31:25] ummm [15:31:38] ahammm [15:31:46] But maybe you do need that. [15:31:59] (If it's around the backup thing, you can do without it) [15:32:12] readlink -f [15:32:13] probably is good [15:34:05] aha [15:34:59] thanks ottomata...very handy [15:37:28] ok ottomata, thanks for merging the puppet stuff [15:37:42] the throughput alarms for "schema" counts are fixed [15:40:31] Analytics / Wikimetrics: Path to recurrent report (from dashboard) should have better semantics - https://bugzilla.wikimedia.org/66087 (nuria) PATC>RESO/FIX [16:44:06] (PS9) Milimetric: Remove limit on recurrent, add throttling [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142007 (https://bugzilla.wikimedia.org/66841) [17:36:23] milimetric, qchris sent you some funky graphs :) [17:36:35] * qchris checks email [17:37:46] i'll check in a moment [17:37:56] thx :))) [17:43:51] yurikR: so this sounds like you won't need limn [17:44:08] milimetric, ?? [17:44:12] Meh ... me is still fetching emails :-/ [17:44:30] you made the graphs and you want to host them somewhere right? [17:44:44] why don't we just put them in some obscure directories and serve them out of apache or something [17:44:46] milimetric, no no, those are old excel graphs [17:44:58] proof of concept [17:45:16] now i have a more stable python code to generate data for lots of diffirent cases [17:45:32] and i can finally automate it [17:45:52] the data.zip file is the new stuff [17:46:02] ah, ok [17:46:15] i'll take a look at the data and ping you in a moment [17:46:21] neither the pdf nor excel is needed, just so that you have some idea of what i'm pursuing [17:52:01] Ugh, does someone have an SSH setup for stat1001 that works that I can check mine against [17:52:38] marktraceur, why are you using stat1? [17:52:40] Or check https://dpaste.de/GLd6 [17:52:46] I'm not Ironholds, stat1001 [17:52:51] aha [17:53:17] permission denied, or...? [17:53:24] Yeah, publickey error [17:53:37] bastion works [17:53:46] Haven't tried tin or gallium yet [17:53:54] tin's fine [17:54:07] I guess, has anyone checked stat1001's still there today? [17:54:11] :) [17:55:54] I guess I don't need it [17:56:05] Just wanted to check if we were generating datasets for UW and it seems we aren't [17:56:10] * marktraceur writes *those* first [17:56:25] huh [18:03:48] Analytics / General/Unknown: Route SSL traffic again into webstatscollector's filter process - https://bugzilla.wikimedia.org/67456#c4 (christian) PATC>RESO/FIX The first hourly webstatscollector files after the above merge contain ssl requests again. Thanks Gage! [18:06:52] uh-oh [18:06:57] ottomata, I just got a really weird hive error. [18:07:04] USE wmf; [18:07:04] FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient [18:07:04] FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask [18:07:09] ha, yeah uhh, cluster is down! [18:07:20] * Ironholds winces [18:07:21] see email? :) [18:07:38] not yet? Did I just miss it earlier? [18:08:37] sent it thurdsay...uhh [18:08:39] double checking [18:09:08] ha, subject: 'Hadoop cluster reinstall' [18:09:14] ahhh [18:09:34] * Ironholds is really glad he pulled out his sampled dataset for the PVs already ;p [18:09:57] :) [18:10:34] but I'm sad I can't write up "When Botnets Attack" [18:13:23] ottomata, wait,d oes this mean we're losing all the historic data? [18:13:29] yes [18:13:31] Crap. There's a really interesting case study in botnet behaviour there :( [18:13:35] but that was discussed and agreed, right? [18:13:38] ah well, swnigs, roundabouts [18:13:42] yeah, I think I just forgot the eta ;p [18:13:49] it looks like this botnet will keep going anyway. [18:14:02] ha ok :) [18:26:20] Ironholds: sorry man. My banner screenshot bot is using the default UA, which is ".*PhantomJS/.*" [18:26:37] Ironholds: there is a second banner screenshot thing running, lemme scrounge details for that... [19:09:39] awight, that's okay! [19:15:58] awight, so those are just the RecordImpression scripts? [19:16:14] oh, wait, no, I see what you mean. Curses! Could you change that? ;p [19:16:34] as long as it's identifiable in the future it's not a problem [20:02:37] milimetric, thank you!!! graphs look great. I will play with them today. Hope it won't be too hard to reconfigure things to add extra pages, change settings, add title, etc. [20:06:11] yurikR: you can't really configure much [20:06:18] name of datafile -> title of graph [20:06:23] column heading -> label in legend [20:06:32] dashboard has a title, subtitle, and tab names [20:06:42] so you can add / remove tabs with one or more graphs on each [20:07:00] and it's all json so it should be fairly self-explanatory [20:07:34] if you want to configure more, you'd have to generate limn metadata which can get ... annoying [20:07:43] gotcha [20:08:03] if it's absolutely necessary, take a look at limnpy which tries to automate some of that for you [20:08:06] where is the actual list of file mappings? how does limn find out which files are available [20:08:31] limn globs everything in the dashboards directory on the server and finds the id you're looking for [20:08:45] and it serves the files out of the datafile directory like static files [20:08:52] *datafiles directory [20:09:34] so, basically, /dashboards/blah maps to that path I gave you /dashboards/blah.json [20:09:52] aww, limn :) [20:09:59] heh [20:10:09] ok. I will play with it. So if I need an extra tab, just place a data file in that dir? [20:10:15] and /data/datafiles/gp_zero_local/blah.tsv maps to /datafiles/blah.tsv [20:10:20] no [20:10:26] tabs are defined in the dashboard [20:10:39] let me walk you through an example of that [20:10:56] so you're starting with this yurikR: [20:10:56] { [20:10:56] "id": "bb93b487e87eeb8e84aadb4411f893973eb38961a34ebe4120278bc8", [20:10:56] "headline": "Dashboard", [20:10:57] "subhead": "subtitle", [20:10:59] "tabs": [ [20:11:01] { [20:11:03] "name": "Graphs", [20:11:05] "graph_ids": [ [20:11:08] "http://gp.wmflabs.org/data/datafiles/gp_zero_local/bb93b487e87eeb8e84aadb4411f893973eb38961a34ebe4120278bc8/states-count-per-day.tsv" [20:11:10] ] [20:11:12] } [20:11:14] ] [20:11:16] } [20:11:44] to add a tab, do this: [20:11:45] { [20:11:45] "id": "bb93b487e87eeb8e84aadb4411f893973eb38961a34ebe4120278bc8", [20:11:45] "headline": "Dashboard", [20:11:45] "subhead": "subtitle", [20:11:47] "tabs": [ [20:11:49] { [20:11:52] "name": "Graphs", [20:11:54] "graph_ids": [ [20:11:56] "http://gp.wmflabs.org/data/datafiles/gp_zero_local/bb93b487e87eeb8e84aadb4411f893973eb38961a34ebe4120278bc8/states-count-per-day.tsv" [20:11:58] ] [20:12:00] },{ [20:12:03] "name": "Second Tab", [20:12:05] "graph_ids": [ [20:12:07] "some-other-datafile-url" [20:12:09] yep, figured [20:12:09] ] [20:12:11] }, [20:12:14] ] [20:12:16] } [20:12:25] sounds very straightforward [20:12:28] and in this case, the datafile-url should probably be prefixed with http://gp.wmflabs.org/data/datafiles/gp_zero_local/bb93b487e87eeb8e84aadb4411f893973eb38961a34ebe4120278bc8/ [20:12:46] ok, cool [20:13:13] and there should be no way I know of to force limn to tell you of the other dashboards / datafiles that exist on the server [20:16:08] sounds perfect. I think all this should be enough to set up a presentation tomorrow [20:16:15] now i need to get some data issues sorted out :( [20:16:25] good luck yurikR, and remember: sleep good [20:16:36] i hear you [20:19:00] !paste [20:22:19] awight, so: can you throw a quick patch in to make those phantomJS projects register properly? [20:22:23] If so, there's no issue. [20:25:02] Ironholds: I have made a card for it... should be easy [20:25:06] milimetric, what steps do i need to do to add an extra site? just add another dir and another dashboard file? [20:25:20] Ironholds: the only issue is *finding* the stupid other projects. [20:25:27] awight, cool! Just lemme know what you pick [20:25:31] Suggest just "WMF Fundraising". [20:25:35] It doesn't need to be impressive. [20:27:19] wickedslickbot@me.org [20:30:43] awight, depends how annoyed you want me to be at you ;p [20:37:03] awight: Make sure there's a newline [20:37:29] And ideally two different kinds of quote marks [20:37:43] marktraceur, I will end you. [20:38:05] (also you forgot to recommend a tab and if possible some really weird special characters. For added lulz make it something python will choke on if it's reading in JSON in strict mode) [20:38:09] <\n3 [20:40:28] "MSIE6.0\t;\nFR battlebot" [20:40:48] Lol battlebot yes [20:46:02] Analytics / General/Unknown: Oozie setup does not reliably set up Oozie's sharelib's directory - https://bugzilla.wikimedia.org/67567#c1 (christian) NEW>RESO/INV It seems the issue only appears when adding the role::analytics::oozie::server role, before the worker had deployed their role::ana... [20:47:14] awight, you forgot a random '