[05:51:14] New patchset: Erosen; "makes NamespaceEdits namespace parameter a custom wtf Field subclass, and adds testing code to deal with instantiating wtforms outside of the request context. current commit still fails tests which use a custom field type with celery, as app context is n" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71579 [05:58:59] Change merged: Erosen; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71579 [06:31:04] New patchset: Rfaulk; "add. proportion aggregator (change existing to rate)." [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/71582 [06:32:35] Change merged: Rfaulk; [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/71582 [10:53:03] drdee: hmm, but from talking to milimetric it looks like its purpose is a lot bigger (tell a story, interactiveness, let people customize without writing code, etc). Fairly heavy, so to 'just get a chart' it takes a long time of manually hand-editing JSONs. I know I could use limnpy, but still... too many moving pieces for 'have csv want chart' [13:29:24] mooorning average [13:29:30] morrning qchris [13:29:33] mooorning ottomata!! [13:29:50] morning milimetric [13:30:06] Morning drdee :-) [13:30:25] YuviPanda we will fix the manual editing of JSON's :) [13:30:37] qchris wanna do a hangout in 20 min? i need my coffee fix [13:31:01] Can we postpone that a bit [13:31:06] drdee: :) Is someone working on it? I see very little activity over the last two months on the repo [13:31:16] ? I can smell a meal getting ready :-) [13:31:37] YuviPanda: patience is a virtue [13:31:54] but the way it works, we need real customers who express real needs [13:32:04] that makes it much easier for us to schedule [13:32:16] I understand. [13:32:34] I was away for exams for two months, so I guess things changed in the meantime. [13:43:02] drdee: hey :-) I was wondering if I could use analytics services to build a dashboard. My idea would be to provide you two lists, have them compared and then in the end have a nice dashboard showing the progress on the # of diffs and possibly the differences [13:43:40] hashar: what lists? [13:43:49] context is continuous integration. I would like to track extensions missing jobs / unit tests or jobs not being triggered by Zuul [13:43:55] (I can help take this up if analytics doesn't have the bandwidth :D) [13:44:16] what would the dashboard show? [13:44:33] a line graph of '% of extensions without tests / jobs over time'? [13:45:25] a line showing the delta between extensions being in jenkins and the one being in gerrit [13:45:43] I am not sure whether the dashboard could be used to provided a table of faulty things that needs to be fixed [13:45:54] maybe I am completely crazy [14:12:57] average are you around? [14:13:32] qchris: https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [14:15:09] ottomata did you also clean up the sampled datasets? [14:15:20] hashar: would such a list be generated by reading the zuul config and the gerrit config? [14:15:46] no [14:27:26] YuviPanda: yeah that is the idea, listing all extensions in Gerrit ( i.e. https://gerrit.wikimedia.org/mediawiki-extensions.txt ) [14:27:50] so zuul config - extensions.txt, over time. [14:27:56] YuviPanda: and then list all jobs generated by Jenkins Job builder + all triggered jobs in Zuul. Then inject all of that somewhere and have nice metrics generated :-] [14:27:57] err [14:27:59] other way around [14:28:10] I should probably write down a spec [14:29:02] hashar: that would be nice :) [14:29:23] hashar: also having this be able to generate data for any date in the past would be nice. [14:29:44] hashar: for example, 'checkout latest commit as of date for both these repos, and then do the diffing' [14:29:50] should enable us to have over-time metrics easily [14:30:16] then I am wondering if that can be done in analytics infrastructure [14:30:59] or whether I should write a python script to generate some basic files / metrics and have them graphed graphite/ganglia :D [14:31:23] hmm, drdee could answer that better, I guess. [14:31:42] though from what we did for EventLogging + the Mobile team, you need to write the python scripts anyway! [14:33:35] I think both ways are valid hashar, just matters how you want it to look [14:33:50] I am happy to set up ci-reportcard.wmflabs.org or something like that [14:33:56] then it'll look like all the other reportcards [14:34:11] milimetric: it would still need the python scripts to generate the data, no? [14:34:15] yep [14:34:30] but it sounds pretty custom, you probably have to write a bit of code no matter what [14:36:16] yeah I am fine writing the glue :-] [14:36:42] do you have any document explaining the input format ? [14:46:46] the input format to Limn can be whatever you're comfortable with [14:46:48] CSV/TSV [14:46:52] (plain values, no quotes) [14:46:54] or JSON [14:46:58] hashar^ [14:51:22] * hashar googles for CSV and JSON [14:51:30] can't I give you a .xls ? [14:52:24] milimetric: will look at writing a spec, then have a jenkins job to generate the data I need daily [14:52:35] milimetric: then one day uI will ping you back to find out what to do with them hehe [14:52:37] thanks! [14:52:49] changing locations, back in a bit [14:52:51] cool, look forward to that day :) [15:47:33] hey milimetric [15:48:50] I just wanted to give you the heads up that I've been doing manual deployments on ee-dashboard (pulling updates from the git repo) as limn-deploy still has dependency problems [15:49:19] it should be nbd as these are all small changes (new data sources or graphs, changes to the dashboard layout) [15:54:44] ok DarTar [15:55:13] so there's nothing that limn-deploy can do, it's just that paramiko is messed up [15:55:26] so the way I got around it was to sudo pip install -U paramiko [15:55:35] give that a try and then try to deploy [16:10:49] milimetric: I think I tried that (following your instructions) but it didn't work [16:11:03] quick hangout to figure it out DarTar? [16:11:11] sure [16:11:39] i'm making one [16:13:44] hey DarTar! is it la bella vita in Italy? [16:14:40] regarding your mail, yes we should have such a feature……….given the july 24th deadline for umapi that's gonna be a bit tricky, milimetric what do you think? [16:15:09] no new limn features until we ship wikimetrics [16:15:22] drdee: yeah, life's pretty good :) [16:15:47] let's focus on nailing that deliverable [16:15:52] yep [16:15:59] tnegrin: that's what I feared, we can wait [16:16:16] D - can you create a card and spec it out with Dario please? [16:16:54] We should start rolling up Limn requests for the next release. [16:30:10] haha, check out this poorly sloped kafka graph! [16:30:11] http://ganglia.wikimedia.org/latest/graph_all_periods.php?hreg[]=analytics100%5B689%5D.eqiad.wmnet&mreg[]=kafka_producer_AsyncProducerStats-.%2B.AsyncProducerEvents&z=large>ype=stack&title=kafka_producer_AsyncProducerStats-.%2B.AsyncProducerEvents&aggregate=1&r=hour [16:30:17] we went over 4B total produce requests [16:30:46] which looks like it overflowed, wooo! [16:31:13] will fix that :) [17:00:26] ottomata, milimetric: scrum [17:00:51] bwerp [17:00:53] ottomata: hi [17:01:04] ottomata: can you help me please to get access to stat1001 and stat1002 ? [17:01:12] ottomata: it worked last time but it seems there are still problems now [17:01:52] ja k, after standup can help [17:08:39] ottomata: after standup, please have a look at https://gist.github.com/wsdookadr/fe39f10953491c7ef416 [17:09:32] ottomata: for after standup, this is basically my configuration and verbose output of connecting to stat1001. [17:10:04] (maybe we can do the same thing as last time, like diagnosing the problem by looking at the logs of sshd) [17:12:47] oh average [17:12:58] i don't thikn you can use analytics1001 as a bastion for the stat hosts [17:13:03] debug1: Executing proxy command: exec ssh -e none spetrea@analytics1001.wikimedia.org exec nc -w 3600 stat1001.wikimedia.org 22 [17:13:16] oh ok [17:13:23] should I use a different machine as proxy ? [17:13:30] that's probably left over from our attempt to get you access to analytics nodes [17:13:30] yes [17:13:31] OHHH [17:13:36] which one please ? [17:13:36] oh hm [17:13:36] wel [17:13:40] for stat1001, you don't need a bastion [17:13:43] it has a public IP [17:13:55] but, for stat1002 you do, you should use bast1001, but this is the issue where you don't have an account on the public bastions yet [17:13:56] sigh [17:13:56] ok [17:13:58] lemme see [17:14:19] ok [17:14:50] sigh, i have to create an RT [17:17:35] ok [17:18:34] oh average, now i'm confused [17:18:38] we used analytics1001 before for this, eh? [17:18:41] it isn't hte proper thing to use [17:18:42] but hm [17:19:47] ottomata: yes [17:20:19] hm [17:21:23] OH [17:21:32] hmm [17:21:40] hmm, no oh [17:21:58] average, what about [17:22:05] ssh -v -A stat1002.eqiad.wmnet [17:22:08] what's that do? [17:22:11] trying [17:22:23] works [17:22:27] this is weird [17:22:34] is there like a map somewhere of all these machines [17:22:41] ottomata: thanks [17:22:46] ottomata: can I do that for stat1001 also ? [17:22:49] try [17:22:51] stat1001.wikimedia.or [17:22:52] g [17:22:53] yeah [17:23:15] oh, average [17:23:19] you don't have an account on stat1001 [17:23:23] do you need one? [17:25:12] so, ssh -v -A spetrea@stat1001.wikimedia.org doesn't work [17:25:15] says permission denied [17:25:21] but stat1002.eqiad.wmnet works [17:31:08] right [17:31:12] you don't have an account on stat1001 [17:31:13] average [17:31:15] do you need one? [17:32:30] yes please [17:33:00] why? [17:33:27] I need access on stat1(yes) ; stat1001(no) ; stat1002(yes) to debug and make sure reports are generated and placed in the right place [17:34:13] this is directly related to card #738 and the migration (of wikistats) which has happened recently [17:34:34] ok [17:34:35] great [17:34:42] then you have access to stat1 and stat1002, right? [17:34:46] yes [17:34:56] ok, so all is good? [17:35:54] I would like stat1001 also because it's part of the machines that are related to wikistats [17:36:22] yeah, but we are limiting as much access to that one as possible [17:36:27] we only grant that if you really need it [17:36:36] all stat1001 does for wikistats is host the rsynced data from stat1002 [17:36:52] you should be able to see that data at stats.wikimedia.or [17:36:52] g [17:39:37] My main argument for this is. I should have access to the same machines Erik has access to because Erik and me are making reports using the same data and we are publishing them on the same machines. [17:41:40] drdee ^? [17:42:00] agree, average should have access to same machines as ez [17:42:06] including stat1001 the web host? [17:42:16] if that's part of the wikistats setup then yets [17:42:26] average, i'm still not sure why you need that though? there's just an rsync cron job [17:42:30] it syncs everything from stat1002 to stat1001 [17:42:36] should be the same on stat1001 as on stat1002 [17:44:25] qchris: a good start for today would be to read ComparePageviewDefinitionsTest.class in kraken-pig and to look at the pig scriptsj [17:44:46] see also http://pig.apache.org/docs/r0.10.0/index.html [17:44:48] Ok. Great. [17:45:04] * qchris starts diving into pig. [17:50:53] drdee, average, I'm still reluctant to give stat1001 access, we're trying to restrict that as much as possible [17:51:00] mmmmmm [17:51:06] i don't think average really needs it to figure things out, everything he needs shoudl be on stat1002 [17:51:19] average: can you state your case :) [17:54:59] will rest my case for the moment. I believe ottomata's reasoning above is correct. [17:55:19] great [17:56:21] average, i submitted an RT to get you access to bastion hosts [17:56:30] we need toby's approval and 3 days of wait tiem [17:56:34] analytics1001 is fine though [17:57:33] ottomata: cool, thank you :) [17:58:17] yup! [17:58:24] i'm available to help with dclass today too [17:58:29] do we have time to work on that? [17:58:31] average? ^ [17:58:50] tnegrin: do you have an RT account? [18:01:10] ottomata: I've to checkout a previous patchset because I made some changes and now I'm getting lots of errors [18:01:31] hm, ok [18:12:27] I think I got disconnected from IRC [18:12:42] is there any place other than http://stats.wikimedia.org/wikimedia/squids/SquidReportClients.htm where I can see browser stats (for mobile)? those squid reports are pretty useless... even 10 most popular browsers with browser name/version would be awesome... [18:12:46] drdee, ^ [18:14:34] jgonera: http://stats.wikimedia.org/kraken-public/webrequest/mobile/ ; one job is broken and we are working on fixing that [18:15:50] drdee, where should I look exactly? which file? [18:16:53] depends on what you need [18:17:23] but probably http://stats.wikimedia.org/kraken-public/webrequest/mobile/device/props/2013/05/ looking at your initial queston [18:18:23] that doesn't seem to contain any versions, there's a huge difference between standards support between Android 4.x and Android 2.x [18:19:26] so is your question: what are the top x most popular user agent strings? [18:19:53] (excluding crawlers and desktop) ? [18:21:12] jgonera: ^^ [18:22:05] not exactly, I for example don't care if an Android 4.2 is a regular one or a modded one (differen user agent string), I just care that it's an Android 4.2 [18:24:33] how prevalent are modded user agent strings? [19:38:45] erosen, milimetric: one of you should fill out https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request to apply for db access for wikimetrics [19:58:29] average, how goes dclass? [20:08:21] New patchset: Milimetric; "celery task running on flask" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71712 [20:09:13] Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71712 [20:09:30] hey erosen, how goes [20:09:40] i just got that dang celery task running through flask [20:12:48] niiice [20:12:54] milimetric: I was still fighting with that [20:13:06] milimetric: want to hang out? [20:13:11] in a bit [20:13:23] k, I'll look at the change set [20:14:33] k, in standup [20:17:47] ottomata: https://plus.google.com/hangouts/_/765c6f76a8dc1e5305bdf657c9e55ca4946bc2d7c1?hl=en [20:18:04] average, be there in 2 mins [20:18:10] ok [20:36:47] New patchset: Stefan.petrea; "Debianization to conform with Ops requirements" [analytics/dclass] (debian) - https://gerrit.wikimedia.org/r/68711 [20:46:50] New patchset: Stefan.petrea; "Debianization to conform with Ops requirements" [analytics/dclass] (debian) - https://gerrit.wikimedia.org/r/68711 [21:11:18] New patchset: Milimetric; "no longer using flask-wtf" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71721 [21:11:31] Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71721 [21:14:23] hey drdee [21:14:26] wanna talk db card? [21:14:37] aight [21:23:56] drdee, didn't notice your last notice, they're not very prevalent, they might change. basically, they need to be parsed and proper info extracted from them. I could do that myself if I had an easy way of obtaining, say, user agents from last month/week/day, possibly sampled. are those published somewhere? [22:01:47] New patchset: Stefan.petrea; "Debianization to conform with Ops requirements" [analytics/dclass] (debian) - https://gerrit.wikimedia.org/r/68711 [22:04:31] erosen, you broke the hangou! [22:04:36] whaaa? [22:04:40] yes you didd [22:04:45] don't deny it [22:04:56] or are you guys blocking me [22:05:01] fine [22:05:07] you've finally caught us [22:05:16] all these times that hangoust were "broken" [22:05:16] :D [22:05:31] we were just trying to chat without the BASSS [22:05:38] mmmmm [22:05:47] i really can't get in [22:05:53] tried both safari and chrome [22:06:00] i am in the hangout [22:06:06] i see both you and dan's icon [22:06:08] but nothing happens [22:06:11] weeeeeiirrrd [22:06:22] can you see me chat? [22:06:46] yeah [22:06:48] i can [22:06:52] and I can see your icon [22:06:55] but no sound or video [22:06:58] weeeeird [22:07:39] duuunooo [22:13:04] sudo chmod /dev/video0 [22:13:10] sudo chmod 777 /dev/snd/* [22:13:19] * average does that on a regular basis [22:46:49] drdee: looks like labsdb access happened [22:47:18] not sure [22:47:21] they just closed it [22:47:27] true [22:54:52] erosen: milimetric: i think i figured it out [22:54:57] the concept is called a service grup [22:54:57] ya? [22:55:00] yup [22:55:02] i created one [22:55:05] great [22:55:06] it's called wikimetrics [22:55:07] oh yea? [22:55:12] perfect [22:55:13] that's the 'account' [22:55:20] but the account can have members [22:55:26] right now myself and dan are member [22:55:34] erosen you will follow in a couple of minutes [22:55:36] aw [22:55:39] no don't add Evan [22:55:45] why not? [22:55:51] :) I'm just messing with him [22:55:57] hehe [22:55:59] jeez [22:56:04] i was worried there [22:56:09] let's make those last 3 weeks count [22:56:14] exactly [22:56:28] i even used his proper name [22:56:37] which in IRC I believe means you're getting reprimanded [22:56:47] Dan Florin Andreescu, you clean that room! [22:56:48] hehe [22:57:14] florin just to be the dutch word for our currency [22:57:20] just = used [22:57:28] yea [22:57:52] erosen, milimetric: check https://wikitech.wikimedia.org/wiki/Special:NovaProject [22:57:52] that's great though drdee [22:58:00] filter for 'tools' [22:58:06] and then scroll down all the way [22:58:31] you will see something likelocal-wikimetrics [22:58:32] Diederik [22:58:33] Milimetric [22:58:33] Erosen [22:58:46] cool, I see that [22:58:55] that's great [22:59:17] so does this mean that a labs instance set up by us will automatically have rights to connect? [22:59:22] is it ipfilters or something? [22:59:22] so i guess the workflow is as follows [22:59:30] connect to the tools instance as yourself [22:59:37] then sudo to wikimetrics user [22:59:41] then connect to mysql [22:59:45] which means......... [22:59:49] we still have a tunnel issue [22:59:53] hrm [23:00:09] i could be wrong though [23:00:18] yeah. it seems like we have some new options at least [23:12:54] okay so tunnel to tools-login and then enter 'become wikimetrics' [23:14:08] and then, for example, mysql --defaults-file=~/replica.my.cnf -h enwiki.labsdb [23:15:40] milimetric, erosen: ^^ i've also updated https://mingle.corp.wikimedia.org/projects/analytics/cards/753 [23:15:43] drdee: nice [23:15:45] that sounds plausible [23:16:20] New patchset: Milimetric; "posting desired responses to /jobs/create/" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71739 [23:16:27] cool [23:16:28] thanks! [23:16:28] not plausible; i tested it ;) [23:16:38] Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/71739 [23:16:42] fair ;) [23:16:51] i was just thinking about wikimetrics code [23:17:11] ok erosen, I got some basic rudimentary responses array to the controller [23:17:17] awesoe [23:17:19] awesome [23:17:21] tomorrow we should sync up and hook it into your stuff [23:17:23] I'm having trouble with celery again [23:17:25] yeah [23:17:26] cool [23:17:28] oh no [23:17:29] ! [23:17:34] i got the basic job working [23:17:38] but it won't run in celery [23:17:50] basic job? [23:17:55] meant to say JobResponse [23:18:00] ok [23:18:10] so JobResponse().run() works [23:18:15] sort of [23:18:29] but JobResponse().run.delay(pass the instance).get() doesn't [23:18:30] if you construct teh cohort like c = Cohort(), yes [23:18:34] alrigty guys, i am going to prepare a mighty fine meal [23:18:39] if you construct it like c = session.query()…. [23:18:41] bon apetit drdee [23:18:43] it doesn't work [23:18:46] oh! [23:18:46] later drdee [23:18:52] laterz [23:18:54] the cohort?! [23:18:58] yeah [23:19:00] super weird [23:19:01] oh ok [23:19:05] no it makes sense [23:19:18] and the weirdest part is that I can serialized the cohort in an interactive session with no problem [23:19:22] ooo, please expalin [23:19:39] maybe try expunging the cohort from the db session after pulling it out with Query [23:19:48] yeah, i was trying to find that code [23:19:54] what was the method to do that? [23:19:56] it's in User [23:20:09] line 44 [23:20:20] session.expunge(instance) [23:20:29] give that a shot [23:21:02] but basically it makes sense to me because SQLAlchemy must be leaving the properties in some funny state where they can't be accessed unless the owner session is still around [23:21:11] so when the Cohort goes to Celery [23:21:11] go tit [23:21:17] Celery's like WTF is this man?! [23:21:24] :) [23:21:27] k, i gotta run [23:21:28] but good luck [23:21:30] laters [23:21:30] thanks [23:21:32] hope I'm right [23:21:33] later [23:21:50] me too