[14:24:03] Ops-Access-Requests, Analytics-Cluster: Access to Hadoop Cluster for Ananth Ramakrishnan (new contractor) - https://phabricator.wikimedia.org/T85229#947068 (Ottomata) Thanks Ananth. According to https://wikitech.wikimedia.org/wiki/Server_access_responsibilities#SSH_key_access, We will need you to use a diffe... [15:14:29] Ops-Access-Requests, Analytics-Cluster: Access to Hadoop Cluster for Ananth Ramakrishnan (new contractor) - https://phabricator.wikimedia.org/T85229#947132 (ananthrk) Thanks Andrew. Please find a different public key attached. {F25577} Send me a link to the Hangout session and I will jump right in [15:21:08] Analytics-Wikimetrics: Wikimetrics backup has no monitoring - https://phabricator.wikimedia.org/T71397#947133 (Nuria) p:Unbreak!>Normal [15:21:33] Analytics-Wikimetrics: Epic: AnalyticsEng has robust code to backfill metrics data - https://phabricator.wikimedia.org/T71252#947135 (Nuria) p:Unbreak!>High [15:21:57] Analytics-Wikimetrics: Story: AnalyticsEng uses TimeSeries support to backfill data - https://phabricator.wikimedia.org/T71253#947138 (Nuria) p:Unbreak!>High [15:32:36] hi nuria. [15:32:40] holaaaa [15:32:48] holaaa leila [15:33:00] how did you end up resolving the counting of uniques nuria? [15:33:08] for apps? [15:33:20] I'm writing the page for hyperloglog and other methods and I wonder what you ended up doing. [15:33:32] I guess your question was about apps at the time, nuria. [15:34:33] i tried sampling but due to how we do not "pre-bucket" records, time consumed in sampling or not -for this query in particular- is the same. It will "save" time if there is a lot of post-processing of results, but hive -in our case- goes through all records regardless of whether the query is bucketed [15:35:39] so no sampling but queries, instead of being "monthly" will be daily and only on "good" data which reduces execution times due to the volume being just smaller [15:36:20] I see. [15:36:29] the hourly query on oozie runs in 5/10 minutes in the partitions we want so a daily query will be acceptable. [15:36:55] yeah, makes sense, nuria. [15:37:05] leila: now, in the case of pageviews -for example- where -by looking at the UDF- there seems to be a lot of processing of records [15:37:13] the sampling might be a good strategy [15:38:33] I see. makes sense. [15:40:12] leila: I read the hyperlog paper, and looked at the hyperlog udf (which seemed a little overkill) and I think we will benefit from using that strategy in the future. [15:40:46] yes, I agree with you, nuria. I'll document it so we can go back to it. will share the link. [15:41:11] (PS21) Ottomata: Add UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 [15:41:15] leila: awesome, cause this IS going to come up again and again. [15:41:17] (CR) Ottomata: Add UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (3 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata) [15:42:07] leila: I also realized that the good paper to read (from the 3 the guy has) is the 1st one cause is the one where he truly explains his method, the rest are improvements on that one [15:42:14] morning [15:42:30] yeah. [15:42:34] morning Ironholds. [15:42:41] how goes? [15:42:47] goes well, how are you? [15:44:56] oh, you know [15:44:59] same as I always am. [15:45:09] I decided to write a wikitext island parser [15:45:28] so we'll be able to turn wikitext tables into R dfs. [15:46:44] or at least we will if stringstream stops complaining [16:04:11] operations, Analytics: Fix Varnishkafka delivery error icinga warning - https://phabricator.wikimedia.org/T76342#947173 (Ottomata) I've abandoned the previous change in favor of this one: https://gerrit.wikimedia.org/r/#/c/182072/ [17:03:18] (PS2) Rtnpro: Integrated logging [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/180828 [17:46:32] ottomata: i'm a little stuck [17:46:47] this sandbox appears cool but seems misconfigured [17:46:59] oh ja? [17:47:01] howso? [17:47:11] specifically, it sets the metadata broker to localhost:9092 [17:47:17] for kafka [17:47:39] and tries to use that from a producer to emit events [17:48:30] which just fails over and over saying it can't connect to localhost:9092 [17:49:26] zookeeper and kafka seem to both be running and configured to know about each other [17:49:49] I'm not sure how to go about debugging... [17:50:05] is kafka listening on port 9092? [17:50:10] check server.properties [17:50:16] wherever that might be on your vm [17:50:20] it is in /etc/kafka in ours [17:50:25] ok, looking [17:51:32] it has port=6667 [17:51:55] but no mention of 9092, is "port" what it listens to? [17:52:09] yay, people! [17:52:11] ja should be [17:52:12] try that [17:52:35] yup we have [17:52:36] # The port the socket server listens on [17:52:36] port=9092 [17:52:39] in ours milimetric [17:52:58] hmm godog, txstatsd is not happy on some of these hosts [17:53:02] what's the difference between "socket server" and "metadata broker" [17:53:22] ooopss, wrong chat [17:53:43] ERROR producer.SyncProducer: Producer connection to localhost:6667 unsuccessful [17:54:12] do you have any way to make sure kafka's up besides making a producer? [17:54:26] oh, your producer is trying to connect to 6667? [17:54:41] no, I just changed that since port was set to 6667 [17:54:44] ah ok [17:55:05] ok, so kafka is listening on 6667, and you are producing using 6667 [17:55:25] i mean, is the kafka process running? [17:55:40] what kind of logs is it outputting? (not sure where those wouldb ein your vm, in ours it is /var/log/kafka [17:55:41] ) [17:55:46] check to see if that port is actually open [17:55:57] sudo netstat -lnp | grep 6667 [17:57:56] kafka was running - it stopped I guess, I'm bouncing that now and checking the port [17:58:17] kafka's putting out logs in /var/log/kafka/* so that seems good [17:58:40] it says something's listening on 6667 too [17:58:48] 14788/java [17:59:36] same "java.net.ConnectException: Connection refused" when I try to run the producer though [17:59:36] cool, should be kafka then [17:59:55] kafka's not configured right or something? [18:00:06] dunno, um, your install is probably very diffferent than ours [18:00:15] do you have some shell scripts somewhere to do stuff with kafka? [18:00:28] how are you stopping/starting kafka? [18:00:47] i'm using ambari [18:00:58] but interestingly - kafka has this in the logs: [18:01:00] TRACE Broker 0 cached leader info (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:3,ControllerEpoch:5),ReplicationFactor:1),AllReplicas:0) for partition [truckevent,0] in response to UpdateMetadata request sent by controller 0 epoch 5 with correlation id 5 (state.change.logger) [18:01:09] which looks like the event I'm trying to produce (truckevent) [18:01:22] and that's in state-change.log [18:01:56] I know you're doing a half day so feel free to ignore this :) [18:01:56] ok cool, that looks fine [18:02:17] shell scripts somewhere? [18:02:18] things like this? [18:02:39] https://github.com/apache/kafka/tree/0.8.1/bin [18:02:42] yes, there are some utilities to check what topics there are and stuff like that [18:02:52] do they work? [18:02:54] e.g. [18:03:05] yes, they say the truckevents topic exists [18:03:11] kafka-topics.sh --describe [18:03:14] ok cool, kafka sounds good then [18:03:17] can you start a consumer? [18:03:28] kafka-console-consumer.sh --topic truckevent [18:04:39] yes, both of those work [18:04:45] --describe shows me the truckevents topic [18:04:52] ok, so keep the consumer running [18:04:55] the consumer just sits there [18:05:02] you'll finally know the whole thing is working when you see output from your producer there [18:05:04] both, btw, require me to pass --zookeeper [18:05:14] and I passed it localhost:2181 for that [18:05:17] which seems to work [18:05:35] cool, ja, consumer yes, producer shouldn't require that, right? [18:05:48] oh, maybe it does. [18:05:48] hm [18:05:49] looked to me like producer required it too [18:06:05] let me try to find the code for this online, then you can see how they implement the producer [18:06:06] hmm actually [18:06:08] it shouldn't [18:06:15] wait, what producer? [18:06:18] what version of kafka is this? [18:09:05] looking [18:11:39] don't see where the version info is [18:12:01] there's a java producer that I'm trying to find the source code for, but it's just in a zip file... gr [18:12:13] what producer? comes with hortonworks? [18:12:16] not kafka console-producer [18:12:18] let's try that first [18:12:19] see if that works [18:12:26] kafka-console-producer.sh [18:12:32] try to produce to your truckevent topic with t hat [18:13:38] ottomata, toby is asking me about the PV UDFs. Would you have any time to CR the legacy UDF today? [18:14:18] ottomata, do you know how many logs we register every month? (roughly) [18:14:21] ^ sounds more important than my muddling [18:15:09] legacy udf? [18:15:38] leila, you are asking how many webrequest log records there are per month? [18:15:47] yes, ottomata [18:15:50] nuria asked me that too, my back of the napkin is: [18:15:59] ottomata, https://gerrit.wikimedia.org/r/#/c/181049/ [18:16:17] 125k*60*60*24*31? :p [18:16:19] (I like how nuria and I are trying to avoid that count(*) ;-) ) [18:16:34] ~120K / second on average [18:16:36] yeah [18:16:37] something like that [18:16:43] makes sense. thanks ottomata [18:17:22] Ironholds, pageview is a subset of it. ;-) [18:17:37] sure, but webrequests is not [18:17:39] Ironholds: I dont't hink I'll ahve time to get to that today [18:17:42] ottomata, okie :) [18:17:43] but it seems your number is very close to ottomata's [18:17:53] yes, ours are both based on the same thing [18:17:58] his count, and my misremembering of his count ;p [18:18:01] so, question [18:18:06] but, in advance, maybe you can take qchris comments abou thtings on the Pageview change and apply them here [18:18:08] thing slike, more tests [18:18:09] if we want to work out how many records there are, why don't we just... [18:18:16] spaces instead of tabs [18:18:37] SELECT SUM(count_actual) FROM webrequest_sequence_stats [18:18:37] alphabetizing regexes [18:18:42] kk [18:19:08] it looks like the kafka-console-producer failed too, same "connection refused" deal with either 6667 or 9092 [18:19:49] hm [18:19:58] milimetric: screen share? [18:20:05] sure, i'll jump in hangout [18:26:53] hey milimetric, how often are mobile-limn-data graphs generated? [18:27:51] bmansurov: should be every hour [18:28:10] milimetric: thanks [18:37:29] (PS3) Mforns: Add Annotations API [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 [18:45:16] (CR) Mforns: Add Annotations API (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [18:45:24] (CR) Mforns: Add Annotations API (8 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [19:33:17] milimetric, mforns : back online [19:33:30] hi nuria [19:33:57] when you have 5 mins, I'd like to talk about your comments [19:34:42] nuria, ^ [19:35:02] mforns: sure, now? [19:35:18] ok [19:35:33] batcave? [19:35:49] k [19:51:58] milimetric: thought you might appreciate: https://tools.wmflabs.org/rfa-tool/, first C# web tool on ToolLabs :) [19:55:25] Server Error in '/rfa-tool' Application [19:55:30] nicely done, YuviPanda ;) [19:55:37] indeed :) [19:58:29] heh [19:58:42] it’s a very classical ASP.NET error :D [20:02:57] (CR) Nuria: Add Annotations API (2 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [20:10:57] (PS4) Mforns: Add Annotations API [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 [20:15:21] (CR) Mforns: Add Annotations API (2 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [20:16:29] (PS4) Fhocutt: Add user names to json report, corresponding test [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/181770 [20:16:34] (CR) Mforns: [C: -1] "Still have to remove the promise cache." [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [20:17:15] milimetric: I know there's a lot to improve (hard-coded strings, for instance), but I'm mostly looking for help chasing down that weird bug [20:17:31] fhocutt: with the test right? [20:17:37] yeah [20:18:00] yep, I am juggling a few things but I had your patch checked out, ran the tests and got the same error, and was about to check it out [20:18:08] i'll do it in the next hour for sure [20:19:50] no worries if not immediately [20:20:07] I'm working on a handful of things too [20:38:21] (PS5) Mforns: Add Annotations API [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 [20:40:48] (CR) Mforns: "Removed the cache, now it should be OK." [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [20:51:49] (CR) Nuria: "I think this is ready to merge once we merge this one: https://gerrit.wikimedia.org/r/#/c/181599/" [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [20:59:22] fhocutt: so there seem to be two problems that you're inheriting here from previous changes [20:59:43] the test that's failing should test if you run it in the commit before yours [21:00:15] I tried to verify that it would, but I ran into the second problem: parse_requirements apparently changed its required signature and now needs a "session" [21:00:52] I can now confidently say that while pip is not as bad as gem, it's not too far off... [21:01:03] oh dear. [21:03:26] milimetric: clearly everything must be debian packages. [21:03:31] * YuviPanda goes back under his bridge [21:03:55] any idea how to work with the new parse_requirements that needs session? [21:07:45] I'm not familiar with this at all [21:13:35] what does parse_requirements belong to? [21:20:07] milimetric: huh. I brought up vagrant and the tests run fine on the branch I wasn't making changes on. [21:24:40] (CR) Mforns: [C: 2] Fix reference to responsive.css file in index.html [analytics/dashiki] - https://gerrit.wikimedia.org/r/181599 (owner: Nuria) [21:25:14] (CR) Mforns: [V: 2] Fix reference to responsive.css file in index.html [analytics/dashiki] - https://gerrit.wikimedia.org/r/181599 (owner: Nuria) [21:25:43] (PS6) Mforns: Add Annotations API [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 [21:27:58] sorry fhocutt - in meeting for 30 more min. [21:32:54] (CR) Nuria: [C: 2 V: 2] Add Annotations API [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns) [23:01:12] fhocutt: very bizarre, I have to look at it more tomorrow [23:01:29] parse_requirements is a function called by the setup system [23:01:51] and apparently some software upgrade changed my pip to where it makes parse_requirements demand a "setup" parameter [23:01:59] so now my wikimetrics install is busted :( [23:02:11] milimetric: :( [23:02:26] like... sigh :) [23:02:45] I don't get why people do make dramatic changes like that in critical pieces of software [23:03:03] senior dev tip #1: don't rebuild a highway people are driving on [23:03:56] yeah. [23:04:59] fhocutt: what do you get from "pip --version"? [23:05:48] pip 1.5.6 from /usr/local/lib/python2.7/dist-packages (python 2.7) [23:06:21] wow, that's different: [23:06:21] pip 6.0.3 from /usr/local/lib/python2.7/dist-packages (python 2.7) [23:06:39] ...wow, yes. [23:06:46] that's on my vagrant