[05:19:44] <grrrit-wm1>	 (CR) Yurik: [C: 2] Initial checkin - some data sanitizing code [analytics/zero-sms] - https://gerrit.wikimedia.org/r/144682 (owner: Yurik)
[07:31:45] <grrrit-wm1>	 (PS1) QChris: Add Python setup instructions to README.md [analytics/refinery] - https://gerrit.wikimedia.org/r/144905
[07:41:15] <grrrit-wm1>	 (CR) Yurik: [V: 2] Initial checkin - some data sanitizing code [analytics/zero-sms] - https://gerrit.wikimedia.org/r/144682 (owner: Yurik)
[07:54:18] <grrrit-wm1>	 (PS4) QChris: Coordinate computing sequence statistics through Oozie [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128) (owner: Milimetric)
[07:54:20] <grrrit-wm1>	 (PS1) QChris: Drop unneeded parts of oozie import [analytics/refinery] - https://gerrit.wikimedia.org/r/144909 (https://bugzilla.wikimedia.org/67128)
[07:54:23] <grrrit-wm1>	 (PS1) QChris: Drop unneeded auxpaths for hive snippets [analytics/refinery] - https://gerrit.wikimedia.org/r/144910
[07:54:25] <grrrit-wm1>	 (PS1) QChris: Switch Hive query variables to camelCase [analytics/refinery] - https://gerrit.wikimedia.org/r/144911
[07:58:14] <grrrit-wm1>	 (PS5) QChris: Coordinate computing sequence statistics through Oozie [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128) (owner: Milimetric)
[08:09:53] <grrrit-wm1>	 (CR) QChris: Coordinate computing sequence statistics through Oozie (5 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128) (owner: Milimetric)
[09:09:02] <wikibugs>	 Analytics / EventLogging: Code icon on schema pages has incorrect height - https://bugzilla.wikimedia.org/64251#c5 (Andre Klapper) PATC>RESO/FIX All patches merged - assuming this bug is FIXED. If that is not the case: Please reopen and elaborate what is left to do here to get this report fixed.
[10:26:01] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c6 (Dario Taraborelli) Other than looking at actual pages requested, I couldn't find any direct evidence from the daily PV stats that the RPS spike was caused by a surge of interest in th...
[13:30:50] <milimetric>	 DarTar: so are you saying that you think the RPS spike was or wasn't caused by the world cup?
[13:31:07] <DarTar>	 hey
[13:31:18] <milimetric>	 I'm still waiting for Andrew to come online to start digging into it more
[13:31:27] <milimetric>	 but until then, wild speculation :)
[13:31:29] <DarTar>	 I am saying that it’s very speculative
[13:31:33] <DarTar>	 yeah
[13:31:41] <milimetric>	 right, ok, agreed
[13:32:00] <DarTar>	 the amount of traffic was pretty high and sustained for the top articles on the world cup
[13:32:58] <DarTar>	 (which is of course totally compatible with a single, short term spike, but I would expected a substantial increase in overall traffic on that date, which was not the case)
[13:33:05] <milimetric>	 yeah, the only thing is, if this caused packets to drop, stats.grok.se wouldn't see the spike for those hours because webstatscollector would've never gotten the data
[13:33:14] <milimetric>	 I'd query hadoop but hive isn't working atm
[13:33:15] <DarTar>	 good point
[13:33:28] <milimetric>	 i think the data's still there though
[13:33:37] <DarTar>	 yes, that’s what Ironholds told me
[13:33:42] <DarTar>	 after checking with Andrew
[13:34:02] <milimetric>	 that game though!
[13:34:03] <milimetric>	 omg
[13:34:07] <DarTar>	 did you see the game or the highlights eventually?
[13:34:12] <DarTar>	 ha, so you did :)
[13:34:18] <milimetric>	 saw the whole thing, yea
[13:34:23] <DarTar>	 insanity
[13:34:34] <milimetric>	 and the funniest was when they scored the 5th goal, out of Nowhere, huge hurricane thunderstorm outside, downed some trees and everything
[13:34:37] <milimetric>	 I was like - woa
[13:34:54] * YuviPanda continues watching the cricket match that started today
[13:35:19] <DarTar>	 I am thinking whether I should really cheer fot the Dutch now just to send them through the German grinder in a few days
[13:35:54] <milimetric>	 oh man, poor Dutch / Argentina, they're gonna beat the crap out of each other and await the Terminators on the other end
[13:36:56] * DarTar nods and either way, whether it’s Argentina playing a final in Brazil or Germany vs Netherlands, it’s going to be like the end of the world
[14:20:50] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Adapt default auxpath for cluster setup [analytics/refinery] - https://gerrit.wikimedia.org/r/144842 (owner: QChris)
[14:23:06] <ottomata>	 qchris, we are still using oozie add partition stuff
[14:23:56] <ottomata>	 i'm ok with removing the drop stuff...but I'd kinda like to leave it in as a usage example
[14:23:58] <ottomata>	 it totally works
[14:24:42] <qchris>	 So the new cluster will use the add_partition stuff from oozie, and drop partitions through cron?
[14:24:48] <ottomata>	 yes
[14:24:54] <ottomata>	 old cluster was doing that too
[14:25:04] <qchris>	 Yes.
[14:25:11] <ottomata>	 https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/analytics/refinery.pp#L54
[14:25:12] <qchris>	 But that was from the precursor of refinery
[14:25:27] <ottomata>	 but add partitions was oozie
[14:25:45] <qchris>	 Yup. But still ... that was the precursor of refinery.
[14:25:47] <ottomata>	 add partitions was running from kraken
[14:25:54] <ottomata>	 in oozie
[14:26:00] <qchris>	 If we're gonna use that part again, let's keep it!
[14:26:05] <ottomata>	 yeah, def gonna use it
[14:26:21] <ottomata>	 that's why I told dan he should bring it over to refinery from kraken
[14:26:51] <qchris>	 But that part still relied on /wmf/krak...^W${precursor_of_refinery} durectories and stuff.
[14:27:05] <qchris>	 It would not work from the new cluster, as this directory
[14:27:08] <qchris>	 would not exist.
[14:27:35] <grrrit-wm1>	 (CR) Ottomata: "Not so sure about this. Just because this script is scheduled via Oozie, doesn't mean we should adopt Oozie conventions for it. I'd pref" [analytics/refinery] - https://gerrit.wikimedia.org/r/144911 (owner: QChris)
[14:27:48] <qchris>	 So can l keep the stuff and make it work for /wmf/refinery?
[14:27:55] <qchris>	 I mean ...
[14:28:00] <ottomata>	 totally
[14:28:01] <ottomata>	 ja
[14:28:03] <qchris>	 ok.
[14:28:11] <ottomata>	 and for wherever your hdfssyncdeploy script puts things
[14:28:15] <ottomata>	 paths need to be changed fo sho
[14:28:23] <qchris>	 :-D
[14:28:31] <qchris>	 Ja, sure.
[14:28:48] <qchris>	 But those path probably need another round of bikeshedding. ... I guess.
[14:28:48] <grrrit-wm1>	 (PS2) Ottomata: Add repository description [analytics/refinery] - https://gerrit.wikimedia.org/r/144676 (owner: QChris)
[14:28:55] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Add repository description [analytics/refinery] - https://gerrit.wikimedia.org/r/144676 (owner: QChris)
[14:29:03] <ottomata>	 yeah, will get to that :)
[14:29:08] <qchris>	 Hahaha. Cool.
[14:29:09] <ottomata>	 doing the easy reviews first :)
[14:29:23] <grrrit-wm1>	 (PS2) Ottomata: Add Python setup instructions to README.md [analytics/refinery] - https://gerrit.wikimedia.org/r/144905 (owner: QChris)
[14:29:44] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Add Python setup instructions to README.md [analytics/refinery] - https://gerrit.wikimedia.org/r/144905 (owner: QChris)
[14:31:11] <grrrit-wm1>	 (PS2) Ottomata: Add basic deployment script [analytics/refinery] - https://gerrit.wikimedia.org/r/144677 (https://bugzilla.wikimedia.org/67129) (owner: QChris)
[14:32:19] <grrrit-wm1>	 (CR) QChris: "I do not care much about whether we use camelCase or not." [analytics/refinery] - https://gerrit.wikimedia.org/r/144911 (owner: QChris)
[14:47:30] <grrrit-wm1>	 (CR) Ottomata: Add basic deployment script (7 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/144677 (https://bugzilla.wikimedia.org/67129) (owner: QChris)
[14:48:35] <ottomata>	 milimetric, qchris, why the rename of sequence_stats?
[14:48:56] <milimetric>	 i no do that?
[14:48:57] <qchris>	 Because scripts that /do/ something should start in a verb.
[14:49:03] <milimetric>	 ah
[14:49:18] <qchris>	 ottomata: that's on me ... milimetric did not know about it.
[14:49:37] <qchris>	 And the hql file is doing something, so it should say what it is doing.
[14:49:38] <milimetric>	 i agree with it, makes sense - verbs are cool
[14:52:15] <ottomata>	 i dunnnooooo, i think I agree with verbs, but more in the german format :p
[14:52:31] <yuvipanda>	 'goaal it?'
[14:52:33] <ottomata>	 having the verb first is often very inconsistent and a little irrelevant ssometimes
[14:52:39] <ottomata>	 is every script going to start with 'compute'?
[14:52:44] <ottomata>	 isn't everyhting 'computing' something?
[14:52:59] <qchris>	 Ok, what about "aggregate" instead?
[14:53:10] <ottomata>	 webrequest/sequence_stats_{generate,aggregate,compute}
[14:53:10] <qchris>	 or "extract"?
[14:53:18] <ottomata>	 _get
[14:53:27] <ottomata>	 _query
[14:53:28] <ottomata>	 ha
[14:53:38] <qchris>	 Any of generate,aggregate,compute is fine by me.
[14:53:49] <ottomata>	 you ok with verb at end...or at least subject at beginning?
[14:53:51] <qchris>	 I do not like get, or query too much.
[14:54:48] <qchris>	 It does not match what you're telling the the computer to do ... I like 'computer, aggregate squence stats' ... that's more natural language instead of
[14:55:02] <qchris>	 computer. sequence stats. aggregate. go. do it.
[14:55:12] <qchris>	 But!
[14:55:38] <qchris>	 We for the hql file, we could just call it "aggregate.hql", as we are already en the coordinator directory.
[14:55:57] <ottomata>	 maybe its my OO training coming through
[14:56:02] <ottomata>	 object.verb(*)
[14:56:04] <ottomata>	 object.verb()
[14:56:19] <qchris>	 Or stack languages :-D
[14:56:30] <ottomata>	 but, it makes organizing hierarchies better too
[14:56:41] <qchris>	 Yes. That argument is obviously true.
[14:57:00] <ottomata>	 what if we had some other grouping of queries?
[14:57:07] <ottomata>	 making something up
[14:57:10] <qchris>	 It seems you really care about object being first, so let's have it first.
[14:57:23] <qchris>	 Will update the changes.
[14:57:25] <ottomata>	 webrequest/refine/{geocode,sanitize,aggregate}.hql
[14:57:33] <ottomata>	 or
[14:57:41] <ottomata>	 webrequest/refine_geocode.hql
[14:57:42] <ottomata>	 etc.
[14:57:54] <ottomata>	 same names, just directories instead of file names
[14:57:58] <ottomata>	 kinda cleaner i think
[14:58:38] <ottomata>	 i think i liked generate more than the other options, qchris
[14:58:46] <ottomata>	 sequence_stats_generate.hql is fine with me
[14:59:06] <qchris>	 Ok. generate it is.
[14:59:19] <qchris>	 But about the directory part, I am not fully sure I understand.
[14:59:24] <qchris>	 "oozie/webrequest/sequence_stats/compute.hql" would become
[14:59:37] <qchris>	 Grr. Sorry.
[14:59:51] <qchris>	 "oozie/webrequest/compute_sequence_stats/compute_sequence_stats.hql" would become
[14:59:59] <qchris>	 "oozie/webrequest/sequence_stats/compute.hql"
[15:00:09] <qchris>	 or "oozie/webrequest/sequence_stats/compute/compute.hql" ?
[15:00:43] <qchris>	 s/compute/generate/g
[15:00:46] <ottomata>	 oozie/webrequest/sequence_stats/sequence_stats_generate.hql
[15:00:46] <ottomata>	 OR
[15:00:46] <ottomata>	 oozie/webrequest/sequence_stats/generate.hql
[15:01:17] <qchris>	 But that would mean we can only have /one/ coordinator for sequence_stats.
[15:01:41] <qchris>	 Maybe we would need another to clear sequence stats after some time?
[15:02:04] <qchris>	 Or one to slice by machines?
[15:02:59] <ottomata>	 why?
[15:03:15] <ottomata>	 why does it mean we can only have one coordinator?
[15:03:24] <ottomata>	 oh because sequence_stats/coordinator.xml
[15:03:25] <qchris>	 Because  "oozie/webrequest/sequence_stats/coordinator.xml"
[15:03:28] <qchris>	 Right.
[15:03:30] <ottomata>	 hmmm
[15:03:42] <qchris>	 So what about "oozie/webrequest/sequence_stats/generate/coordinator.xml"
[15:03:43] <ottomata>	 I'm ok with naming the .xml and property files
[15:03:48] <ottomata>	 i'm ok with that too
[15:04:02] <qchris>	 and  "oozie/webrequest/sequence_stats/generate/generate.hql"
[15:04:08] <qchris>	 (accordingly)
[15:04:12] <ottomata>	 haha
[15:04:14] <ottomata>	 not sure I'm for this
[15:04:20] <qchris>	 Hahahaha :-D
[15:04:23] <ottomata>	 but if you go that far, you could even abstract conventions for the query name
[15:04:51] <ottomata>	 a/b/c/{query.hql,coordinator.xml,workflow.xml,coordinator.properties}
[15:04:57] <qchris>	 But  "oozie/webrequest/sequence_stats/generate/coordinator.xml" implies that we need a name for the hql file in that directory, doesn't it?
[15:05:05] <ottomata>	 and just default in your workflow to look for query.hql
[15:05:11] <ottomata>	 in the cur direcotry
[15:05:17] <qchris>	 (I mean not strictly requiring, just ... it's the logical structure)
[15:05:38] <qchris>	 But queryies need not have only one hql file.
[15:06:25] <qchris>	 For example I wanted to make this coordinator's workflow do three actions: Compute, write to a file, alert.
[15:06:41] <qchris>	 Compute and "write to a file" would be two different hqls.
[15:09:47] <grrrit-wm1>	 (CR) Ottomata: Coordinate computing sequence statistics through Oozie (9 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128) (owner: Milimetric)
[15:13:19] <ottomata>	 yeah, das true qchris
[15:13:27] <ottomata>	 good point
[15:13:33] <qchris>	 http://etherpad.wikimedia.org/p/analytics-bikeshedding
[15:13:44] <qchris>	 ^ has all files in two variants
[15:14:00] <qchris>	 also .... IT'S A TRAP!
[15:15:37] <ottomata>	 hah
[15:16:30] <milimetric>	 qchris: mind pasting your local crontab for the check_slave_lag.sh script?
[15:17:09] <milimetric>	 i don't get how to execute
[15:17:47] <milimetric>	 ottomata: so I'd be interested in learning more about gadolinium, oxygen, and analytics1003, machines talking to those and machines reading from those
[15:17:55] <milimetric>	 how would I go about that in puppet / elsewhere?
[15:20:30] <milimetric>	 I mean I grepped around a little bit but I'm not sure if you have a process
[15:24:37] <ottomata>	 milimetric: in trap chat with qchris
[15:24:42] <milimetric>	 ah
[16:13:46] <ottomata>	 milimetric: I need to update this, but:
[16:13:47] <ottomata>	 https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Overview#UDP_Webrequest_Log_Stream
[16:14:00] <ottomata>	 actually: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Overview#Current_Architecture
[16:14:11] <ottomata>	 ack, hmm, that is hdfs related
[16:14:12] <ottomata>	 but the first part is kinda correct
[16:52:21] <ottomata>	 milimetric: not currently in any meetings
[16:52:25] <ottomata>	 for 40 mins....
[16:52:25] <ottomata>	 :)
[16:54:31] <qchris>	 milimetric: It's probably too late now, but you asked for the crontab entry for the check_slave_lag script. Here it is:
[16:54:32] <qchris>	 0 21 * * * /home/qchris/check_slave_lag/check_slave_lag.sh --quiet
[17:02:14] <Ironholds>	 milimetric, pingeth
[17:02:41] <Ironholds>	 Toby pointed me to you as someone with experience in handling complex workflows; got any good modelling software recommendations for Linux? I'm hoping to be able to diagram the PV workflow as well as prose it out :)
[17:23:04] <wikibugs>	 Analytics / Tech community metrics: Bugzilla response time: "Longest time without comment" is actually "Longest time without any comment by non-reporter"? - https://bugzilla.wikimedia.org/67589#c1 (Quim Gil) NEW>ASSI a:Quim Gil Yes, this must be notes in the final descriptions that I must writ...
[17:23:17] <wikibugs>	 Analytics / Tech community metrics: Key performance indicator: Bugzilla response time - https://bugzilla.wikimedia.org/61561 (Quim Gil)
[18:14:32] <wikibugs>	 Analytics / General/Unknown: Packetloss issues on oxygen (and analytics1003) - https://bugzilla.wikimedia.org/67694#c7 (Dan Andreescu) Just writing some notes for myself on how to troubleshoot this.  For now, I'm putting it on hold until I get two more things:  1. kafka broker data is available to quer...
[18:14:57] <ottomata>	 milimetric: you want to chat about that?
[18:15:04] <ottomata>	 about udp2log setup ?
[18:15:08] <ottomata>	 sorry we didn't get to before
[18:15:09] <ottomata>	 i can now
[18:15:13] <milimetric>	 no problem
[18:15:15] <milimetric>	 k, trap
[18:51:33] <grrrit-wm1>	 (PS1) Terrrydactyl: [WIP] Add autcomplete to tags [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/145039
[18:51:40] <terrrydactyl>	 milimetric ^
[18:52:46] <milimetric>	 oh, you should probably get rid of that authorized column in your models
[18:53:01] <milimetric>	 but maybe talk to Jaime first
[18:55:48] <ottomata>	 driving back to NYC now, ttyl
[20:18:53] <milimetric>	 hey jgonera, have you seen karma test runner?
[20:19:06] <jgonera>	 milimetric, no, what is it?
[20:19:20] <milimetric>	 I've been using it while trying out tools for dashboarding, it's so cool
[20:19:35] <milimetric>	 so it's an ultra-fast test runner written by the angular team
[20:19:56] <milimetric>	 it doesn't care what you write your tests in, but it watches your files and instantly runs all your tests again, IN a browser
[20:20:08] <milimetric>	 so it runs, spawns a browser, and hoists tests over and results back
[20:20:24] <milimetric>	 http://karma-runner.github.io/0.12/index.html
[20:20:39] <milimetric>	 I use it with Jasmine, but you can use it with qunit, etc.
[20:20:56] <milimetric>	 thought I'd mention since I know you were into testing and using Cucumber which I always found somewhat painful
[20:21:36] <jgonera>	 milimetric, is this for unit or acceptance tests?
[20:22:24] <milimetric>	 unit jgonera, but since it runs stuff in the browser I guess you could do whatever
[20:22:46] <milimetric>	 though selenium might be best for that
[20:23:24] <jgonera>	 milimetric, I'll have a look at it for our unit tests, but I doubt the QA team would consider rewriting all their tests in something different than Cucumber/watir ;)
[20:24:27] <milimetric>	 well, it's worth a look, short video explains it and I have a configuration for my setup if you get stuck jgonera, just let me know
[20:24:38] <jgonera>	 ok, thanks!
[20:26:12] <grrrit-wm1>	 (PS2) Terrrydactyl: Add autcomplete to tags [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/145039
[20:42:51] <grrrit-wm1>	 (PS3) Terrrydactyl: Add autcomplete to tags [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/145039
[20:54:38] <grrrit-wm1>	 (Abandoned) Jdlrobson: Add graph showing the successful edits per device [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/110470 (owner: Jdlrobson)
[22:04:33] <milimetric>	 there definitely seems to be a smaller but still significant spike in UDP packets received corresponding with today's game:
[22:04:33] <milimetric>	 http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Miscellaneous+eqiad&h=analytics1003.eqiad.wmnet&jr=&js=&v=859712030&m=UDP_InDatagrams&vl=packets&ti=UDP+Packets+Received
[22:04:41] <milimetric>	 Ironholds / DarTar: ^
[22:08:17] <Ironholds>	 milimetric, cool!
[23:55:15] <marktraceur>	 milimetric: nlwiki < dewiki
[23:55:46] <marktraceur>	 Just wait for the final.