[00:00:22] <legoktm>	 HaeB: no, it just only joins channels when it has a message to send. it always idles in #wikimedia-labs though
[00:03:19] <HaeB>	 legoktm: ok thanks, i thought i had seen some eligible phabricator updates in the meantime (i.e. which wikibugs usually posts here), but maybe these projects are not covered here
[01:21:28] <wikibugs>	 Analytics, Operations, Performance-Team, Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#2328835 (Jdlrobson)
[05:43:47] <wikibugs>	 Quarry: feature for quarry tool: show quarry time used - https://phabricator.wikimedia.org/T136266#2329010 (Yamaha5)
[07:38:09] <wikibugs>	 Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2329153 (mobrovac) We plan to start enforcing the limits next week. I think we should start with the currently-set limits (10 reqs/s/endpoint/cli...
[07:47:26] <elukey>	 o/
[08:20:36] <joal>	 Hi elukey :)
[08:20:41] <joal>	 cassandra debrief ?
[08:26:47] <elukey>	 joal: I am trying to figure out a raid configuration for analytics1047 atm, do you mind if we do it later?
[08:26:57] <joal>	 absolutely not :)
[08:27:12] <elukey>	 the drives are not coming up in the right order in /dev
[08:27:13] <elukey>	 sigh
[08:27:18] <elukey>	 and I think I found the problem
[08:27:19] <joal>	 arrf :(
[08:27:28] <joal>	 Well done ! Good luck with that elukey
[08:27:38] <elukey>	 but I need to figure out the megacli command :(
[08:27:43] <elukey>	 that is always a nightmare
[08:27:52] <elukey>	 joal: sync in ~30 mins?
[08:27:59] <joal>	 when you have time :)
[08:55:07] <elukey>	 ah joal, whenever you have time can you show me how to try the spark dynamic allocation?
[08:55:14] <elukey>	 I am curious
[08:55:32] <joal>	 elukey: sure :)
[09:11:36] <wikibugs>	 Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2329328 (mobrovac) RB logged a total of [30K excesses in the last 24h](https://logstash.wikimedia.org/#dashboard/temp/AVTsUtpi_LTxu7wlBfI-). The...
[09:14:29] <elukey>	 joal: I am ready
[09:14:59] <elukey>	 figured out the issue but I have no idea about what command I need to execute to fix the disk :P
[09:16:34] <elukey>	 it seems that our RAID controller on analytics1028->1058 creates Virtual Drives, each one implementing a single disk RAID0
[09:18:00] <joal>	 elukey: I tought hadoop was using JBOD?
[09:18:27] <joal>	 elukey: batcave ?
[09:18:31] <elukey>	 yeah Chris suggested the same to me but it is not what I am seeing
[09:18:42] <elukey>	 https://phabricator.wikimedia.org/T134056#2329295
[09:18:45] <elukey>	 joal --^
[09:18:50] <joal>	 hmmm ... We should discuss that with andrew as well then
[09:18:52] <elukey>	 going to the cave
[09:19:08] <elukey>	 well it is fine for our purposes
[09:19:18] <elukey>	 it doesn't change a lot
[09:19:22] <elukey>	 but it is weird :D
[09:19:31] <joal>	 sure it's weird
[10:57:29] <elukey>	 elukey@analytics1047:~$ sudo service hadoop-hdfs-datanode status * Hadoop datanode is running
[10:57:42] <elukey>	 aah pasted incorrectly, buut works now :)
[11:17:19] <elukey>	 /eqiad/B/3RUNNINGanalytics1047.eqiad.wmnet:8041analytics1047.eqiad.wmnet:8042
[11:17:25] <elukey>	 again
[11:17:30] <elukey>	 :@
[11:17:38] <elukey>	 anyhow, analytics1047 running
[11:19:11] <wikibugs>	 Quarry: show time of execution in quarry - https://phabricator.wikimedia.org/T136266#2329589 (Ladsgroup)
[11:27:33] <joal>	 elukey: spark cached timeout confirmed :)
[11:29:48] <elukey>	 wowwwww
[11:41:41] * elukey lunch!
[12:28:46] <aharoni>	 Hallo
[12:29:08] <aharoni>	 A couple of weeks ago I got some queries from Ellery,
[12:29:25] <aharoni>	 and I'm trying to run them in beeline now
[12:29:35] <aharoni>	 they used to work then, but they don't work now
[12:29:52] <aharoni>	 "ERROR : Failed with exception Unable to move source hdfs://analytics-hadoop/tmp/hive-staging_hive_2016-05-26_12-26-15_685_793302850668519493-4/-ext-10001 to destination hdfs://analytics-hadoop/user/hive/warehouse/ill.db/cross_wiki_navigation"
[12:31:27] <aharoni>	 madhuvishy: milimetric ^
[12:33:17] <aharoni>	 ... actually, it probably worked on hive, and trying beeling is the new things
[12:33:38] <aharoni>	 no, fails on hive, too.
[12:46:59] <milimetric>	 hm, aharoni's problem looks like a permissions issue maybe
[12:47:36] <wikibugs>	 Analytics-Kanban, DC-Ops, Operations, ops-eqiad: I/O issues for /dev/sdd on analytics1047.eqiad.wmnet - https://phabricator.wikimedia.org/T134056#2329742 (elukey)
[12:48:36] <wikibugs>	 Analytics-Kanban, DC-Ops, Operations, ops-eqiad: I/O issues for /dev/sdd on analytics1047.eqiad.wmnet - https://phabricator.wikimedia.org/T134056#2253604 (elukey) a:Cmjohnson>elukey
[13:18:56] <aharoni>	 hi halfak_
[13:19:30] <halfak_>	 o/ aharoni
[13:19:32] <halfak_>	 (At hackathon.  Might disappear suddenly)
[13:22:05] <halfak_>	 Wifi is bad too
[13:22:11] <wikibugs>	 Analytics: hive / beeline query for interlanguage links fails - https://phabricator.wikimedia.org/T136295#2329877 (Amire80)
[13:22:28] <wikibugs>	 Analytics: hive / beeline query for interlanguage links fails - https://phabricator.wikimedia.org/T136295#2329890 (Amire80)
[13:23:00] <aharoni>	 halfak_: would you, by any chance, have an idea how to fix the bug above? ^ https://phabricator.wikimedia.org/T136295
[13:24:06] <aharoni>	 the query seems to work. The error messages makes me suspect that it's some kind of a permission or server configuration problem.
[13:25:38] <aharoni>	 ottomata: hi. can you maybe take a look at https://phabricator.wikimedia.org/T136295 ?
[13:29:32] <aharoni>	 hi leila
[13:29:37] <aharoni>	 [ I'm bugging everybody today ]
[13:29:39] <leila>	 hey aharoni.
[13:29:50] <leila>	 [haha. sure.]
[13:29:53] <aharoni>	 can you maybe take a look at https://phabricator.wikimedia.org/T136295 ?
[13:30:08] <elukey>	 ottomata: o/
[13:30:14] <elukey>	 just added https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Swapping_broken_disk
[13:30:26] <elukey>	 far from complete but it might be useful in the future
[13:30:54] <wikibugs>	 Analytics-Kanban, DC-Ops, Operations, ops-eqiad: I/O issues for /dev/sdd on analytics1047.eqiad.wmnet - https://phabricator.wikimedia.org/T134056#2329937 (elukey) Added some documentation in:  https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Swapping_broken_disk
[13:33:07] <leila>	 aharoni: I'm not sure what's going on. It's Andrew's territory as far as I can tell.
[13:33:40] <ottomata1>	 ahhh internet....
[13:34:07] <aharoni>	 Andrew = ottomata1 , right?
[13:34:21] <joal>	 aharoni: Hi
[13:34:42] <joal>	 aharoni: Can you tell me your username on stat1002 please (for checking permissions and so0
[13:34:42] <aharoni>	 hi joal
[13:35:08] <aharoni>	 joal : amire80
[13:35:49] <ottomata1>	 ja hi, with you all shortly...having problems
[13:35:57] <joal>	 np ottomata1
[13:36:07] <joal>	 aharoni: currently checking stuff
[13:37:30] <joal>	 aharoni: my thoughts are that since the table already exists and was created by Ellery, you don't have persmission to use it
[13:37:55] <aharoni>	 joal: maybe, but I thought that I dropped it and created a new one
[13:37:56] <joal>	 aharoni: I suggest creating your own db: CREATE DATABASE amire80
[13:38:10] <aharoni>	 Am I supposed to create a new one every time? That's what I understood, but I might be wrong.
[13:38:18] <aharoni>	 OH.
[13:38:20] <aharoni>	 Let me try.
[13:39:22] <aharoni>	 joal: running...
[13:39:41] <joal>	 aharoni: forgot the ';' at the end
[13:40:26] <aharoni>	 joal: I figured that out...
[13:40:36] <joal>	 huhu :D
[13:40:37] <aharoni>	 joal but CREATE TABLE failed with the same error
[13:41:16] <joal>	 aharoni: definitely permissions issue - let's wait for ottomata when he's back to double check tou are in the correct group
[13:42:05] <ottomata>	 elukey:  nice work with megacli!
[13:42:07] <ottomata>	 "Note for future readers: I had to create the "/var/lib/hadoop/data/hdfs" and "/var/lib/hadoop/yarn" directories "
[13:42:20] <ottomata>	 doesn't puppet create them?
[13:43:12] <aharoni>	 ottomata seems to be here
[13:43:14] <aharoni>	 :)
[13:43:16] <elukey>	 mmmm now that I think about it I might have created it before running puppet
[13:43:41] <joal>	 ottomata: can you double check user amire80 is in the correct group to create hive tables ?
[13:43:49] <joal>	 ottomata: please :)
[13:43:58] <elukey>	 ottomata: ah yes I didn't run puppet before that because I wanted to check the hdfs daemeon
[13:44:01] <elukey>	 *daemon
[13:44:10] <ottomata>	 aye cool
[13:44:12] <ottomata>	 checking
[13:44:45] <ottomata>	 joal:  amire80 looks good
[13:44:49] <ottomata>	 looking at that ticket
[13:45:03] <joal>	 ottomata: he has permissions issues creating his own DB in hive :(
[13:45:15] <joal>	 ottomata: I'm gonna create the folder and change rights, then retest
[13:46:09] <joal>	 aharoni: Actually the database creation seems to have worked
[13:46:41] <aharoni>	 joal: yes, but after that something fails, and I don't see it when I run `show tables;`
[13:47:07] <joal>	 aharoni: have you changed the db name in the table creation query ?
[13:47:42] <aharoni>	 joal: <facepalm> no
[13:47:44] <ottomata>	 am looking, things seem normal, althouhg ill.db looks like it was created by ellery
[13:47:56] <joal>	 aharoni: classical mistake :)
[13:48:25] <joal>	 ottomata: looks like we should be good, sorry for having disturb ;)
[13:48:30] <ottomata>	 oook :)
[13:51:22] <wikibugs>	 Analytics: hive / beeline query for interlanguage links fails - https://phabricator.wikimedia.org/T136295#2330065 (Amire80) Open>Resolved a:JAllemandou Resolved by creating my own database and creating the table in it.  Thanks to @Ottomata and @JAllemandou!
[13:51:53] <joal>	 aharoni: this could be of interest to you : https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Queries
[13:53:15] <aharoni>	 joal: thanks. should it be hive or beeline? is there any difference?
[13:53:33] <milimetric>	 aharoni: beeline is better but a bit newer
[13:53:34] <joal>	 aharoni: there are difference, and you should go for beeline
[13:53:59] <joal>	 only readon for which beeline might cause problem is when loading local jars
[13:54:06] <aharoni>	 joal: OK. and how do I exit the prompt?
[13:54:12] <joal>	 CTRL-D
[13:54:53] <joal>	 aharoni: CTRL+D is the usual termination shortcut on CLI on linux
[13:55:25] <aharoni>	 joal: That's what I did, but I wondered whether there's a Real Command.
[13:55:47] <joal>	 hmm aharoni, I don't know
[13:56:29] <joal>	 Found it: !quit
[13:57:16] <joal>	 And actually aharoni: !<SQLLIne-COMMAND> works on beeline (http://sqlline.sourceforge.net/)
[14:00:36] <joal>	 elukey, ottomata : Any idea on what happens on misc? Still a lot of errors
[14:01:30] <joal>	 Also, ottomata elukey : Shall I manually mark misc-2016-05-25T19:00Z as ok to be processed ?
[14:02:46] <ottomata>	  joal lemme ask in #traffic
[14:02:51] <joal>	 sure ottomata
[14:03:24] <milimetric>	 yall already talk about the loss yesterday? 5.7%?
[14:03:41] <joal>	 milimetric: that's exactly the topic :)
[14:04:28] <milimetric>	 oh, right, I got lost in the other thread, so we don't know
[14:05:00] <joal>	 milimetric: ottomata is asking in traffic, double checking those warning/errors are normal
[14:05:29] <milimetric>	 what in the world is #traffic?!
[14:05:52] <elukey>	 wikimedia-traffic :)
[14:05:55] <joal>	 milimetric: on the other hand, I just realized that merging load and refine jobs prevent me from rerunning that and forcing it - I need to modify the oozie code !
[14:06:26] <elukey>	 milimetric: ops spin off for Varnish/Traffic related discussions
[14:06:36] <milimetric>	 oh wow, I had no idea, thanks elukey
[14:06:41] <milimetric>	 did it start recently?
[14:07:00] <elukey>	 more or less when ema joined, so January/February
[14:07:20] <milimetric>	 cool, thx
[14:09:00] <milimetric>	 woah... can someone else check out store.wikipedia.org?
[14:09:19] <milimetric>	 it looks like an attack where someone hijacks my session and sends me to random places
[14:09:34] <milimetric>	 asking to get my location, to install add-ons, etc.
[14:09:42] <joal>	 milimetric: works for me
[14:09:45] <milimetric>	 wondering if it's just my computer or everyone
[14:09:55] <milimetric>	 woah working for me too now!
[14:10:00] <milimetric>	 ... hmmmm
[14:10:25] <milimetric>	 oh!! it was a typo!! store.wikipeda.org
[14:10:43] <elukey>	 srsly?
[14:10:53] <milimetric>	 they're typo-squatting us!!
[14:10:54] <milimetric>	 ew
[14:11:06] <wikibugs>	 Analytics-Kanban: Add a way to force oozie load job to run even if unsuccesful dataloss check. - https://phabricator.wikimedia.org/T136308#2330129 (JAllemandou)
[14:11:10] <elukey>	 what the hell..
[14:11:11] <elukey>	 hahahahh
[14:11:12] <milimetric>	 I thought that went out of fashion
[14:11:36] <milimetric>	 is this the kind of thing we tell ops about?
[14:11:47] <joal>	 milimetric: it'll never be out of fashion unfortunately :(
[14:11:51] <milimetric>	 like, do we actively fight this by buying the domains?
[14:12:05] <milimetric>	 I'll mention it just in case
[14:12:18] <elukey>	 maybe with security
[14:12:24] <elukey>	 reporting just in case
[14:15:01] * joal is AFK for a few minutes
[14:25:54] <ottomata>	 milimetric:  joal, can't remember what we said for disk layout for the druid hosts
[14:26:01] <ottomata>	 raid 10 across all 4 ssds?
[14:26:47] <milimetric>	 that's what we said yea
[14:27:15] <ottomata>	 k
[14:27:17] <milimetric>	 disk seems unimportant in druid for two reasons
[14:27:25] <milimetric>	 1. if we can't memory map it it's no good anyway
[14:27:32] <milimetric>	 2. we keep backups of this in hdfs anyway
[14:27:53] <milimetric>	 so as far as I'm concerned, we can do whatever for disk for now until we get more experience
[14:29:44] <milimetric>	 mforns: you around?
[14:30:16] <milimetric>	 joal: after a bunch of TSV / JSON craziness yesterday I'm convinced we should just use sqoop
[14:30:28] <milimetric>	 you were pretty against it so do you wanna chat?
[14:32:57] <mforns>	 milimetric, hi!
[14:33:26] <milimetric>	 hey mforns wondering if you thought more about sqoop since yesterday
[14:33:32] <mforns>	 milimetric, I'm testing the json serde
[14:33:38] <milimetric>	 k
[14:33:41] <mforns>	 not yet started with swoop
[14:33:49] <mforns>	 *sqoop
[14:39:53] <milimetric>	 k, I'll do that now then
[14:40:02] <wikibugs>	 Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2330259 (Nuria) @mobrovac I think last burst has UA as ruby, we know some popular tools on labs cause bursts too but there are many automated scr...
[14:43:46] <mforns>	 milimetric, ok
[14:47:36] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: More solid Eventlogging alarms for raw/validated {oryx} [8 pts] - https://phabricator.wikimedia.org/T116035#2330292 (ilevy) Resolved>Open It looks like a local fork, check_graphite_until_temp, is still in use....
[14:48:26] <elukey>	 ottomata, joal: can you point me to the oozie snippet that calculates the loss alarm?
[14:48:34] <elukey>	 Just want to add all the info in the task
[14:49:26] <elukey>	 mmm maybe grepping webrequest-load-check_sequence_statistics in refinery
[14:50:25] <ottomata>	 elukey:
[14:50:25] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/generate_sequence_statistics.hql
[14:50:27] <ottomata>	 and then also
[14:50:27] <ottomata>	 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/generate_sequence_statistics_hourly.hql
[14:51:01] <elukey>	 thanks :)
[14:51:15] <ottomata>	 :)
[14:51:53] <milimetric>	 ottomata: I've got a permissions riddle revolving around stat1002,3,4 and /etc/mysql/conf.d/research-client.cnf
[14:52:07] <ottomata>	 haha
[14:52:10] <ottomata>	 yeah?
[14:52:49] <milimetric>	 sorry, AC went weird
[14:52:50] <ottomata>	 milimetric:
[14:52:50] <ottomata>	 ls -l /etc/mysql/conf.d/
[14:52:52] <ottomata>	 can help
[14:52:52] <ottomata>	 :)
[14:53:00] <ottomata>	 what's the riddle?
[14:53:08] <milimetric>	 ottomata: ok, so on 1003 everything works, file's there and I have access
[14:53:14] <milimetric>	 on 1002 it's there but I have no access
[14:53:18] <milimetric>	 on 1004 it's not there
[14:53:40] <ottomata>	 right
[14:53:41] <milimetric>	 ideally it'd be there on 1002 or 1004 because I'd like to sqoop from the dbs it gives me access to
[14:53:50] <milimetric>	 also, on 1002 I have no access to my own home dir
[14:53:53] <milimetric>	 even though it says I do
[14:53:59] <ottomata>	 milimetric:  on 1002 you can use
[14:54:03] <ottomata>	 analytics-research-client.cnf
[14:54:07] <ottomata>	 on 1004, it doesn't exist
[14:54:14] <ottomata>	 but it could!
[14:54:18] <ottomata>	 the analytics-research-client.cnf one
[14:55:12] <milimetric>	 aha, gotcha
[14:55:26] <milimetric>	 I'll try on 1002, ok last problem:
[14:55:37] <milimetric>	 on 1002, I can go to my home directory and then:
[14:55:40] <milimetric>	 touch ~/something
[14:55:42] <milimetric>	 but if I try:
[14:55:46] <milimetric>	 touch ~/test.tsv
[14:55:47] <milimetric>	 it fails
[14:55:51] <ottomata>	 haha
[14:55:55] <ottomata>	 htat is weird
[14:56:00] <milimetric>	 not so if I do touch ~/test2.tsv
[14:56:02] <milimetric>	 that's fine
[14:56:11] <ottomata>	 haha, does test.tsv exist?
[14:56:15] <ottomata>	 with some weird perms?
[14:56:15] <milimetric>	 I thought my home dir was broken but it's just that one file name!
[14:56:16] <milimetric>	 no, it doesn't
[14:56:27] <milimetric>	 or if it does someone else made it and I can't even see it in ls
[14:56:35] <ottomata>	 ja it does
[14:56:36] <ottomata>	 lrwxrwxrwx   1 milimetric wikidev       24 Nov 20  2015 test.tsv -> /home/joal/json_test.csv
[14:57:25] <milimetric>	 oh!  it's a link
[14:57:43] <ottomata>	 want me to remove it?
[14:57:57] <milimetric>	 hm those don't list when just doing "ls"... weird
[14:58:04] <ottomata>	 they should that is weird
[14:58:30] <ottomata>	 haa, uh i don't see any of your test. files!
[14:58:32] <ottomata>	 in just ls
[14:58:47] <ottomata>	 oh you just removed them?
[14:58:48] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330347 (elukey) p:Triage>High
[14:58:57] <elukey>	 ottomata --^
[14:58:58] <milimetric>	 I just removed them
[14:59:04] <milimetric>	 yeah, I tried rm test.tsv and it didn't work
[14:59:11] <milimetric>	 but then I tried rm test* and it removed the link too
[14:59:22] <milimetric>	 it's settled I shall never understand the command line
[14:59:27] <milimetric>	 thx ottomata
[15:02:55] <ottomata>	 :)
[15:04:35] <wikibugs>	 Quarry: puppet disabled on quarry-main-01 - https://phabricator.wikimedia.org/T136315#2330367 (valhallasw)
[15:06:24] <joal>	 elukey: I think I have found a hack not to have to change the oozie for rerun things
[15:06:43] <joal>	 milimetric: yes, let's talk around sqoop :)
[15:07:06] <elukey>	 joal: you can call it "elegant workaround" to make it more appealing :P
[15:07:21] <milimetric>	 joal: k, cave
[15:07:22] <joal>	 ottomata: any news on data loss?
[15:07:26] <joal>	 OMW milimetric
[15:07:37] <elukey>	 joal: I opened a phab task, --^
[15:07:55] <joal>	 elukey: did that as well :)
[15:08:02] <elukey>	 ah snap
[15:08:20] <elukey>	 mine is https://phabricator.wikimedia.org/T136314
[15:08:41] <joal>	 elukey: https://phabricator.wikimedia.org/T136308
[15:09:49] <elukey>	 ahhh that's related
[15:09:57] <elukey>	 okok non overlapping tasks
[15:43:08] <milimetric>	 ottomata: do you know how ellery gets sqoop to work despite the problem that hadoop nodes don't have access to connect to the database?
[15:44:02] <ottomata>	 milimetric:  they dont'?
[15:44:09] <ottomata>	 i think they do, no? i think we poked a hole in the firewall for that
[15:44:32] <milimetric>	 hm... i get Access denied for user 'research'@'10.64.5.102'
[15:44:38] <milimetric>	 I assumed that IP was a node
[15:44:45] <milimetric>	 oh wait, access denied means it did connect
[15:44:55] <milimetric>	 so does he use a different user or something?
[15:45:16] <milimetric>	 *that IP was a hadoop node I mean
[15:46:17] <ottomata>	 hm
[15:46:18] <ottomata>	 that i'm not sure
[15:55:02] <milimetric>	 ottomata: sorry, the --password-file option in sqoop was reading my password incorrectly
[15:55:12] <milimetric>	 false alarm about the access / connection problem
[15:55:18] <ottomata>	 cool
[16:01:32] <madhuvishy>	 ottomata: the script yesterday was working fine - those misnamed files were just sticking around from some small bug before. the reason it wasn't adding the jars was that those jars already exist (i'm using versions that already exist in refinery to test)
[16:03:43] <ottomata>	 ah! :)
[16:29:03] <wikibugs>	 Analytics-Kanban: Add a way to force oozie load job to run even if unsuccesful dataloss check. - https://phabricator.wikimedia.org/T136308#2330808 (JAllemandou) Open>Invalid
[16:36:51] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330849 (ema) When are we seeing those inconsistencies? Any specific timeframes?
[16:39:01] <wikibugs>	 Analytics-Kanban: Extract edit oriented data from MySQL for small  wiki - https://phabricator.wikimedia.org/T134790#2330852 (Milimetric) a:Milimetric
[16:39:16] <wikibugs>	 Analytics: Get pageview dataset ready to be loaded to druid to test querying capabilities - https://phabricator.wikimedia.org/T136330#2330856 (Nuria)
[16:40:50] <wikibugs>	 Analytics: Get pageview dataset ready to be loaded to druid to test querying capabilities - https://phabricator.wikimedia.org/T136330#2330873 (Nuria) 1. start with 1 day of data and test quqry capabilities 2. test a month later 3. test 3 months of data   This is contingent on having druid on puppet and worki...
[16:45:12] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330888 (elukey)
[16:45:45] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330335 (elukey) >>! In T136314#2330849, @ema wrote: > When are we seeing those inconsistencies? Any specific timeframes?   Need to query Hive and Hadoop...
[16:46:58] <wikibugs>	 Analytics: Enable pivot ui so non analytics engineers can query druid pageview data (poc) - https://phabricator.wikimedia.org/T136331#2330895 (Nuria)
[16:48:24] <wikibugs>	 Analytics: Get pageview dataset ready to be loaded to druid to test querying capabilities - https://phabricator.wikimedia.org/T136330#2330856 (Nuria) Format? Json or tsv
[16:53:43] <wikibugs>	 Analytics-Kanban: Get pageview dataset ready to be loaded to druid to test querying capabilities - https://phabricator.wikimedia.org/T136330#2330856 (Nuria)
[17:04:45] <wikibugs>	 Analytics: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2330988 (Nuria)
[17:06:50] <wikibugs>	 Analytics: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2330992 (Nuria) 1.  Try to find users of files: - e-mail analytics@ - see users on unixgroup that has access to files and e-mail them individually - communicate with research-int...
[17:09:20] <wikibugs>	 Analytics-Kanban: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2144772 (Nuria)
[17:09:29] <wikibugs>	 Quarry: puppet disabled on quarry-main-01 - https://phabricator.wikimedia.org/T136315#2330998 (yuvipanda) Yes, when it restarted it came up trying to use python3 instead of 2, and I disabled it and hand hacked to fix it. Must fix puppet...
[17:19:31] <wikibugs>	 Analytics, DBA: dbstore1002 crashed - https://phabricator.wikimedia.org/T136333#2331074 (jcrespo)
[17:26:09] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2330335 (Nuria) 1. On labs or perhaps prod: Generate lots of request and a sighup and see if all requests ids are present, try to find repro for dropping...
[17:28:33] <wikibugs>	 Analytics-Kanban, Operations, Traffic: Verify why varnishkafka stats and webrequest logs count differs - https://phabricator.wikimedia.org/T136314#2331150 (Ottomata) https://gist.github.com/ottomata/7048012
[17:30:13] <wikibugs>	 Analytics-Kanban: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2331163 (Nuria) a:madhuvishy
[17:39:27] <wikibugs>	 Analytics-Kanban, EventBus, Patch-For-Review: Propose evolution of Mediawiki EventBus schemas to match needed data for Analytics need - https://phabricator.wikimedia.org/T134502#2331201 (Ottomata) @aaron can we find some mins to discuss above?  I'm not sure if you are the right person, but if not you...
[17:44:39] <elukey>	 logging off a-team!
[17:44:41] <elukey>	 byeeee
[17:47:24] <ottomata>	 laters!
[18:09:44] <wikibugs>	 Analytics-Kanban, RESTBase, Services, RESTBase-API, User-mobrovac: Enable rate limiting on pageview api - https://phabricator.wikimedia.org/T135240#2331321 (GWicke) Lets also document these limits for each API end point by adding a list item along `Stability`.
[18:27:08] <wikibugs>	 Analytics, DBA: dbstore1002 crashed - https://phabricator.wikimedia.org/T136333#2331371 (jcrespo) The slave was not using GTIDs, which meant that at least x1 got desynced. S1, however, was stopped. Skipping a single delete on x1 seemed to do the work.
[18:27:44] <wikibugs>	 Analytics, DBA: dbstore1002 crashed - https://phabricator.wikimedia.org/T136333#2331373 (jcrespo) p:Triage>Low Low because issues are no longer ongoing.
[18:46:44] <milimetric>	 mforns: I'm back working on this export, didn't get a chance to do it during metrics because the db crashed
[18:46:47] <milimetric>	 (jaime's message)
[18:47:00] <milimetric>	 mforns: it does look like at least some of the fields are weird: select * from milimetric.simplewiki_archive limit 10;
[18:47:14] <milimetric>	 those are deleted pages though so maybe they were deleted because they were nonse
[18:47:17] <milimetric>	 *nonsense
[18:47:21] <mforns>	 milimetric, I see
[18:47:26] <milimetric>	 or maybe they just didn't get exported / imported ok
[18:47:28] <mforns>	 let me execute
[18:47:40] <milimetric>	 yep, I'll keep doing the other tables
[18:48:36] <mforns>	 milimetric, OK, I still have a lot of work with the queries, also started 20 mins ago
[18:48:36] <joal>	 milimetric: hmm, have you checked the db crash was unrealted to the export ?
[18:48:43] <milimetric>	 yes
[18:48:49] <milimetric>	 jaime said it was fine
[18:48:49] <joal>	 ok :)
[18:48:58] <milimetric>	 but that queries will be slow from now on
[18:48:58] <joal>	 just double checking\ :)
[18:49:08] <milimetric>	 he was right, there was half as much data in archive as logging, but it took twice as long
[18:49:31] <milimetric>	 np mforns I'll help you when I see how long these jobs are taking
[19:12:07] <nuria_>	 milimetric: where is the advice of  "limit of 500 reqs per second" for users posted to our docs?
[19:12:28] <nuria_>	 milimetric: i updated our internal docs but cannot find this info on the external ones
[19:12:29] <milimetric>	 nuria_: looking
[19:12:35] <nuria_>	 milimetric: internal docs: https://wikitech.wikimedia.org/wiki/Analytics/AQS#2016-05-26 cc elukey
[19:15:17] <milimetric>	 aha, nuria_ it's a more general thing: https://wikimedia.org/api/rest_v1/?doc
[19:15:23] <milimetric>	 it's right in the intro text there
[19:15:24] <nuria_>	 k
[19:15:29] <milimetric>	 so that's nothing to do with us
[19:15:51] <nuria_>	 right, i will document 429 on our wikitech docs though
[19:16:09] <nuria_>	 milimetric: will send links to list with docs updated in a bit
[19:16:26] <milimetric>	 sweet
[19:16:29] <milimetric>	 thx!
[20:04:59] <wikibugs>	 Analytics, Analytics-Wikistats, Internet-Archive: Total page view numbers on Wikistats do not match new page view definition - https://phabricator.wikimedia.org/T126579#2331761 (ezachte) Ah good to see there is white list.   > differences are between 1 and 4%, correct?  Actually 1/10 of that, the ave...
[20:17:42] <ottomata>	 milimetric:  btw, I am prepping for puppetizing druid boxes in prod
[20:17:53] <ottomata>	 gotta get some deb crap straight, but ja
[20:18:06] <milimetric>	 ottomata: thanks, I think joseph will get that pageview data tomorrow so no worries
[20:18:18] <milimetric>	 we're in the middle of some insane edit data juggling :)
[20:18:20] <milimetric>	 it's kinda fun
[20:19:05] <ottomata>	 :)
[20:19:09] <joal>	 milimetric: I'll ask mforns tomorrow, I want to know more on data jugling :)
[20:19:33] <milimetric>	 it's so crazy dude, I can't tell you now you'll dream about it
[20:19:34] <joal>	 ottomata: I have a script to generate data
[20:19:37] <milimetric>	 tty tomorrow :)
[20:19:42] <joal>	 inideed
[20:19:52] <mforns>	 :]
[20:20:22] <joal>	 ottomata: how should I format druid timestamp? timestamp in second, in millis, , other formt?
[20:20:36] <ottomata>	 ISO8601! :)
[20:21:07] <ottomata>	 2015-09-01T00:00:00Z
[20:21:11] <ottomata>	 i think druid wants them that way anywa
[20:21:14] <ottomata>	 or likes them that way
[20:21:32] <joal>	 ok ottomata, will do :)
[20:22:11] <joal>	 ottomata: one last thing: can I have fields in the json that are not used ( I assume yes, but prefer to check|)
[20:22:44] <ottomata>	 uhm, i don't know, that are not used in the indexing task?
[20:22:45] <ottomata>	 yes
[20:22:46] <ottomata>	 pretty sure
[20:22:50] <ottomata>	 but, um
[20:22:51] <ottomata>	 actually, i don't konw
[20:23:15] <joal>	 ottomata: I need a date (2016-05-25) format to split data daily
[20:23:29] <joal>	 I'm pretty sure as well I can have that and not use it
[20:24:21] <ottomata>	 aye probably
[20:26:52] <joal>	 ottomata: I'll daily and monthly data tomorrow :)
[20:27:37] <ottomata>	 i might have some problems getting the hadoop deps into jessie in prod
[20:27:49] <ottomata>	 not sure how to do it properly...and my european helpers are all offline :/
[20:28:08] <joal>	 hm, I can't really help I guess
[20:28:26] <joal>	 if not tomorrow, then next week :)
[20:49:03] <madhuvishy>	 joal: wanted to ask you - when we do the next refinery deployment, can we dry run the jenkins jobs?
[20:49:09] <madhuvishy>	 actually
[20:49:14] <madhuvishy>	 not dry run
[20:49:24] <madhuvishy>	 for rea
[20:49:26] <joal>	 madhuvishy: For sure !
[20:49:26] <madhuvishy>	 real*
[20:49:41] <joal>	 Probably a dry-run first, then a real one :)
[20:49:51] <madhuvishy>	 joal: ha ha yeah that's possible
[20:50:26] <madhuvishy>	 ottomata: do the new archiva creds work, or are you still working on it?
[20:50:49] <ottomata>	 they shoudl work
[20:50:56] <madhuvishy>	 ok cool
[20:51:38] <joal>	 ottomata: hdfs dfs -text /user/joal/pv_druid/2016-01-01/part-00031.gz | less
[20:51:59] <madhuvishy>	 ottomata: then we still need https://gerrit.wikimedia.org/r/#/c/290639/8 to be merged, and https://phabricator.wikimedia.org/T136221 to be set up before doing the test run. No hurry etc
[21:05:19] <ori>	 hue is really slow
[21:05:28] <ori>	 i don't mean the underlying queries it performs -- just the web interface
[21:27:59] <milimetric>	 :) it's much much faster than it used to be
[21:28:05] <milimetric>	 but yeah, that's what you get with python
[21:28:06] <milimetric>	 :P
[23:21:08] <wikibugs>	 Analytics, Pageviews-API, RESTBase-API: AQS: query multiple articles at the same time - https://phabricator.wikimedia.org/T118508#1802156 (GWicke) As with large & random time ranges, querying many titles in a single request makes individual requests more expensive & hard to effectively rate-limit, an...
[23:52:32] <ottomata>	 milimetric:  y still there?