[00:15:59] <wikibugs>	 Phabricator, Analytics-Tech-community-metrics: SQL user/grant for phabricator statistics script - https://phabricator.wikimedia.org/T78311#941010 (Springle) phstats now has access to both phabricator_maniphest and phabricator_user.
[01:49:22] <grrrit-wm1>	 (PS4) Nuria: [WIP] Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017
[03:12:44] <grrrit-wm1>	 (PS5) Nuria: [WIP] Mobile apps oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/181017
[13:02:32] <wikibugs>	 Engineering-Community, Analytics-Tech-community-metrics, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#941542 (Dzahn)
[13:02:35] <wikibugs>	 Analytics-Tech-community-metrics, Phabricator: SQL user/grant for phabricator statistics script - https://phabricator.wikimedia.org/T78311#941540 (Dzahn) Open>Resolved thanks :)
[13:39:58] <wikibugs>	 Engineering-Community, Analytics-Tech-community-metrics, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#941556 (Dzahn) Quim said:  ( for some reason I can't hit "quote" on his comment) >Is it possible to add these to the report? >Number of accounts cr...
[13:42:43] <wikibugs>	 Analytics-Tech-community-metrics, Phabricator: Metrics for key Wikimedia projects software in Maniphest - https://phabricator.wikimedia.org/T28#941559 (Dzahn)
[13:42:46] <wikibugs>	 Engineering-Community, Analytics-Tech-community-metrics, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#941557 (Dzahn) Open>Resolved resolving per: "This task can be closed as soon as we can publish monthly metrics for total amount of users and...
[13:43:25] <wikibugs>	 Engineering-Community, Analytics-Tech-community-metrics, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#941560 (Dzahn) a:Dzahn>None
[14:00:02] <wikibugs>	 Engineering-Community, Analytics-Tech-community-metrics, Phabricator: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#941592 (Aklapper) ♥! Thanks Dzahn!
[14:15:44] <Ironholds>	 we salt the EventLogging IPs, right?
[14:17:32] <Ironholds>	 I mean, I'm hoping not
[14:17:35] <Ironholds>	 but it seems improbable ;p
[14:33:49] <qchris>	 Ironholds: yes, they are salted.
[14:33:50] <qchris>	 https://git.wikimedia.org/blob/mediawiki%2Fextensions%2FEventLogging/9522a747e665ef8d5229ccfb71363f7a1aff9597/server%2Feventlogging%2Fparse.py#L111
[14:34:59] <Ironholds>	 neat!
[14:35:05] <Ironholds>	 ...any chance I could get my hands on the key? :P
[14:58:38] <grrrit-wm>	 (CR) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata)
[15:00:14] <grrrit-wm>	 (CR) OliverKeyes: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023 (owner: Ottomata)
[15:06:49] <grrrit-wm>	 (PS19) Ottomata: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023
[15:07:32] <Ironholds>	 ottomata, is the #19 change just the tolower() moving into Java?
[15:08:14] <ottomata>	 no, just a comment change
[15:08:18] <Ironholds>	 ahh
[15:09:11] <ottomata>	 ah you think we should lower in java?
[15:09:30] <ottomata>	 ok
[15:10:53] <ottomata>	 Ironholds: is there any worry with what toLowerCase mentions here? http://docs.oracle.com/javase/7/docs/api/java/lang/String.html#toLowerCase()
[15:11:20] <ottomata>	 uri hosts are not ever in weird locales, i assume?
[15:12:56] <Ironholds>	 I mean, they shouldn't be?
[15:13:06] <ottomata>	 k
[15:13:10] <ottomata>	 just double checking
[15:13:15] <ottomata>	 hm, what about other fields?
[15:13:18] <ottomata>	 uriPath, for example?
[15:13:21] <Ironholds>	 Other fields, yes :/
[15:13:24] <Ironholds>	 particularly path
[15:13:35] <ottomata>	 content type?
[15:13:45] * Ironholds thinks
[15:13:49] <Ironholds>	 no, I've always seen that be sane
[15:13:56] <Ironholds>	 generally speaking path, query and userAgent are the bugbears
[15:14:12] <ottomata>	 you want to lowercase userAgent?
[15:21:13] <mforns>	 kevinator, milimetric, when do you want to try to install vagrant?
[15:21:34] <mforns>	 milimetric, I suppose you'd like to deploy annotations first
[15:21:59] <kevinator>	 I have it installed, should I delete it and download it again?
[15:22:41] <kevinator>	 mforns: before the standup I did a vagrant git-pull and many “missing files” errors came up
[15:23:45] <mforns>	 kevinator, I don't know exactly, I think we should have a look in the batcave
[15:23:54] <mforns>	 maybe after staff?
[15:24:01] <kevinator>	 ok, after stagg
[15:24:04] <kevinator>	 er staff
[15:24:15] <mforns>	 although, milimetric wants to deploy annotations today...
[15:24:25] <mforns>	 so maybe we should prioritize that?
[15:24:34] <mforns>	 what do you thin
[15:24:37] <mforns>	 think?
[15:25:26] <kevinator>	 yes, deploying annotations first is more better
[15:25:44] <Ironholds>	 ottomata, oh, no, we should leave that alone
[15:25:52] <Ironholds>	 (sorry, got distracted by looking in horror at my emails)
[15:26:46] <ottomata>	 Ironholds:
[15:26:54] <ottomata>	 which fields shoudl be lower cased?
[15:28:08] <Ironholds>	 just host, I think? As your patch had
[15:29:15] <ottomata>	 ah
[15:29:41] <ottomata>	 ah, Ironholds, no path?
[15:29:46] <qchris_meeting>	 Argh. I am trying to join the batcave since 5 mins.
[15:29:46] <ottomata>	 you are giving me mixed messages!
[15:29:52] <Ironholds>	 no path!
[15:30:05] <qchris_meeting>	 It's hanging at "Trying to join the call. Please wait ...." can others join?
[15:30:07] <Ironholds>	 well you kept asking what fields contained weird encoding sso I assumed you wanted them for something!  :D
[15:30:09] <ottomata>	 Ironholds: Other fields, yes
[15:30:09] <ottomata>	 Ironholds: particularly path
[15:30:26] <Ironholds>	 I don't think the others need lower-casing, naw
[15:30:28] <qchris_meeting>	 kevinator: Can you join the hangout?
[15:30:30] <ottomata>	 oh, no i was asking what other fields should be to-lowered
[15:30:30] <ottomata>	 ok
[15:30:31] <Ironholds>	 in fact, some are probably case-sensitive
[15:31:40] <milimetric>	 qchris_meeting: we're all in, I invited you again
[15:31:41] <milimetric>	 does that work?
[15:31:41] <kevinator>	 qchris_meeting: yes, I just joined
[15:37:15] <grrrit-wm>	 (PS20) Ottomata: [WIP] UDF for classifying pageviews according to https://meta.wikimedia.org/wiki/Research:Page_view/Generalised_filters [analytics/refinery/source] - https://gerrit.wikimedia.org/r/180023
[16:20:55] <wikibugs>	 Analytics-Cluster, Ops-Access-Requests: Access to Hadoop Cluster for Ananth Ramakrishnan (new contractor - https://phabricator.wikimedia.org/T85229#941898 (Ottomata) NEW a:Ottomata
[16:21:01] <wikibugs>	 Analytics-Cluster, Ops-Access-Requests: Access to Hadoop Cluster for Ananth Ramakrishnan (new contractor) - https://phabricator.wikimedia.org/T85229#941898 (Ottomata)
[16:33:11] <grrrit-wm>	 (PS1) Mforns: Add annotations to graphs [analytics/dashiki] - https://gerrit.wikimedia.org/r/181591
[16:39:02] <grrrit-wm>	 (CR) Mforns: [C: -1] "Still WIP" (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181591 (owner: Mforns)
[16:49:08] <tnegrin>	 hey nuria 1:1?
[16:56:41] <nuria>	 tnegrin: SORRY
[16:56:46] <tnegrin>	 np
[16:57:00] <nuria>	 tnegrin: getting there, with kids out of school I am loosing track of time
[16:57:12] <tnegrin>	 heh -- I feel your pain
[16:57:24] <nuria>	 tnegrin: on hangout
[17:08:15] <milimetric>	 mforns_lunch: sorry I couldn't get to the dashiki review
[17:08:18] <milimetric>	 let's do it when I get back
[17:08:22] <milimetric>	 i gotta run to a flight
[17:08:32] <milimetric>	 happy holidays! bybybybye
[17:12:37] <kevinator_away>	 milimetric: happy holidays!
[17:52:47] <wikibugs>	 Analytics-Dashiki: Failure to retrieve a metric json file should not break the UI - https://phabricator.wikimedia.org/T85233#942020 (Nuria) NEW
[18:04:04] <grrrit-wm>	 (PS1) Nuria: Fix reference to responsive.css file in index.html [analytics/dashiki] - https://gerrit.wikimedia.org/r/181599
[18:26:57] <grrrit-wm>	 (CR) Nuria: "Most of my comments deal with code loading. I think a small refactor is in order to make sure that annotation component is loaded lazily " (4 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181591 (owner: Mforns)
[18:28:10] <grrrit-wm>	 (CR) Nuria: "Please merge this change and rebase to fix 404s with responsive.css: https://gerrit.wikimedia.org/r/181599" (2 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns)
[18:50:00] <grrrit-wm>	 (CR) Mforns: Add Annotations API (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424 (owner: Mforns)
[18:50:07] <grrrit-wm>	 (PS2) Mforns: Add Annotations API [analytics/dashiki] - https://gerrit.wikimedia.org/r/181424
[19:13:39] <tnegrin>	 warning
[19:13:45] <tnegrin>	 I'm using hive and I have a question
[19:14:46] <tnegrin>	 my computer fan is on a lot -- could this be connected?
[19:23:51] <qchris>	 tnegrin: Typically, those two things would not be connected.
[19:24:13] <qchris>	 I suppose you're running hive through ssh to stat1002?
[19:24:21] <tnegrin>	 hah -- got you! I actually have a question about joins :)
[19:24:46] <tnegrin>	 yes -- I'm on stat1002
[19:25:02] <qchris>	 Then your fan and hive usage should not be connected.
[19:25:04] <tnegrin>	 I'm trying to join two tables and I can't get the syntax right
[19:25:11] <tnegrin>	 I know -- I was just being silly
[19:25:27] <qchris>	 I never tried joins in hive, but let's figure it out together.
[19:25:50] <ottomata>	 hello from da buuuus
[19:25:51] <tnegrin>	 thanks qchris -- I think it's pretty straightforward, it's just been a while
[19:25:55] <tnegrin>	 ohai
[19:25:57] <qchris>	 What parts are you trying to join?
[19:26:06] <qchris>	 ottomata: Hello ottomata on the bus :-D
[19:26:27] <tnegrin>	 ellery has made 2 tables -- ellery.oozie_mc and ellery.en_revisions
[19:26:46] <tnegrin>	 one with page views, one with editors and pages they have edited
[19:27:15] <tnegrin>	 I'd like to create a dataset with the users joined with the stats about the pages they have edited
[19:27:34] <tnegrin>	 here's the instructions: prev (referer title) , curr (current page title, n (count)
[19:27:34] <tnegrin>	 The table ellery.en_revisions contains a sample 2000 users and the pages they have edited. The schema is:
[19:27:35] <tnegrin>	 user_id, page_id, page_title
[19:27:36] <tnegrin>	 The two table can be joined on page_title = curr.
[19:28:55] <tnegrin>	 um -- do we have a local gist?
[19:29:01] <ottomata>	 yup
[19:29:07] <ottomata>	 https://phabricator.wikimedia.org/paste/create/
[19:29:40] <tnegrin>	 thanks
[19:29:48] <tnegrin>	 so here's the query I'm trying https://phabricator.wikimedia.org/P177
[19:30:15] <tnegrin>	 I'm running it via hive -f <script>
[19:30:47] <tnegrin>	 and hive chokes with a parse error
[19:31:28] <qchris>	 I think implicit joins are only allowed from Hive 0.13 on.
[19:31:32] <qchris>	 We're at Hive 0.12.
[19:31:43] <tnegrin>	 heh
[19:31:52] <tnegrin>	 that would explain it
[19:32:05] <qchris>	 But something like
[19:32:21] <qchris>	 use ellery; select * from oozie_mc join en_revisions on oozie_mc.curr=en_revisions.page_title where ...
[19:32:24] <qchris>	 should work.
[19:32:36] <qchris>	 (only the partition conditions are missing)
[19:32:56] <tnegrin>	 thanks! let me try...
[19:33:39] <qchris>	 use ellery; select * from oozie_mc join en_revisions on oozie_mc.curr=en_revisions.page_title where oozie_mc.year=2014 and oozie_mc.month=12 and oozie_mc.day=22 limit 10;
[19:33:44] <qchris>	 ^ that query at least started.
[19:34:01] <ottomata>	 dear qchris, wanna give a looky again?
[19:34:01] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/180023/
[19:34:26] <tnegrin>	 awesome -- thanks qchris
[19:34:32] <tnegrin>	 sorry about the fan joke
[19:34:45] <qchris>	 :-P
[19:35:38] <nuria>	 qchris: so I dug out a bit on the tablesample method of hive and looks like the whole table is scanned reagrdless to create buckets, so as long as you are not post-processing the output much there is no gain in sampling.   let me know if this doesn't sound right, cc Ironholds
[19:35:46] <qchris>	 ottomata: Sure. But let me finish some other stuff before. Do you need it by today?
[19:35:52] <tnegrin>	 works
[19:35:55] <ottomata>	 nope
[19:35:59] <ottomata>	 its just ready
[19:36:04] <qchris>	 tnegrin: \o/
[19:36:10] <ottomata>	 nuria: i think that's not true if the table is created with buckets in mind
[19:36:16] <tnegrin>	 indeed -- thank you for taking pity on the manager
[19:36:21] <Ironholds>	 nuria, sounds about right :(
[19:36:21] <ottomata>	 but for the wmf_raw.webrequest_* tables, they are not
[19:36:23] <ottomata>	 so ja
[19:37:20] <nuria>	 ottomata: but to create bucket tables you have to be very explicit like: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL+BucketedTables
[19:37:21] <ottomata>	 nuria: e.g. http://archive.cloudera.com/cdh4/cdh/4/hive/language_manual/working_with_bucketed_tables.html
[19:37:26] <ottomata>	 yes.
[19:37:28] <qchris>	 nuria: We're not bucketing the partitions on import, so the only correct thing for hive to do is scan through the whole table.
[19:37:39] * qchris is too late again :-)
[19:37:41] <ottomata>	 cough cough, etl cough cough
[19:38:05] <nuria>	 ottomata, qchris : righttttt
[19:38:20] <qchris>	 ottomata: the bus ride is not good for your health. Your coughing up three letter abbreviations!
[19:38:46] <qchris>	 cough cough OMG cough cough
[19:41:15] <ottomata>	 hah
[19:43:54] <ottomata>	 OH, camus is not running in the essential queue1
[19:44:00] <ottomata>	 i think that is why it was backnig up
[19:46:44] <nuria>	 ottomata, one question if you may
[19:49:24] <wikibugs>	 Analytics-Cluster, Ops-Access-Requests: Access to Hadoop Cluster for Ananth Ramakrishnan (new contractor) - https://phabricator.wikimedia.org/T85229#942240 (Tnegrin) approved by manager
[19:50:37] <ottomata>	 yessss?
[19:50:39] <ottomata>	 nuria: ^?
[19:50:59] <nuria>	 ottomata: : where is the code that gets the data out from the "processed" pageview tables and puts that data into files available for download?
[19:52:26] <ottomata>	 ah
[19:52:53] <ottomata>	 nuria: https://github.com/wikimedia/analytics-refinery/tree/master/oozie/webstats/generate_hourly_files and https://github.com/wikimedia/analytics-refinery/tree/master/oozie/util/archive_job_output
[19:53:29] <nuria>	 ottomata: but those put data into hadoop "files"
[19:53:35] <ottomata>	 oh
[19:53:43] <ottomata>	 you are asking how they get to dumps?
[19:53:47] <nuria>	 ottomata: yes
[19:54:25] <ottomata>	 https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/dataset.pp#L20
[19:54:47] <ottomata>	 https://github.com/wikimedia/operations-puppet/blob/production/modules/dataset/manifests/cron/pagecounts_all_sites.pp
[19:55:12] <Deskana>	 Ironholds: I'm going to use the useragent "HEADERTEST"
[19:55:31] <ottomata>	 and one more
[19:55:32] <ottomata>	 https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/analytics.pp#L75
[19:55:33] <Deskana>	 Ironholds: I'm assembling the APK now then I'll send a few example requests
[19:57:16] <Ironholds>	 ta
[19:57:16] <Deskana>	 Ironholds: I sent an email with the pages I looked at.
[19:57:27] <Deskana>	 Thanks!
[19:58:10] <nuria>	 ottomata: so the $source is passed to those crons as being the hadoop path where the archives are? but ....
[19:58:55] <ottomata>	 nuria
[19:59:03] <ottomata>	 stat1002 has /mnt/hdfs
[19:59:08] <ottomata>	 al Fuse mount
[19:59:24] <ottomata>	 there is an rsync daemon module defined to allow read access from a few specific servers on that mount
[19:59:41] <ottomata>	 there is a cron job on dataset1001 that rsyncs from stat1002 -> dataset's public http dir
[20:00:03] <ottomata>	 this is indirect, as it means that stat1002 must get the data from HDFS and then send it do dataset
[20:00:18] <ottomata>	 we could alternatively mount hdfs on dataset1001, but that would take more security review
[20:00:24] <ottomata>	 and, the data is not that large (at least not yet)
[20:02:42] <nuria>	 ottomata: k, understood
[20:05:42] <grrrit-wm>	 (PS1) Ottomata: Camus should run in the 'essential' queue [analytics/refinery] - https://gerrit.wikimedia.org/r/181624
[20:06:48] <Ironholds>	 Deskana, cool! Okay, will check in on at about 3pm PST
[20:06:48] <grrrit-wm>	 (CR) QChris: [C: 2 V: 2] Camus should run in the 'essential' queue [analytics/refinery] - https://gerrit.wikimedia.org/r/181624 (owner: Ottomata)
[20:07:15] <ottomata>	 so fas!
[20:07:27] <ottomata>	 qchris: mind if I deploy that?
[20:07:42] <qchris>	 I guess just the git-deploy?
[20:07:47] <qchris>	 I don't mind at all :-)
[20:07:48] <ottomata>	 yup
[20:08:35] <ottomata>	 cool, next camus job should be in essential queue
[20:08:54] <qchris>	 \o/
[20:08:59] <ottomata>	 will check it
[20:09:01] <ottomata>	 when it runs
[20:16:34] <ottomata>	 oo, yuck, qchris: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&title=&vl=&x=&n=&hreg[]=analytics10%2812%7C18%7C21%7C22%29&mreg[]=diskstat_.%2A_read_bytes_per_sec&gtype=stack&glegend=show&aggregate=1
[20:16:35] <ottomata>	 that is camus lagging
[20:16:52] <ottomata>	 i think because luster was busy and it was in the wrong queue
[20:16:52] <ottomata>	 cluster*
[20:16:58] <ottomata>	 it was taking longer than 10 minutes to run a job
[20:17:21] <ottomata>	 which causes the brokers to have to hit disk more
[20:17:31] <ottomata>	 well, the fact that it had to read more older data out out of brokers at a time
[20:18:32] <qchris>	 Yup.
[20:18:41] <qchris>	 But is that the only cause?
[20:18:50] <qchris>	 The highest spike is after midnight.
[20:18:53] <ottomata>	 not sure, but that is when ellery noticed partitions missing
[20:19:01] <ottomata>	 that's ok, jobs are still running then
[20:19:04] <ottomata>	 everyone is running long jobs
[20:19:17] <ottomata>	 not sure though.
[20:19:18] <qchris>	 But that's the time with lowest traffic.
[20:19:41] <ottomata>	 ja, but in this case it would be the cluster load that is causing problems, not traffic load
[20:19:50] <ottomata>	 i mean, traffic load does affect it
[20:19:58] <ottomata>	 but, if camus is slowed (or stopped)
[20:20:05] <qchris>	 Yes.
[20:20:08] <ottomata>	 the next time it runs it will have to read much more data
[20:20:17] <ottomata>	 and brokers will have to hit disk more
[20:20:28] <ottomata>	 the current camus job has been running for 20 minutes already
[20:22:16] <ori>	 ottomata: do you mind if i merge <https://gerrit.wikimedia.org/r/#/c/181110/> ? i can babysit
[20:22:39] <ottomata>	 ah, ori, i was asking a q the other day when you had a meeting
[20:22:58] <ottomata>	 so, ja
[20:23:07] <ottomata>	 why not just use statsd/graphite?
[20:23:08] <ori>	 oh, why not graphite. the ability to associate additional datapoints with counts
[20:23:29] <ottomata>	 ?
[20:24:05] <ori>	 well, for one, i'd like a count for each referrer, so two problems with that:
[20:24:41] <ori>	 a) there are a lot of referrers, and whisper files are preallocated to their full size on creation, so the storage for that would get very expensive
[20:26:03] <ori>	 b) the key is the full URL, which contains characters that are (afaik) not valid for graphite keys, so you need to encode them somehow
[20:26:47] <ottomata>	 aye, hm, k
[20:26:54] <ottomata>	 does package 'abacist' just work?
[20:27:00] <ottomata>	 do you need analytics/abacist?
[20:27:06] <ori>	 it works
[20:27:16] <ottomata>	 how does it know?!
[20:27:41] <ori>	 well, the mapping from 'abacist/abacist' to the repo url is defined in <https://gerrit.wikimedia.org/r/#/c/181110/1/manifests/role/deployment.pp>
[20:27:49] <ottomata>	 oh
[20:27:51] <ottomata>	 duHHH
[20:27:52] <ottomata>	 forgot about that
[20:27:56] <ori>	 and the trebuchet package provider knows to convert unqualified names in the format 'foo' to 'foo/foo'
[20:28:32] <ori>	 though your intuition is valid in that i think the way it *ought* to work is you just have package { 'analytics/abacist': } without the need to declare a mapping to a gerrit repo explicitly
[20:29:50] <ottomata>	 why would you want to clone it at abacist/abacist instead of analytics/abacist?
[20:31:39] <ottomata>	 also, ori, there's nuthin in your abacist repo, so maybe commit something there before deploying and running puppet?
[20:31:46] <ottomata>	 but, whatevs, +1ed, looks good :)
[20:32:54] <ori>	 ah yeah, i need to merge the initial commit too
[20:32:55] <ori>	 thanks!
[20:33:53] <ottomata>	 yup :)
[20:35:01] <ottomata>	 cool, fairscheduler is working
[20:35:25] <ottomata>	 nuria and qchris, i can see you both have oozie jobs that are now just in 'accepted' state
[20:35:27] <ottomata>	 pending to be run
[20:35:52] <ottomata>	 ohp, now they are running
[20:36:03] <qchris>	 :-)
[20:36:09] <nuria>	 ottomata: mines can be killed
[20:36:15] <ottomata>	 naw its cool
[20:36:24] <ottomata>	 i want the cluster to have to deal with this
[20:36:28] <ottomata>	 it should be able to
[20:36:35] <ottomata>	 and we need to learn how to best configure it eventually
[20:36:42] <ottomata>	 the job queues are not something i have though much about
[20:36:49] <ottomata>	 if we have to, we can do per-user queues i think
[20:36:56] <ottomata>	 but, now that camus is higher priority, i think it is ok
[20:39:08] <ottomata>	 ha, halfak is using 51 containers!
[20:39:19] <ottomata>	 417G of ram! :)
[20:43:15] <nuria>	 ottomata: in hadoop? hive? elsewhere?
[20:43:26] <ottomata>	 streaming, i thikn
[20:46:09] <halfak>	 ottomata, my streaming processes don't need that much RAM.  Do you know why so much would be allocated?
[20:46:15] <halfak>	 It could just be buffers.
[20:53:51] <nuria>	 lunchinggg... will be back
[21:01:19] <halfak>	 ottomata, just updated row for json-snappy here: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/xmldumps#Results
[21:01:55] <halfak>	 It looks like we get an order of magnitude better compression with bz2.
[21:02:11] <halfak>	 That might not matter for all but the largest datasets.
[21:02:16] <ottomata>	 yes, that makes sense
[21:02:36] <ottomata>	 for the webrequest json, snappy only compressed to about 1/4 of the size
[21:03:01] <ottomata>	 oh whoa, this is worse though, only about 1/2
[21:05:03] <halfak>	 A job I just ran output 11TB when I expected to write about 1-10GB.  I'm not sure if bz2 just fares way better with that dataset or what.
[21:05:17] <halfak>	 I guess I could just recompress it in hadoop.
[21:07:40] <ottomata>	 hm, that doesn't sound right.  halfak that is several orders of magnitude, no?
[21:08:22] <halfak>	 yes.  three orders.  I was honestly surprised when I first saw bz2's ability to compress this data.  It is denormalized so it repeats a lot.
[21:08:40] <halfak>	 I figured bz2 was doing it's job well, but now I'll check to make sure I wasn't crazy.
[21:12:08] <wikibugs>	 Analytics-Engineering, Phabricator: Migration of Analytics to Phabricator - https://phabricator.wikimedia.org/T826#942352 (Qgil)
[21:55:31] <grrrit-wm>	 (PS1) Gergő Tisza: Fix URLs from stat1001.wikimedia.org to datasets.wikimedia.org [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/181685
[21:56:21] <grrrit-wm>	 (CR) Gergő Tisza: [C: 2 V: 2] "Self-merge, trivial change which should fix the dashboards which are currently broken." [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/181685 (owner: Gergő Tisza)
[21:58:12] <wikibugs>	 Multimedia, Analytics: Update Multimedia dashboards to use datasets.wikimedia.org instead of stat1001.wikimedia.org - https://phabricator.wikimedia.org/T85250#942435 (Tgr)
[22:13:43] <wikibugs>	 Analytics, Multimedia: Update Multimedia dashboards to use datasets.wikimedia.org instead of stat1001.wikimedia.org - https://phabricator.wikimedia.org/T85250#942490 (Tgr) limn-deploy does not work because git.wikimedia.org is also broken (T85245). Fun!
[22:22:50] <wikibugs>	 Analytics, Multimedia: Update Multimedia dashboards to use datasets.wikimedia.org instead of stat1001.wikimedia.org - https://phabricator.wikimedia.org/T85250#942503 (Tgr) Worked around by manually checking out the commit from gerrit (which still works). I think limn-deploy will be able to revert that once it...
[22:39:49] <wikibugs>	 Analytics, Multimedia: Update Multimedia dashboards to use datasets.wikimedia.org instead of stat1001.wikimedia.org - https://phabricator.wikimedia.org/T85250#942528 (QChris) >>! In T85250#942490, @Tgr wrote: > limn-deploy does not work because git.wikimedia.org is also broken (T85245). Fun!  git.wikimedia.or...
[22:44:40] <wikibugs>	 Analytics, Multimedia: Update Multimedia dashboards to use datasets.wikimedia.org instead of stat1001.wikimedia.org - https://phabricator.wikimedia.org/T85250#942533 (Tgr) Indeed, limn-deploy runs fine now (and did revert to master without problems).
[23:12:23] <wikibugs>	 Analytics, Multimedia: Update Multimedia dashboards to use datasets.wikimedia.org instead of stat1001.wikimedia.org - https://phabricator.wikimedia.org/T85250#942574 (Tgr) p:Triage>High