[01:17:42] <HaeB>	 ori: out of idle historical curiosity, what does "millisec" mean in this context when the actual timestamp values only have a resolution of entire seconds? https://meta.wikimedia.org/w/index.php?title=Schema:EventCapsule&diff=5222982&oldid=5152617
[02:45:43] <nuria>	 Hive seems to be kaput throwing tons of errors like:
[02:45:47] <nuria>	 https://www.irccloud.com/pastebin/BM0fGRTJ/
[02:50:40] <nuria>	 ah no, maybe my tablescan was too large?
[04:26:18] <madhuvishy>	 nuria: it looks like you were querying a partition that was just being written into
[04:26:35] <madhuvishy>	 i see this happen in that case sometimes
[07:18:44] <wikibugs>	 Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2143580 (Danielsberger)
[07:27:20] <wikibugs>	 Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2143590 (Danielsberger)
[07:44:36] <grrrit-wm>	 (CR) Mforns: [C: 2 V: 2] Adding piwik reporting to browser-reports [analytics/dashiki] - https://gerrit.wikimedia.org/r/278960 (https://phabricator.wikimedia.org/T130653) (owner: Nuria)
[09:01:29] <wikibugs>	 Analytics: Upgrade aqs* to nodejs 4.3 - https://phabricator.wikimedia.org/T123629#2143751 (MoritzMuehlenhoff) Upgrade procedure: - Depool one of the aqs servers via conftool - Stop restbase  - nodetool drain && systemctl stop cassandra - upgrade nodejs - start Cassandra and check whether it fully rejoined -...
[09:04:34] <wikibugs>	 Analytics, Operations: Upgrade aqs* to nodejs 4.3 - https://phabricator.wikimedia.org/T123629#2143752 (MoritzMuehlenhoff)
[09:35:36] <joal>	 dcausse: around ?
[09:35:46] <dcausse>	 joal: yes
[09:35:50] <joal>	 Hi :)
[09:35:52] <dcausse>	 hi! :)
[09:36:07] <joal>	 dcausse: do you want us to merge / deploy this change https://gerrit.wikimedia.org/r/#/c/255105/ ?
[09:36:17] <joal>	 In other words: has in been tested and all ?
[09:36:49] <dcausse>	 joal: I think so, this patch has been here for a long a time :/
[09:36:55] <joal>	 It has indeed
[09:37:18] <joal>	 I'm deployong today, so reviewing patches that have been left around in refinery-source
[09:37:38] <dcausse>	 joal: do you deploy the new ApiAction schema?
[09:37:38] <joal>	 Ok, so I merge and deploy that today - Is there anything to restart with that ?
[09:37:48] <joal>	 dcausse: yes
[09:38:08] <dcausse>	 joal: ok, everything has been updated to comply with this patch I think
[09:38:34] <joal>	 dcausse: nothing happens with ApiAction for the moment IIRC
[09:39:02] <dcausse>	 ok
[09:39:05] <joal>	 I'll deploy refinery after source, so at that moment, if we start the new oozie import job, then yes,
[09:39:33] <joal>	 also dcausse, there have been a big refactor from bd808 on oozie - I don't know if you've looked at it
[09:39:52] <dcausse>	 no I only looked at the avro schema
[09:39:59] <joal>	 It would be interesting for you and your team to read it :)
[09:40:09] <dcausse>	 ok :)
[09:40:15] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Add automatic failover to the Hadoop's name node - https://phabricator.wikimedia.org/T129838#2143788 (elukey) The last code review should add the necessary code, but I'd prefer to get the puppet-compiler working with submodules first (https://phab...
[09:40:23] <joal>	 basically: generalisation of the mediawiki import jobs to include not CirrusSeasrch but other channels
[09:40:30] <joal>	 dcausse: --^
[09:40:44] <dcausse>	 joal: nice!
[09:40:49] <joal>	 dcausse: Merging youre patch :)
[09:41:10] <grrrit-wm>	 (CR) Joal: [C: 2] "Merging for deploy." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/255105 (owner: DCausse)
[09:41:10] <dcausse>	 joal: when will you mvn release?
[09:41:17] <joal>	 dcausse: now ;0
[09:41:29] <dcausse>	 ok, +2 a schema change then
[09:41:35] <joal>	 sure dcausse
[09:43:45] <grrrit-wm>	 (Abandoned) Joal: Revert "Update uap-core with latest code from upstream" [analytics/ua-parser] - https://gerrit.wikimedia.org/r/277306 (owner: Joal)
[09:44:40] <dcausse>	 joal: merged, let me know when the artefact is deployed to archiva, I'll check that the new schema has been included
[09:45:13] <joal>	 dcausse: submodule doesn't say it needs an update: weird
[09:45:36] <joal>	 Have you updated refinery-source with new submodule sha?
[09:45:40] <dcausse>	 https://gerrit.wikimedia.org/r/#/c/274312
[09:46:49] <dcausse>	 joal I don't know this part, I thought it would just update to the latest automatically
[09:49:31] <joal>	 dcausse: trying to release, we'll check
[09:50:28] <grrrit-wm>	 (PS1) Joal: Update changelog.md for v0.0.28 release [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279093
[09:50:34] <joal>	 elukey: around ?
[09:53:42] <elukey>	 joal: o/
[09:54:00] <joal>	 Do you mind having a look at --^ and merging ?
[09:55:39] <grrrit-wm>	 (CR) Elukey: [C: 2] Update changelog.md for v0.0.28 release [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279093 (owner: Joal)
[09:56:43] <elukey>	 joal: is it --^
[09:56:54] <joal>	 elukey: Thanks mate :)
[09:56:59] <grrrit-wm>	 (PS1) DCausse: Update schema_repo submodule to latest [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279094
[09:57:27] <joal>	 right dcausse, I assumed th
[09:57:29] <dcausse>	 joal: you were right... I should have updated the submodule ref
[09:57:33] <joal>	 :)
[09:57:37] <joal>	 ok, merging that
[09:57:51] <grrrit-wm>	 (CR) Joal: [C: 2] "Merging before deploy." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279094 (owner: DCausse)
[09:58:20] <grrrit-wm>	 (CR) Joal: [V: 2] "Actually merging." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279094 (owner: DCausse)
[10:00:39] <grrrit-wm>	 (PS1) Joal: Update changelog.md before v0.0.28 release. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279096
[10:01:23] <grrrit-wm>	 (CR) Joal: [C: 2] "Self merging for deploy." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279096 (owner: Joal)
[10:08:39] <grrrit-wm>	 (CR) Joal: [V: 2] Update changelog.md before v0.0.28 release. [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279096 (owner: Joal)
[10:11:01] <joal>	 !log Release refinery-source v0.0.28 to archiva
[10:26:14] <dcausse>	 joal: refinery-camus-0.0.28 looks good, thanks!
[10:54:49] <joal>	 great dcausse, thanks !
[11:05:58] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Merging for release." [analytics/refinery] - https://gerrit.wikimedia.org/r/278325 (https://phabricator.wikimedia.org/T130187) (owner: Joal)
[11:11:47] <grrrit-wm>	 (PS1) Joal: Add v0.0.28 jar artifacts [analytics/refinery] - https://gerrit.wikimedia.org/r/279105
[11:12:17] <joal>	 elukey: if still around, couple of reviews to come :)
[11:12:32] <elukey>	 suuureeee
[11:12:37] <joal>	 Thanks man :)
[11:13:06] <elukey>	 joal: I am working with Giuseppe to fix the puppet-compiler to work with submodules change, after that I'll test the HA change on analytics1001
[11:13:18] <elukey>	 this is why I didn't ask you to stop HDFS this morning
[11:13:18] <joal>	 Sounds great !
[11:13:31] <joal>	 np elukey, I am in no rush :)
[11:18:20] <grrrit-wm>	 (PS1) Joal: Bump jar version and record version in refine job [analytics/refinery] - https://gerrit.wikimedia.org/r/279107
[11:21:42] <elukey>	 joal: I have no idea (shame on me) about https://gerrit.wikimedia.org/r/#/c/279105/1, but if you checked all the hash and you are sure I'll merge
[11:22:19] <elukey>	 don't also know about what the "record version" mean in https://gerrit.wikimedia.org/r/279107, but if it is ok the rest looks fine :)
[11:22:23] <joal>	 elukey: I've add no issues so far with that process, but I didn't double check hashes
[11:23:28] <wikibugs>	 Analytics: Fix phab script to gather stats also from point field , not only from title - https://phabricator.wikimedia.org/T130543#2143962 (mforns) a:mforns
[11:23:41] <wikibugs>	 Analytics-Kanban: Fix phab script to gather stats also from point field , not only from title - https://phabricator.wikimedia.org/T130543#2138954 (mforns)
[11:28:19] <elukey>	 joal: would it be something to do or we can proceed anyway?
[11:29:35] <joal>	 elukey: I think we're fine :)
[11:29:52] <elukey>	 okok!
[11:29:57] <elukey>	 Merging
[11:30:28] <grrrit-wm>	 (CR) Elukey: [C: 2] "Discussed with Joseph on IRC!" [analytics/refinery] - https://gerrit.wikimedia.org/r/279107 (owner: Joal)
[11:30:41] <grrrit-wm>	 (CR) Elukey: [V: 2] "Discussed with Joseph on IRC!" [analytics/refinery] - https://gerrit.wikimedia.org/r/279107 (owner: Joal)
[11:31:25] <grrrit-wm>	 (CR) Elukey: [C: 2 V: 2] "Discussed with Joseph on IRC" [analytics/refinery] - https://gerrit.wikimedia.org/r/279105 (owner: Joal)
[11:32:01] <elukey>	 ---^ joal
[11:32:35] <joal>	 great, a last one if you don't mind https://gerrit.wikimedia.org/r/279107
[11:36:14] <joal>	 elukey: --^
[11:36:37] <elukey>	 already merged joal!
[11:36:57] <joal>	 Oh sorry elukey !
[11:37:00] <joal>	 Thanks you :)
[11:37:39] <elukey>	 joal: going afk for ~1hr, will brb!
[11:37:45] <joal>	 Sure elukey, np
[11:46:51] <joal>	 !log Deploy refinery
[11:50:34] <joal>	 !log stop/restart load and refine jobs
[12:27:55] <joal>	 !log fix bug in load job manually
[12:30:42] <mforns_afk>	 team, will be back before standup :]
[12:30:50] <joal>	 Bye mforns_afk
[12:34:18] <wikibugs>	 Analytics-Engineering, Analytics-Wikimetrics: Removing usernames from a cohort doesn't work - https://phabricator.wikimedia.org/T130714#2144078 (Halibutt)
[12:36:59] <wikibugs>	 Analytics-Engineering, Analytics-Wikimetrics: Can't correct names in a cohort - https://phabricator.wikimedia.org/T130715#2144091 (Halibutt)
[12:40:55] <grrrit-wm>	 (PS1) Joal: Correct bug in error email text in load job [analytics/refinery] - https://gerrit.wikimedia.org/r/279111
[12:41:23] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self-merging, already corrected manually in prod." [analytics/refinery] - https://gerrit.wikimedia.org/r/279111 (owner: Joal)
[12:51:43] <elukey>	 back!
[13:37:59] <joal>	 a-team, I'm AFK, will be back for standup
[13:38:10] <ottomata>	 hi ok!
[13:38:42] <elukey>	 ottomata: hellooooo
[13:38:50] <ottomata>	 hiiya!
[13:39:13] <elukey>	 do you have time for a puppet-compiler discussion?
[13:39:38] <elukey>	 I am trying to add submodules support to it, but my brain is kinda twisted
[13:44:32] <ottomata>	 ohhh
[13:44:33] <ottomata>	 sure yeah
[13:44:34] <ottomata>	 sounds fun
[13:44:42] <ottomata>	 i don't know puppet compiler well
[13:44:46] <ottomata>	 but i do know puppet-merge well
[13:44:53] <ottomata>	 and i bet the issue might be silmiar
[13:45:00] <ottomata>	 elukey: ^
[13:45:01] <elukey>	 bat-cave?
[13:45:05] <ottomata>	 ok
[13:45:08] <ottomata>	 one sec lemme get headphones
[14:09:16] <elukey>	 ottomata: mmmm git submodules update does get the new commits and ends up in a detached state.. I tried rebase and it does the same
[14:09:32] <ottomata>	 the submodules detached?
[14:09:33] <ottomata>	 ja, makes sense
[14:09:40] <ottomata>	 it just checks out annamed sha
[14:09:43] <ottomata>	 unnamed
[14:09:46] <ottomata>	 which == detached, no?
[14:10:05] <ottomata>	 iyeah
[14:10:08] <ottomata>	 on my local ops puppet
[14:10:11] <ottomata>	 cd modules/nginx
[14:10:12] <ottomata>	 git status
[14:10:16] <ottomata>	 HEAD detached at e74a5c7
[14:10:18] <ottomata>	 that's normal
[14:11:15] <elukey>	 all right but didn't you end up having the code on the host (testing it too)?
[14:11:31] <elukey>	 maybe I am missing something trivial
[14:12:05] <mobrovac>	 detached state is the default for git submodules
[14:12:32] <ottomata>	 elukey:  detached just means that the checked out commit doesn't have a name
[14:12:33] <mobrovac>	 i don't even think it's possible to make a submodule track a specific branch
[14:12:34] <ottomata>	 like a branch or a tag
[14:13:24] <elukey>	 ottomata: sure sure but all the new commits gets fetched and merged theoretically, and you end up testing all of them rather than the one that you want no?
[14:15:00] <mobrovac>	 elukey: you can do whatever you want to the submodule, e.g. cd submodule && git cherry-pick your_sha1_here or whatever
[14:15:10] <mobrovac>	 then just git checkout -- submodule_dir
[14:15:14] <mobrovac>	 to get it back
[14:15:29] <ottomata>	 elukey: ?
[14:15:41] <ottomata>	 the new commits will only get merged for submodules that actually had sha changes
[14:18:41] <elukey>	 ottomata: ok I checked git submodule status after the git submodule update and the sha is sound
[14:19:22] <elukey>	 I am probably confused because if I go in 'cd modules/cdh' I can see the new detached state
[14:19:43] <ottomata>	 yeah detached is 100% normal
[14:19:50] <elukey>	 mobrovac: sorry you are right, I seem a bit crazy but I am trying to add submodules support to puppet-compiler
[14:20:16] <mobrovac>	 elukey: you are? then i'm afraid you are indeed crazy :D
[14:20:36] <mobrovac>	 good luck!
[14:20:38] <elukey>	 mobrovac: thank you! I like supporting people
[14:20:40] <elukey>	 :P
[14:20:43] <mobrovac>	 hahaha
[14:21:00] <mobrovac>	 fwiw it shouldn't be too hard
[14:21:19] <mobrovac>	 git submodule update --init is a no-op if no changes were made to them
[14:23:19] <wikibugs>	 Analytics-Kanban, Wikipedia-Android-App-Backlog: Count requests to RESTBase from the Android app - https://phabricator.wikimedia.org/T128612#2144362 (Nuria) > should be sending requests with "x-analytics: preview=1".  Can android devs confirm (via wireshark or similar) that app is sending the header?
[14:24:06] <elukey>	 mobrovac: yeah I was talking with andrew about the use case of having multiple modules changed (so alredy merged in their repo) but only testing a code review to merge in the main puppet repo the new sha (like https://gerrit.wikimedia.org/r/#/c/279091/)
[14:24:28] <ottomata>	 OHH, elukey ah, that's where you are confused, i see
[14:24:34] <ottomata>	 a merge in a submodule's repo does nothing to ops/puppet
[14:24:48] <elukey>	 no no yes I know tha!
[14:24:51] <elukey>	 *that!
[14:24:51] <ottomata>	 oh ok
[14:25:12] <ottomata>	 the only thing that will change ops puppet is if the submodule's sha changes
[14:25:15] <ottomata>	 in ops/puppet
[14:25:21] <ottomata>	 whihc takes a commit to ops/puppet
[14:25:23] <ottomata>	 which is what you want to test
[14:34:14] <milimetric>	 ottomata: you wanna deploy this thing?
[14:34:19] <milimetric>	 (the dumps change)
[14:36:38] <ottomata>	 oh ok!
[14:36:39] <ottomata>	 yeah
[14:37:13] <ottomata>	 milimetric:  i lost link, you should add me as reviewer
[14:37:54] <milimetric>	 done: https://gerrit.wikimedia.org/r/#/c/269696/
[14:38:25] <milimetric>	 a-team: all ok with my draft on the etherpad?
[14:38:28] <milimetric>	 https://etherpad.wikimedia.org/p/analytics-email-drafts
[14:39:43] <ottomata>	 ok milimetric should I merge?
[14:40:05] <milimetric>	 ottomata: sure, and then we'll deploy and revert if needed
[14:40:16] <elukey>	 ottomata: https://gerrit.wikimedia.org/r/#/c/279117/3/puppet_compiler/prepare.py <-- this is basically what we discussed
[14:40:20] <milimetric>	 best way to test evah :/
[14:41:09] <ottomata>	 looks good elukey!
[14:44:07] <ottomata>	 milimetric:  merged and run
[14:44:13] <milimetric>	 checking
[14:44:21] <ottomata>	 http://dumps.wikimedia.org/analytics
[14:44:35] <milimetric>	 ooops
[14:44:37] <milimetric>	 :(
[14:44:43] <milimetric>	 why 404?  I didn't set it up right?
[14:44:51] <ottomata>	 where is it supposed to go?
[14:44:54] <milimetric>	 there
[14:45:13] <milimetric>	 https://gerrit.wikimedia.org/r/#/c/269696/3/modules/dataset/manifests/html.pp
[14:45:26] <ottomata>	 OH sorry
[14:45:29] <ottomata>	 i didn't read puppet output
[14:45:30] <ottomata>	 ja
[14:45:33] <ottomata>	 analytics dir doesn't exist
[14:45:40] <ottomata>	 you gotta create it if you want to put something in it
[14:45:43] <milimetric>	 aha
[14:45:47] <milimetric>	 oops, sorry
[14:47:37] <milimetric>	 ottomata: are you making the dir or should I submit a patch?
[14:48:55] <ottomata>	 milimetric:  submit a patch
[14:48:59] <ottomata>	 i can make the dir real quick to fix
[14:49:10] <ottomata>	 but puppet should make the dir
[14:49:18] <ottomata>	 milimetric:  sorry, am also chatting in other rooms
[14:49:54] <ottomata>	 oh milimetric
[14:49:58] <ottomata>	 maybe it goes in dataset::dirs?
[14:50:04] <ottomata>	 (this module is nuts!)
[14:50:44] <ottomata>	 milimetric: ^ oh man this module makes no sense
[14:51:01] <milimetric>	 uh :)
[14:51:03] <ottomata>	 why the heck would you always create these dirs, and then conditionally create index files?!
[14:51:15] <ottomata>	 anyway, i guess just be consistent with the module, add the dir in dataset::dirs class
[14:51:39] <milimetric>	 k
[14:57:39] <milimetric>	 ok, thanks ottomata, all looks good now, sending the email
[14:58:43] <ottomata>	 cool!
[14:59:43] <moritzm>	 joal, milimetric: is aqs stateless, can I simply restart the service via "systemctl restart aqs.service"? (need to restart it for an openssl update)
[14:59:59] <milimetric>	 moritzm: yes, but would you please restart one node at a time?
[15:00:12] <milimetric>	 that way we can still serve requests while it's coming up
[15:00:33] <moritzm>	 sure, will then start in 10 mins or so
[15:00:37] <milimetric>	 thx!
[15:01:05] <milimetric>	 and thanks for keeping us up to date, moritzm! much appreciated
[15:13:43] <wikibugs>	 Analytics-Cluster, Discovery, Maps, RESTBase-Cassandra, Patch-For-Review: Create separate Kibana dashboards for production Cassandra clusters - https://phabricator.wikimedia.org/T130393#2144505 (Eevans) Open>Resolved Separate dashboards for Maps and AQS have been created using the new `c...
[15:13:59] <wikibugs>	 Analytics-Cluster, Discovery, Maps, RESTBase-Cassandra, Patch-For-Review: Create separate Kibana dashboards for production Cassandra clusters - https://phabricator.wikimedia.org/T130393#2144507 (Eevans) p:Triage>Normal
[15:21:02] <moritzm>	 aqs has been restarted on all aqs* hosts
[15:22:29] <joal>	 Thanks moritzm :)
[15:31:10] <wikibugs>	 Analytics-Kanban, EventBus, Patch-For-Review: Add --topic-prefix config to eventlogging-service - https://phabricator.wikimedia.org/T130562#2139475 (Ottomata)
[15:53:33] <wikibugs>	 Analytics, Pageviews-API, Services, RESTBase-API: Document that wikimedia pageviews API is blocked by ad blockers - https://phabricator.wikimedia.org/T126947#2144588 (Milimetric) Does anyone have any contact with the block list maintainers?  I can poke randomly around the internet otherwise :)
[15:58:10] <wikibugs>	 Analytics-Kanban: Back-fill pageviews data for dumps.wikimedia.org to May 2015 - https://phabricator.wikimedia.org/T126464#2144609 (Milimetric) @elukey: I forgot to mention, the process is that @Nuria is the only one who closes tasks as resolved.  That way she can "accept" that they're done.
[16:01:05] <grrrit-wm>	 (PS1) Ottomata: Remove single use of x_forwarded_for in insert_hourly_pagecounts.hq [analytics/refinery] - https://gerrit.wikimedia.org/r/279132 (https://phabricator.wikimedia.org/T118557)
[16:01:16] <ottomata>	 joal: ^
[16:02:19] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "LGTM !" [analytics/refinery] - https://gerrit.wikimedia.org/r/279132 (https://phabricator.wikimedia.org/T118557) (owner: Ottomata)
[16:04:30] <ottomata>	 joal:  fyi
[16:04:33] <ottomata>	 looks fine
[16:04:36] <ottomata>	 hive (otto)> select uri_path, x_forwarded_for from otto.webrequest_null_xff;
[16:04:36] <ottomata>	 OK
[16:04:36] <ottomata>	 uri_path	x_forwarded_for
[16:04:36] <ottomata>	 /w/load.php	NULL
[16:05:03] <joal>	 awesome ottomata
[16:05:11] <joal>	 I'd have expected that, good to confirm
[16:06:09] <joal>	 ottomata: shall we go and merge bd808 change?
[16:06:20] <ottomata>	 its merged, no?
[16:06:27] <ottomata>	 just not deployed?
[16:06:54] <joal>	 the x_forwarded_for one ?
[16:18:20] <wikibugs>	 Analytics: Evaluate deprecation of udp-like files produced on oozie  (legacy tsvs) - https://phabricator.wikimedia.org/T130725#2144649 (Nuria)
[16:35:59] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Add piwik reporting to browser reports [1] - https://phabricator.wikimedia.org/T130653#2144710 (Nuria)
[16:50:47] <wikibugs>	 Analytics: Check if we can deprecate legacy TSVs production (same time as pagecounts?) - https://phabricator.wikimedia.org/T130729#2144772 (JAllemandou)
[16:52:40] <wikibugs>	 Analytics: Make webrequest load and refine jobs a single bundle - https://phabricator.wikimedia.org/T130731#2144802 (JAllemandou)
[16:53:59] <wikibugs>	 Analytics: Standardise naming in oozie jobs (particularly for top level ones),  and/or force top-level jobs  to be bundles - https://phabricator.wikimedia.org/T130732#2144819 (JAllemandou)
[16:55:09] <wikibugs>	 Analytics: Check if we acn merge maps partition into misc partition at varnishkafka level - https://phabricator.wikimedia.org/T130733#2144836 (JAllemandou)
[16:56:46] <wikibugs>	 Analytics: Document a proposal for bundling other than load-refine jobs together (see refine/diagram) - https://phabricator.wikimedia.org/T130734#2144849 (JAllemandou)
[16:57:07] <joal>	 a-team: five new tasks created, please review :)
[16:57:29] <elukey>	 :)
[16:58:29] <joal>	 ottomata: about bd808 change, I think this one https://gerrit.wikimedia.org/r/#/c/253474/ has not been merged
[16:58:33] <joal>	 Maybe I'm wrong
[17:05:57] <mforns>	 nuria, the phab script now seems to work, but as we are marking the tasks in Done as resolved in a more batched manner, the script returns distorted results
[17:07:14] <mforns>	 nuria, we should either: count the points of the tasks that have been moved to Done, instead of marked as resolved; or we mark the tasks in Done as resolved as soon as possible
[17:07:30] <nuria>	 mforns: on meeting can talk later
[17:07:35] <mforns>	 sure, sorry
[17:11:17] <bd808>	 I have data! `kafkacat -b kafka1012 -t mediawiki_ApiAction -c 1` is pulling records down on stat1002
[17:11:57] <bd808>	 joal: I don't think that patch from bblack is directly related to my metrics.
[17:16:38] <wikibugs>	 Analytics, MediaWiki-API, Reading-Infrastructure-Team, MW-1.27-release-notes, and 4 others: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#2144937 (bd808) >>! In T108618#2132875, @bd808 wrote: > New checklist: > [x] [[https://gerrit.wikimedia.o...
[17:17:11] <wikibugs>	 Analytics, MediaWiki-API, Reading-Infrastructure-Team, MW-1.27-release-notes, and 4 others: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#2144939 (bd808)
[17:19:31] <wikibugs>	 Analytics, MediaWiki-API, Reading-Infrastructure-Team, MW-1.27-release-notes, and 4 others: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#2144970 (bd808) Verified that data is reaching Kafka from MediaWiki via `kafkacat -b kafka1012 -t mediawi...
[17:24:23] <wikibugs>	 Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2145023 (dr0ptp4kt) @bearND do you have the code samples handy for header enrichment for previews?
[17:32:40] <ottomata>	 lunchin!  back in a bit
[17:32:55] <ottomata>	 ah joal
[17:33:01] <ottomata>	 that is b bblack, not bd808 that's why i was confused
[17:33:05] <ottomata>	 wasn't sure what you were talking about
[17:33:12] <ottomata>	 yes, that has not been merged yet
[17:37:01] <ottomata>	 (ah still here for a bit...)
[17:38:03] <ottomata>	 joal: i think xff vk patch can be merged even if we haven't restarted the legacy tsv job
[17:38:09] <ottomata>	 or xff = '-' will just do nothing
[17:38:12] <ottomata>	 since it will always be null
[17:38:29] <ottomata>	 bd808:  awesome!
[17:39:05] <ottomata>	 milimetric:  i think you don't care where the actual analytics/index.html lives, right?
[17:39:11] <ottomata>	 as long as the main other page links to it?
[17:39:26] <ottomata>	 so, add redirect in nginx if you like, but i think the link being other/analytics/index.html is fine, no?
[17:39:48] <milimetric>	 yeah, I just did that with my new patch
[17:39:55] <milimetric>	 so I both moved it and added the rewrite
[17:40:30] <ottomata>	 ah ok
[17:40:53] <ottomata>	 oh, hm
[17:41:06] <ottomata>	 wait milimetric what's the point of doing a rewrite
[17:41:07] <ottomata>	 ?
[17:41:08] <ottomata>	 permanent
[17:41:16] <ottomata>	 i thought you were maybe going to do it transparently
[17:41:20] <ottomata>	 so the file lives at other/analytics
[17:41:24] <ottomata>	 but the link is still /analytics
[17:41:37] <ottomata>	 this link is new enough that i htink you can leave off a full permanent redirect
[17:41:38] <ottomata>	 if you want
[17:41:40] <milimetric>	 I just copied what was there for fundraising, it seemed to be the same thing, I don't really know nginx
[17:41:45] <ottomata>	 the way you have it
[17:41:52] <ottomata>	 it will cahnge the url in the browser to other/analytics
[17:42:04] <milimetric>	 yea, that
[17:42:07] <ottomata>	 you want that?
[17:42:08] <ottomata>	 why
[17:42:08] <milimetric>	 that's ok, right?
[17:42:09] <ottomata>	 ?
[17:42:11] <ottomata>	 sure it sok
[17:42:14] <ottomata>	 but what's the point?
[17:42:36] <milimetric>	 it just seemed closer to what Ariel would like?
[17:42:52] <ottomata>	 i think ariel just wants you to keep files out of root
[17:42:54] <ottomata>	 that's it
[17:43:05] <ottomata>	 a perm redirect would be for if you needed to move it,but wanted to keep something backwards compatible
[17:43:09] <ottomata>	 but this is new and you dont' need that
[17:43:18] <milimetric>	 aha, ok, so what's the code for the transparent kind of rewrite?
[17:43:26] <ottomata>	 a transparent rewrite might be nice if you wanted it, so that the url would be /analytics, but the file would be other/analytics
[17:43:28] <ottomata>	 not sure!
[17:43:29] <ottomata>	 heheh
[17:43:47] <ottomata>	 i gotta run to lunch, sorry! but i'd say don't worry about it, if all the other non xml dumps are in /other/<somehting>
[17:43:52] <ottomata>	 might as well just use /other/analytics
[17:43:54] <ottomata>	 no?
[17:44:08] <ottomata>	 my opinion: don't do any rewrite, just use other/analytics
[17:44:12] <ottomata>	 unless you really want the url to be /analytics
[17:44:24] <ottomata>	 ok, gotta go, will merge after lunch, sok? or you can get ariel or   someoe else to merge
[17:44:40] <wikibugs>	 Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2145178 (bearND) @Jdlrobson, @dr0ptp4kt We just add a header like this:  ``` x-analytics: preview=1 ```  This is for both [[ https://phabricator.wikimedia.org/diffusion/APAW/browse/master/app/src/ma...
[17:44:44] <milimetric>	 oh, ottomata I would agree except the email went out and it has /analytics in there
[17:44:54] <milimetric>	 so I figure if that's how people click through they shouldn't see a 404
[17:45:25] <milimetric>	 ottomata: have a good lunch, no worries or rush, I think this is a non-breaking style change, we can merge it when Ariel's ok with it
[17:46:12] <ottomata>	 ook ok!
[18:01:39] <joal>	 bd808: many apologizes for the name confusion :)
[18:02:30] <joal>	 bd808: I deployed today a new version of refinery and refinery source
[18:02:46] <joal>	 bd808: Are you hove/oozie jobs ready to be running\ ?
[18:05:56] <wikibugs>	 Analytics, Operations, hardware-requests, Patch-For-Review: eqiad: (3) AQS replacement nodes - https://phabricator.wikimedia.org/T124947#2145301 (RobH) The systems that can be used for this were ordered today on T130738.  I'm now assigning this task to @Mark.Otaris  @Mark:  Please review the above...
[18:13:17] <kaldari>	 milimetric: When exactly do stats for a new day become available via the pageviews API? Right after midnight UTC? midnight some other time zone? Some time later?
[18:13:20] <wikibugs>	 Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2145344 (RobH) Sorry about that @ottomata, this was assigned to @mark so I missed your question.  System WMF4541 is an older spare pool system with...
[18:13:59] <milimetric>	 kaldari: it's not a fixed time, and we haven't figured out the average or maximum
[18:14:31] <milimetric>	 kaldari: we're continuously ingesting data, and sometimes there's loss, duplication, etc. so cleaning it takes longer.  Sometimes jobs have to be restarted, and that takes longer too
[18:14:43] <kaldari>	 milimetric: any rough guess?
[18:14:50] <milimetric>	 and it's a somewhat complicated pipeline so things could go wrong in 3 or 4 places.
[18:15:20] <milimetric>	 kaldari: from what I've seen, data for the previous day is usually available within 5 hours of midnight.
[18:15:33] <milimetric>	 but that's not very scientific
[18:15:51] <wikibugs>	 Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2145365 (mark) Approved.
[18:16:38] <kaldari>	 milimetric: and that's midnight UTC right?
[18:16:46] <milimetric>	 yes
[18:16:58] <wikibugs>	 Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2145374 (RobH) a:mark>RobH
[18:17:14] <elukey>	 a-team: talk with you tomorroooowwww o/
[18:17:20] <mforns>	 bye elukey !
[18:17:42] <joal>	 bye elukey
[18:40:01] <wikibugs>	 Analytics-Cluster, Operations, hardware-requests: eqiad: New Hive / Oozie server node in eqiad Analytics VLAN - https://phabricator.wikimedia.org/T124945#2145490 (Ottomata) Thanks!
[19:19:21] <wikibugs>	 Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2145687 (Nuria) Loopin @Tbayer here so he knows what to expect, please do not do any code changes quite yet as we are meeting to talk about this in the next couple days.
[19:20:19] <wikibugs>	 Analytics, Hovercards: Capture hovers as previews in analytics - https://phabricator.wikimedia.org/T129425#2145690 (Nuria) >The moment more than one key-value pair starts to be sent for this header, a more deluxe solution will be required to ensure different client side authors don't trump each other's c...
[19:25:51] <grrrit-wm>	 (PS1) Mforns: Fix script after Phabricator layout changes [analytics/limn-analytics-data] - https://gerrit.wikimedia.org/r/279177 (https://phabricator.wikimedia.org/T130543)
[20:17:39] <nuria>	 I am about to deploy a new versions of browser-reports with some piwik code cc milimetric
[20:29:56] <bd808>	 joal: I think that there should be data for the ApiAction oozie job now. I'm not sure how to check to see if camus is pulling the data from kafka into hdfs though.
[20:30:14] <ottomata>	 bd808:  checkin /wmf/data/raw/mediawiki...
[20:31:54] <bd808>	 ottomata: /mnt/hdfs/wmf/data/raw/mediawiki/mediawiki_ApiAction/hourly exists but is empty
[20:33:20] <ottomata>	 bd808:  in meeting, with you afer
[20:33:27] <bd808>	 np. thx
[20:44:13] <grrrit-wm>	 (PS1) Madhuvishy: TEST COMMIT DO NOT MERGE [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279187
[20:44:37] <grrrit-wm>	 (Abandoned) Madhuvishy: TEST COMMIT DO NOT MERGE [analytics/refinery/source] - https://gerrit.wikimedia.org/r/279187 (owner: Madhuvishy)
[20:46:05] <grrrit-wm>	 (PS1) Madhuvishy: Test commit do not merge [analytics/refinery/source] (release) - https://gerrit.wikimedia.org/r/279188
[21:00:24] <ottomata>	 bd808:  ok hey
[21:00:25] <ottomata>	 looking
[21:04:10] <ottomata>	 hmm, bd808
[21:04:10] <ottomata>	 java.lang.RuntimeException: null record
[21:04:24] <ottomata>	 	at com.linkedin.camus.etl.kafka.mapred.EtlRecordReader.nextKeyValue(EtlRecordReader.java:295)
[21:07:03] <bd808>	 ottomata: hmm... does that mean my Avro schema is wrong or ??
[21:07:31] * bd808 is multi-tasking with RfC meeting
[21:08:35] <ottomata>	 no tsure
[21:09:11] <ottomata>	 i'm looking at code to see if i could see a path for that
[21:13:30] <bd808>	 ottomata: looks like it might be schema related -- https://github.com/linkedin/camus/blob/master/camus-etl-kafka/src/main/java/com/linkedin/camus/etl/kafka/mapred/EtlRecordReader.java#L139-L156
[21:16:11] <ottomata>	 yeah, maybe, but there's only one version of that schema out there, no?
[21:16:15] <ottomata>	 the one in event-schemas?
[21:16:28] <ottomata>	 unless..maybe refinery-camus hasn't been rebuilt and deployed with that schema
[21:16:30] <ottomata>	 checking
[21:18:18] <ottomata>	 gotta run in 3 mins tough...
[21:18:38] <bd808>	 ottomata: if we have data by the end of the week I'll be happy :)
[21:18:43] <ottomata>	 hmm, i tlooks there
[21:18:44] <ottomata>	 ok
[21:18:47] <ottomata>	 bd808:  yeah sorry
[21:18:51] <bd808>	 no worries
[21:18:58] <ottomata>	 i'm unfortunetly not familiar with the avro/camus integration...
[21:19:05] <ottomata>	 madhuvishy:  miiiight be able to help you
[21:19:21] <ottomata>	 fwiw i was able to consume an offending offset that caused that error in camus from kafka, and then read the data out of it using that schema
[21:19:30] <ottomata>	 bd808:  it looks to me like its there in refinery-camus.jar
[21:19:34] <ottomata>	   1464 Wed Mar 23 11:20:44 UTC 2016 schema_repo/avro/mediawiki/ApiAction/101453221640.avsc
[21:19:45] <ottomata>	 was deployed this morning i think by joal maybe
[21:19:49] <ottomata>	 but could have been there longer too
[21:20:05] <bd808>	 I'll try to decode a packet in a bit. If I get really stuck I'll poke some discovery folks who have dealt with this pipeline.
[21:20:37] <ottomata>	 btw, in case you need the schemas, they are now also checked out on stat1002 at /srv/event-schemas
[21:20:50] <ottomata>	 ok, laters!