[00:33:41] <wikibugs>	 Analytics-EventLogging: Investigate duplicate EventLogging rows - https://phabricator.wikimedia.org/T142667#2542598 (Tbayer)
[07:57:05] <grrrit-wm>	 (PS1) Joal: Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184
[07:58:50] <grrrit-wm>	 (PS2) Joal: Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184
[08:17:17] <elukey>	 a-team: I need to restart the kafka analytics cluster for JVM upgrades, if you see anything weird please let me know :)
[08:17:27] <joal>	 elukey: Sure !
[08:17:27] * elukey looks at eventlogging
[08:28:19] <elukey>	 EL seems fine
[08:28:21] <elukey>	 woa
[08:33:36] <joal>	 elukey: woa?
[08:34:11] <elukey>	 I meant "WOOOOOOOO!"
[08:34:24] <joal>	 Ahhhh :)
[08:34:55] <elukey>	 I can see errors in the logs but then everything recovers with a metadata refresh
[08:40:20] <elukey>	 great work from Andrew
[08:53:18] <grrrit-wm>	 (PS18) Joal: [WIP] Refactor Mediawiki History scala code [analytics/refinery/source] - https://gerrit.wikimedia.org/r/301837 (https://phabricator.wikimedia.org/T141548)
[10:00:53] <grrrit-wm>	 (PS1) Joal: Modify bin/camus script to follow refinery update [analytics/refinery] - https://gerrit.wikimedia.org/r/304194
[10:04:47] <mforns>	 hi a-team!
[10:04:53] <joal>	 Hi mforns !
[10:05:08] <joal>	 mforns: I pushed some changes on scala code if you want to rebase :)
[10:05:44] <mforns>	 joal, ok, cool, I pushed yesterday after logging off, so I think I won't need to
[10:05:55] <joal>	 mforns: Arf, I didn't update
[10:05:58] <joal>	 mforns: will do !
[10:06:23] <mforns>	 joal, oh, np I can do that, don't worry
[10:06:35] <joal>	 mforns: nono, really, I should have double checked!
[10:13:11] <elukey>	 mobrovac: I'd need to restart kafka on kafka200[12] for jvm upgrades, anything against it?
[10:13:31] <mobrovac>	 nope elukey, go ahead
[10:13:39] <mobrovac>	 elukey: will you need to do it in eqiad too?
[10:19:31] <elukey>	 mobrovac: it will be done by Andrew next week as part of the kafka 0.9 upgrade
[10:20:03] <mobrovac>	 kk awesome
[10:27:32] <joal>	 elukey: loading of 2nd month done
[10:27:39] <joal>	 elukey: waiting for compaction
[10:28:14] <elukey>	 goooood
[10:29:04] <elukey>	 let me know after lunch when you'll have 20 mins to chat about manual loading and next steps for the password thing
[10:31:40] <joal>	 elukey: sure, in like 15mins?
[10:31:57] <joal>	 elukey: I'd like to finish something with mforns :)
[10:32:11] <joal>	 mforns: I have an issue with some tests you wrote: they fail :(
[10:32:23] <mforns>	 joal, oh...
[10:32:36] <mforns>	 they were passing for me... in mvn at least
[10:32:48] <joal>	 mforns: I don't understand because mvn passed for your patch, yes exactly
[10:33:08] <mforns>	 joal, batcave?
[10:33:10] <joal>	 sure
[10:33:13] <mforns>	 ok
[10:38:28] <elukey>	 joal: let's do it in a couple of hours ok?
[10:47:33] <grrrit-wm>	 (PS19) Joal: [WIP] Refactor Mediawiki History scala code [analytics/refinery/source] - https://gerrit.wikimedia.org/r/301837 (https://phabricator.wikimedia.org/T141548)
[10:47:40] <joal>	 mforns: --^
[10:47:49] <mforns>	 thanks a lot joal
[10:47:50] <joal>	 mforns: Sorry for the mess of rebases :(
[10:48:04] <joal>	 mforns: I added comments with todo on tests that failed for me
[10:48:09] <mforns>	 np, thanks for changing the subgraph thing
[10:48:14] <mforns>	 ok
[10:48:15] <joal>	 a-team: taking a break :)
[10:59:14] * elukey lunch!
[11:39:40] <wikibugs>	 Analytics: Test - https://phabricator.wikimedia.org/T142702#2543554 (SaiLahariMedikonda)
[11:40:11] <wikibugs>	 Analytics: Test - https://phabricator.wikimedia.org/T142702#2543575 (SaiLahariMedikonda) a:SaiLahariMedikonda>None
[12:16:22] <wikibugs>	 Analytics-Cluster: Queries in Hue always return an empty result set - https://phabricator.wikimedia.org/T128039#2543720 (akosiaris)
[12:56:11] <elukey>	 team I am going to restart cassandra on aqs100[123] for security upgrades
[12:59:03] <wikibugs>	 Analytics-Cluster, Operations: Migrate titanium to jessie (archiva.wikimedia.org upgrade) - https://phabricator.wikimedia.org/T123725#2543774 (MoritzMuehlenhoff) a:MoritzMuehlenhoff>None
[13:06:52] <elukey>	 aqs1001 done, will proceed in a bit with 1002
[13:09:05] <grrrit-wm>	 (CR) Ottomata: [C: 2] Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (owner: Joal)
[13:09:10] <grrrit-wm>	 (CR) Ottomata: [C: 1] Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (owner: Joal)
[13:09:27] <grrrit-wm>	 (CR) Ottomata: [C: 1] Modify bin/camus script to follow refinery update [analytics/refinery] - https://gerrit.wikimedia.org/r/304194 (owner: Joal)
[13:42:19] <elukey>	 joal: I have a short-medium term idea for the cassandra admin username
[13:44:22] <elukey>	 but I'd need to know first what credentials a 'aqs-loader' user will need
[13:44:37] <elukey>	 atm 'aqs' has CREATE, ALTER, DROP, SELECT, MODIFY
[13:45:05] <elukey>	 not sure if INSERT is allowed but I guess so
[13:45:44] <elukey>	 anyhow, the idea would be to create the new user with minimal permissions and assign the pass that we are using for 'cassandra'
[13:46:26] <elukey>	 then we'll be able to 1) avoid using the admin user 2) change the admin password 3) force oozie to use a user with less priviledges
[13:46:55] <elukey>	 in this way we wouldn't solve completely the problem but at least we'd limit the security exposure
[13:47:11] <elukey>	 then the last step would be to secure the aqs-loader password
[13:59:39] <joal>	 elukey: sounds a very good idea :)
[14:00:32] <joal>	 elukey: we'll need to test, but I think SELECT and MODIFY are enough for the loader (i'd even say modify only, but select is cheap ;)
[14:02:49] <elukey>	 ah MODIFY == INSERT, DELETE, UPDATE, TRUNCATE
[14:04:33] <elukey>	 joal: my idea is to create a similar adduser.cql file (like addaqsloaderuser.cql) storing the initial password in the private repo
[14:04:54] <elukey>	 push the file to our aqs hosts
[14:05:08] <elukey>	 and create the user with it
[14:05:22] <elukey>	 so it will be easy to redo it
[14:05:25] <elukey>	 and more documented
[14:05:30] <elukey>	 then we just try it
[14:06:43] <elukey>	 let me see if I can create a puppet patch
[14:07:13] <joal>	 sounds good elukey
[14:07:33] <joal>	 elukey: only thing I don't yet forsee is how to pass the password to oozie job
[14:08:17] <elukey>	 I tried to think something today and failed miserably
[14:08:31] <joal>	 huhu :)
[14:08:41] <elukey>	 I mean, failed == ended up with a big hack
[14:10:15] <wikibugs>	 Analytics-Cluster, Graphoid, Operations, Services, and 2 others: Graphoid access logs are missing from Hadoop - https://phabricator.wikimedia.org/T99372#2544017 (akosiaris) parsoidcache has been deprecated and graphoid is now exposed via the text cluster. That solves the problem and graphoid logs...
[14:10:37] <wikibugs>	 Analytics-Cluster, Graphoid, Operations, Services, and 2 others: Graphoid access logs are missing from Hadoop - https://phabricator.wikimedia.org/T99372#2544018 (akosiaris) Open>Resolved a:akosiaris
[14:16:03] <wikibugs>	 Analytics, Operations, Traffic: Correct cache_status field on webrequest  dataset - https://phabricator.wikimedia.org/T142410#2544038 (akosiaris) p:Triage>Normal
[14:16:44] <wikibugs>	 Analytics, Operations, Traffic: Correct cache_status field on webrequest  dataset - https://phabricator.wikimedia.org/T142410#2544040 (BBlack) Open>Resolved a:BBlack I think we're done here, assuming the data looks sane on the analytics end.
[14:22:33] <elukey>	 ouch not that easy to put into the aqs module
[14:22:50] <elukey>	 it references a lot of cassandra instance puppet class things
[14:22:51] <elukey>	 grr
[14:23:55] <wikibugs>	 Analytics-Cluster, Operations: Migrate titanium to jessie (archiva.wikimedia.org upgrade) - https://phabricator.wikimedia.org/T123725#2544059 (akosiaris) p:Triage>Normal
[14:37:16] <mforns>	 joal, milimetric, hi! do you have 10 mins for opinion on tests?
[14:37:24] <joal>	 Hi mforns
[14:37:28] <joal>	 mforns: sure
[14:37:34] <milimetric>	 hi mforns, I'm struggling with oozie but I have no opinion on tests
[14:37:39] <milimetric>	 I'll do it however you guys like
[14:37:44] <mforns>	 milimetric, xD
[14:38:04] <mforns>	 milimetric, continue with oozie, it's OK I'll ask joseph
[14:38:08] <mforns>	 joal, batcave?
[14:38:13] <joal>	 mforns: OMW !
[14:38:21] <mforns>	 milimetric, join if you think you can't miss this hehe
[14:38:48] <milimetric>	 :) thx
[14:53:17] <ottomata>	 dudes my internet at this care seems really really bad
[14:53:21] <ottomata>	 will try standup but i dunno
[14:53:29] <ottomata>	 also, my phone is still busted so i can't use phone internet :/
[14:55:40] <milimetric>	 we could see who's on and just IRC standup
[14:55:57] <joal>	 no prob for me
[15:05:38] <wikibugs>	 Analytics-Kanban: Update refinery to solve dependency conflicts - https://phabricator.wikimedia.org/T142717#2544163 (JAllemandou)
[16:00:30] <ottomata>	 elukey:  re aqsloader user.  why not just 'aqs' for name.  second:  i think it would be totally fine to put the oozie .properties file in puppet as a template.
[16:00:33] <ottomata>	 don't put any .xml files in there
[16:00:52] <ottomata>	 just the one .properties file that will be used to launch the job.
[16:00:52] <ottomata>	 OR
[16:00:53] <ottomata>	 OR
[16:01:01] <ottomata>	 elukey: don't put the password anywhere
[16:01:01] <ottomata>	 actually
[16:01:02] <ottomata>	 yeah
[16:01:12] <ottomata>	 the password could be passed in on the CLI.
[16:01:13] <ottomata>	 HMMMM
[16:01:18] <ottomata>	 but then again.
[16:01:31] <ottomata>	 all the properties will be exposed in the job browser / in hue
[16:01:31] <elukey>	 yeah today I had a lot of "hmmmmm"s too
[16:01:49] <ottomata>	 so yeah , hm, it will be readable no matter what we do.
[16:01:50] <ottomata>	 i thikn
[16:02:33] <joal>	 ottomata, elukey : We could modify cassandra loading job to be given a file with the password for cassandra
[16:02:58] <ottomata>	 was about to suggest something like that
[16:03:00] <joal>	 This file should be hdfs readable-only on hdfs
[16:03:04] <ottomata>	 +1
[16:03:29] <ottomata>	 a-team i can barely hear anything in the meeting, sorry
[16:03:33] <ottomata>	 if there is something you need me to task let me know
[16:04:14] <elukey>	 let me if you want to apply my short to medium solution (so maybe adding the user  manually as temp step is fine) or if we want to go straight to the longer term one
[16:04:22] <elukey>	 I am ok either way :)
[16:06:08] <wikibugs>	 Analytics-Kanban: Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder - https://phabricator.wikimedia.org/T142724#2544366 (JAllemandou)
[16:06:21] <wikibugs>	 Analytics-Kanban: Productionize scala code for edit reconstruction - https://phabricator.wikimedia.org/T142552#2544380 (JAllemandou) a:JAllemandou>None
[16:06:33] <wikibugs>	 Analytics-Kanban: Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder - https://phabricator.wikimedia.org/T142724#2544366 (JAllemandou) p:Triage>Normal
[16:06:41] <wikibugs>	 Analytics-Kanban: User history: Adapt the user history reconstruction to use scaling by clustering - https://phabricator.wikimedia.org/T141774#2544383 (Nuria) This task was about parallelization strategy, which worked,  but code is not final.   Following up will be done on a different task.
[16:10:52] <wikibugs>	 Analytics-Kanban: Productionize scala code for edit reconstruction - https://phabricator.wikimedia.org/T142552#2544395 (Nuria)
[16:13:17] <wikibugs>	 Analytics-Kanban: Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder - https://phabricator.wikimedia.org/T142724#2544401 (Nuria)
[16:13:19] <wikibugs>	 Analytics-Kanban: Productionize scala code for edit reconstruction - https://phabricator.wikimedia.org/T142552#2544400 (Nuria)
[16:13:35] <ottomata>	 elukey:  to puppet?  naw i think its fine not to puppetize the cassandra user
[16:13:39] <ottomata>	 i'd just add it and document.
[16:14:23] <ottomata>	 parameterize the username and pw in puppet somehow, but not the creattion/grant in cassandra
[16:15:50] <wikibugs>	 Analytics-Kanban: Page History: Add unit tests to PageHistoryDataExtractors and PageHistoryBuilder - https://phabricator.wikimedia.org/T142724#2544366 (Nuria) Some tests will be units, others -due to nature of code- will only be integration tests.  We need to create some fixtures with fake data to test input...
[16:16:32] <elukey>	 yeah the adduser.cql needs to be manually run
[16:16:35] <elukey>	 ottomata: --^
[16:16:37] <elukey>	 so something like that
[16:19:11] <elukey>	 ottomata: just configured the new disk on an1045 and rebooted the host
[16:19:14] <elukey>	 all good
[16:19:45] <ottomata>	 nice!  thanks elukey
[16:19:55] <ottomata>	 elukey:  yeah, just manually add the user and document, i think thats totally enough for creating the user
[16:21:17] <musikanimal>	 hello, all! any idea why only some pageviews get recorded for Special:Random? http://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=user&range=latest-20&pages=Special:Random
[16:21:50] <wikibugs>	 Analytics-Kanban: Update Refinery to solve dpendency onflicts - https://phabricator.wikimedia.org/T142717#2544409 (Nuria)
[16:23:46] <musikanimal>	 those numbers are far too low, but obviously Special:Random isn't a page you can actually browse to (and returns a 302 redirect and not a 301, if that matters). So it's interesting we're still seeing some pageviews
[16:24:48] <elukey>	 ottomata: all right, will work on that tomorrow, thanks :)
[16:36:15] <wikibugs>	 Analytics-Kanban: Productionize edit history extraction for all wikis using Sqoop - https://phabricator.wikimedia.org/T141476#2544435 (Nuria)
[16:41:46] <joal>	 ottomata: ready for a big deploy time now ?
[16:42:00] <grrrit-wm>	 (PS3) Joal: Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (https://phabricator.wikimedia.org/T142717)
[16:45:05] <grrrit-wm>	 (PS1) Joal: Update changelog for v0.0.34 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304259
[16:45:28] <joal>	 Arf ... looks like ottomata is not here anymore :(
[16:46:36] <grrrit-wm>	 (PS2) Joal: Modify bin/camus script to follow refinery update [analytics/refinery] - https://gerrit.wikimedia.org/r/304194 (https://phabricator.wikimedia.org/T142717)
[16:48:21] <ottomata>	 here!
[16:48:21] <ottomata>	 joal:
[16:48:26] <joal>	 Yay :)
[16:48:26] <ottomata>	 soryr, was in 1:1
[16:48:30] <joal>	 np ottomata :)
[16:48:31] <ottomata>	 ja let's do it
[16:48:33] <joal>	 cool
[16:48:42] <joal>	 ottomata: 2 patches on refinery-source to start :)
[16:48:44] <joal>	 please :)
[16:49:41] <ottomata>	 holy moly slow internet
[16:49:46] <ottomata>	 waiting for gerrit pages to load
[16:49:54] <joal>	 :d
[16:52:12] <ottomata>	 joal:  +2, merge whatever you can
[16:52:14] <ottomata>	 i'll do the puppet one
[16:52:17] <ottomata>	 when its time
[16:52:21] <joal>	 ottomata: Ok, doing
[16:52:26] <ottomata>	 i think you'llh ave to deploy
[16:52:31] <joal>	 ottomata: no prob
[16:52:41] <grrrit-wm>	 (CR) Joal: [C: 2] Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (https://phabricator.wikimedia.org/T142717) (owner: Joal)
[16:53:12] <grrrit-wm>	 (CR) Joal: [C: 2] Update changelog for v0.0.34 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304259 (owner: Joal)
[16:53:56] <grrrit-wm>	 (CR) Joal: [C: 2] Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (https://phabricator.wikimedia.org/T142717) (owner: Joal)
[16:54:09] <joal>	 Man was on the wrong patch set
[16:55:54] <ottomata1>	 YES phone decided to work
[16:55:57] <ottomata1>	 phone internet activated!
[16:56:41] <ottomata1>	 joal:  lemme know when you have deployed and I can merge and run puppet
[16:56:59] <joal>	 ottomata1: Yup. Jenkins taking his time for the moment :)
[16:57:12] <grrrit-wm>	 (CR) Joal: [V: 2] Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (https://phabricator.wikimedia.org/T142717) (owner: Joal)
[16:57:16] <grrrit-wm>	 (Merged) jenkins-bot: Update maven jar building in refinery [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304184 (https://phabricator.wikimedia.org/T142717) (owner: Joal)
[16:57:18] <grrrit-wm>	 (Merged) jenkins-bot: Update changelog for v0.0.34 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/304259 (owner: Joal)
[17:01:24] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Merging for deploy." [analytics/refinery] - https://gerrit.wikimedia.org/r/304194 (https://phabricator.wikimedia.org/T142717) (owner: Joal)
[17:03:58] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] Modify bin/camus script to follow refinery update [analytics/refinery] - https://gerrit.wikimedia.org/r/304194 (https://phabricator.wikimedia.org/T142717) (owner: Joal)
[17:07:40] <joal>	 elukey: have you seen aqs latencies since you restarted the machines?
[17:07:52] <joal>	 elukey: I want to know more !!!
[17:12:28] <wikibugs>	 Analytics-Kanban: Update Refinery to solve dependencies conflicts - https://phabricator.wikimedia.org/T142717#2544630 (JAllemandou)
[17:19:10] <joal>	 ottomata: the old build and dpeloy system was less good, but it was faster ;)
[17:27:14] <ottomata>	 aye
[17:27:16] <ottomata>	 haha
[17:27:19] <ottomata>	 well, i mean, sorta
[17:27:26] <ottomata>	 except for maybe the upload artifacts to archiva
[17:27:28] <ottomata>	 that was still really slow
[17:39:48] <joal>	 ottomata: I was doing it from stat1002, reasonnably fast
[17:40:20] <joal>	 ottomata: refinery-source done
[17:40:42] <joal>	 ottomata: We need to sync the refinery and puppet deploy
[17:45:00] <joal>	 ottomata: Ready whenever you want :)
[18:02:13] <ottomata>	 was eating lunch
[22:31:28] <grrrit-wm>	 (PS5) Milimetric: [WIP] Oozify sqoop import of mediawiki tables [analytics/refinery] - https://gerrit.wikimedia.org/r/303339 (https://phabricator.wikimedia.org/T141476)
[22:48:54] <lzia>	 hey milimetric. do you have few min (if you're still around.)?
[23:12:21] <milimetric>	 hey lzia
[23:12:23] <milimetric>	 what's up
[23:12:45] <lzia>	 ow hey milimetric. :)
[23:13:12] <lzia>	 so, I wanted to see if I'm listing the right steps in https://phabricator.wikimedia.org/T142780 if I'm requesting access to webrequest logs for two researchers?
[23:13:58] <milimetric>	 lzia: task looks good, just tag Ops-Access-Requests on it
[23:14:05] <milimetric>	 it'll get discussed at the meeting and done
[23:14:09] <lzia>	 perfect. thank you milimetric.
[23:14:17] <milimetric>	 lzia: but they're under the NDA right?
[23:14:20] <lzia>	 yup
[23:14:27] <lzia>	 NDA and MOU are already signed by all parties. :)
[23:14:47] <milimetric>	 k, and in the future if you forget, just remember to type "access-requests" in phab and you can see the other examples that tag it
[23:14:54] <milimetric>	 cool, welcome to more researchers :)
[23:14:54] <milimetric>	 yay
[23:15:12] <lzia>	 oh! thank you! :) and thank you! :)
[23:15:34] <wikibugs>	 Analytics-Kanban, Trash: ---- DISCUSSED BELOW ---- - https://phabricator.wikimedia.org/T114124#2546198 (Danny_B) Hello. Why don't you create another column then actually?  You should be aware of that it is highly unreliable way of sorting and absolutely without any guarantee that that sorting will remain...
[23:17:58] <wikibugs>	 Analytics-Kanban, Trash: ---- RUBICON ---- - https://phabricator.wikimedia.org/T104390#2546207 (Danny_B) Hello. Why don't you create another column then actually?  You should be aware of that it is highly unreliable way of sorting and absolutely without any guarantee that that sorting will remain the sam...