[08:41:36] <elukey>	 joal: morning!
[08:41:51] <elukey>	 sudo -u hdfs /usr/bin/yarn application -appStates RUNNING -list | egrep -o 'analytics10[0-9][0-9].eqiad.wmnet' | sort | uniq -c gives me the active containers
[08:41:57] <elukey>	 and their application masters
[08:43:29] <elukey>	 (sorry it only returns the app masters)
[08:50:54] <elukey>	 anyhow, I am going to restart the daemons on the hadoop worker nodes
[09:40:10] <wikibugs_>	 10Analytics-Tech-community-metrics: Kibana's Mailing List data sources are outdated - https://phabricator.wikimedia.org/T146632#2810227 (10Lcanasdiaz) Fixed: * https://wikimedia.biterg.io:443/goto/c7f681328f4cf6bcb467985117718dcb
[09:44:53] <wikibugs_>	 10Analytics-Tech-community-metrics: Maniphest support to be added to GrimorieLab - https://phabricator.wikimedia.org/T138003#2810241 (10Lcanasdiaz) This is done.  * https://github.com/grimoirelab/GrimoireELK/commit/4837abdf6b12a2809e4b930f0047da28cd01027e (lastest commit fixing current backend phab's maniphest)...
[09:45:19] <wikibugs>	 10Analytics-Tech-community-metrics: Deployment of IRC panel - https://phabricator.wikimedia.org/T138004#2810242 (10Lcanasdiaz) We are working on this. Will be done before Friday Nov 25th.
[09:48:07] <wikibugs_>	 10Analytics-Tech-community-metrics: Check whether mailing list activity per person on korma is in sync with current "mlstats_mailing_lists.conf" - https://phabricator.wikimedia.org/T132907#2810245 (10Lcanasdiaz) a:03Lcanasdiaz
[10:14:10] <joal>	 Hi elukey, I'll keep an eye on jobs while you're restarting, but I won't do mor
[10:14:48] <joal>	 elukey: I'm not feeling good today, very bad headache
[10:15:35] <elukey>	 !log created 0039892-161020124223818-oozie-oozi-C and 0039895-161020124223818-oozie-oozi-C for webrequest-load-wf-(text|upload)-2016-11-21-8
[10:15:36] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[10:16:02] <elukey>	 joal: already restarted all the nodes except masters, oozie, etc.. will do it later on :)
[10:16:09] <elukey>	 take care!
[10:16:12] <joal>	 awesome, thanks mate :)
[11:12:51] * elukey afk!
[11:45:12] <elukey>	 !log stopped all the oozie bundles via Hue as prep-step for Hive/Oozie daemons restarts on an1003
[11:45:13] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[12:07:55] <elukey>	 !log restarted oozie bundles via hue after oozie/hive restart
[12:07:56] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[12:36:55] <grrrit-wm>	 (03CR) 10Elukey: [C: 032] Improve oozie data loss alarms [analytics/refinery] - 10https://gerrit.wikimedia.org/r/319582 (https://phabricator.wikimedia.org/T148980) (owner: 10Mforns) 
[13:48:55] <ottomata>	 goodmornnniiing
[13:55:36] <elukey>	 o/
[14:38:37] <elukey>	 a-team, as I wrote in the E-Scrum I am going afk for today, but I'll check later on IRC/emails so please let me know if you have anything for me :)
[14:38:46] <ottomata>	 ok, laters elukey!
[14:38:51] <ottomata>	 mforns:  can you do a video chat test with me?
[14:39:03] <ottomata>	 i'm not sure how my internet will be have here, need to decide if i should go to a cafe
[14:39:06] <mforns>	 ottomata, sure
[14:39:14] <mforns>	 to the batcave!
[14:51:29] <grrrit-wm>	 (03CR) 10Ottomata: [C: 031] "I'm all for doing it, but I don't want to focus on it right now. I'm just trying to get some basic improvements into this thing so we can" [analytics/statsv] - 10https://gerrit.wikimedia.org/r/321911 (https://phabricator.wikimedia.org/T150765) (owner: 10Ottomata) 
[14:54:00] <grrrit-wm>	 (03CR) 10Ottomata: [V: 032] Use kafka-python instead of pykafka [analytics/statsv] - 10https://gerrit.wikimedia.org/r/321550 (owner: 10Ottomata) 
[14:54:21] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Use argparse to make statsv configurable [analytics/statsv] - 10https://gerrit.wikimedia.org/r/321911 (https://phabricator.wikimedia.org/T150765) (owner: 10Ottomata) 
[15:06:08] <wikibugs_>	 10Analytics, 06Research-and-Data: Hash IPs on webrequest table - https://phabricator.wikimedia.org/T150545#2810727 (10Tbayer) >>! In T150545#2806515, @leila wrote: > @Tbayer two points to share: 1) I've signed up to reach out to all the people who may be using this data. I will reach out to you and other resea...
[15:28:49] <wikibugs>	 10Analytics, 06Research-and-Data: Hash IPs on webrequest table - https://phabricator.wikimedia.org/T150545#2810760 (10Tbayer) The task description currently reads (in full):  //"Hash IPs on webrequest table if the research for understanding how this data is being currently used or can be used in the future res...
[15:32:31] <grrrit-wm>	 (03PS1) 10Ottomata: Convert statsd port into int [analytics/statsv] - 10https://gerrit.wikimedia.org/r/322677 
[15:33:22] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Convert statsd port into int [analytics/statsv] - 10https://gerrit.wikimedia.org/r/322677 (owner: 10Ottomata) 
[16:14:37] <wikibugs_>	 10Analytics: kafka alarms audit - https://phabricator.wikimedia.org/T151211#2810853 (10Nuria)
[16:15:51] <wikibugs>	 10Analytics, 10Analytics-Cluster: kafka alarms audit - https://phabricator.wikimedia.org/T151211#2810868 (10Ottomata)
[16:18:45] <wikibugs_>	 10Analytics, 10Analytics-Cluster: kafka alarms audit - https://phabricator.wikimedia.org/T151211#2810853 (10Ottomata) Also related: T150277
[16:21:12] <wikibugs_>	 10Analytics-Cluster, 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: Kafka1022 needs a new disk - https://phabricator.wikimedia.org/T151028#2810879 (10Nuria)
[16:24:35] <wikibugs>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Oct-Dec-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2810890 (10Aklapper)
[16:32:53] <wikibugs>	 06Analytics-Kanban: Varnishkafka testing framework - https://phabricator.wikimedia.org/T147432#2810911 (10Nuria)
[16:33:00] <wikibugs_>	 10Analytics-Cluster, 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: Kafka1022 needs a new disk - https://phabricator.wikimedia.org/T151028#2810913 (10Cmjohnson) @ottomata @elukey the disk has been replaced, you will most likely need to add it back
[16:39:07] <wikibugs_>	 10Analytics-Tech-community-metrics: "Changesets" on "Gerrit-Delays" should not list ABANDONED changesets? - https://phabricator.wikimedia.org/T151215#2810948 (10Aklapper)
[16:39:37] <wikibugs_>	 10Analytics-Tech-community-metrics: Deployment of Gerrit Delays panel for engineering - https://phabricator.wikimedia.org/T138752#2408971 (10Aklapper) >>! In T138752#2597638, @Qgil wrote: > There are metric about MERGED and ABANDONED changesets there. I would expect to see only metrics about open changesets in t...
[16:39:45] <wikibugs>	 10Analytics: Figure out a rollback strategy if the release job fails {hawk} - https://phabricator.wikimedia.org/T132179#2190755 (10Nuria) Some info on this, not sure if it provides enough value for us to tackle it: http://maven.apache.org/maven-release/maven-release-plugin/examples/rollback-release.html
[16:40:05] <wikibugs_>	 06Analytics-Kanban: Get jenkins to automate releases {hawk} - https://phabricator.wikimedia.org/T130122#2810981 (10Nuria)
[16:40:07] <wikibugs>	 10Analytics: Figure out a rollback strategy if the release job fails {hawk} - https://phabricator.wikimedia.org/T132179#2810980 (10Nuria) 05Open>03declined
[16:40:12] <wikibugs_>	 10Analytics-Cluster, 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: Kafka1022 needs a new disk - https://phabricator.wikimedia.org/T151028#2810982 (10Ottomata) k will look at it shortly.
[16:48:23] <wikibugs_>	 10Analytics, 10Pageviews-API: Improve pageviews error messages on invalid project - https://phabricator.wikimedia.org/T129899#2811007 (10Nuria) Issues have been corrected when parsing projects IT.wikipedia will not return an error, jhgkjgkg.wikipedia will return an error like: "The date(s) you used are valid,...
[16:48:38] <wikibugs>	 10Analytics, 10Pageviews-API: Improve pageviews error messages on invalid project - https://phabricator.wikimedia.org/T129899#2811008 (10Nuria) 05Open>03declined
[16:51:36] <wikibugs_>	 10Analytics-Tech-community-metrics: "Patchsets Statistics Per Review" widget on "Gerrit" is incomprehensible - https://phabricator.wikimedia.org/T151218#2811027 (10Aklapper)
[16:52:42] <wikibugs_>	 10Analytics, 10RESTBase, 06Services, 15User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2811049 (10Nuria) @GWicke : do we still care about this issue or can this ticket be closed?
[17:01:59] <wikibugs>	 10Analytics, 10RESTBase, 06Services, 15User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2811076 (10GWicke) @Nuria, this is still something we'd like to do. The difficult part is settling on a layout.
[17:27:45] <icinga-wm>	 RECOVERY - Kafka Broker Server on kafka1022 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties
[17:28:24] <wikibugs_>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Oct-Dec-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2811192 (10Aklapper)
[17:28:29] <elukey>	 !log unmasked kafka* on kafka1022 after disk swap
[17:28:30] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:28:33] <elukey>	 ottomata: --^
[17:31:04] <ottomata>	 thanks elukey !
[17:31:32] <wikibugs>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Oct-Dec-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2811205 (10Aklapper)
[17:33:06] <nuria>	 ottomata: in order to deploy the new oozie alarms we just have to deploy the hql code and restart jobs correct?
[17:33:16] <nuria>	 cc mforns 
[17:33:32] <ottomata>	 nuria:  correct, hql needs to be in hdfs, ja
[17:33:57] <ottomata>	 but yeah, a restart (with the latest oozie_directory, or refinery_directory) set, will pick up the newly deployeed hql file
[17:34:00] <grrrit-wm>	 (03CR) 10Nuria: [V: 032] Improve oozie data loss alarms [analytics/refinery] - 10https://gerrit.wikimedia.org/r/319582 (https://phabricator.wikimedia.org/T148980) (owner: 10Mforns) 
[17:34:27] <nuria>	 ottomata: which means .. ahem ... deploying just refinery w/o building jars correct?
[17:34:30] <mforns>	 nuria, copy
[17:35:00] <ottomata>	 nuria:  correct
[17:35:24] <nuria>	 mforns: ok, i have merged the code. 
[17:35:34] <nuria>	 ottomata: is there a reason why we should not deploy it?
[17:35:50] <ottomata>	 i see no reason not to deploy it
[17:35:53] <ottomata>	 doooO it!
[17:36:08] <mforns>	 nuria, I'm in meeting with milimetric, can we do it in a bit>?
[17:38:22] <nuria>	 mforns: yes, if I am not arround i should be back shortly, let's try to deploy alarms today
[17:38:35] <elukey>	 nice :)
[17:38:45] <elukey>	 (moment of calm, checking IRC)
[17:39:14] <elukey>	 ottomata: kafka1022 looks good, but it is writing and getting network dataz like crazy from https://grafana.wikimedia.org/dashboard/db/kafka?from=now-1h&to=now
[17:39:23] <mforns>	 nuria, ok
[17:39:24] <elukey>	 expected but let's keep an eye on it :)
[17:42:23] <ottomata>	 great
[17:43:19] * elukey afk again, have a good day/evening people :)
[17:49:14] <wikibugs_>	 10Analytics, 10RESTBase, 06Services, 15User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2811276 (10GWicke) Here is another proposal:  ``` /page/views/{title}/{access}/{agent}/... /feed/views/total/{access}/{agent}/.....
[18:23:20] <wikibugs_>	 10Analytics, 06Research-and-Data: Hash IPs on webrequest table - https://phabricator.wikimedia.org/T150545#2811438 (10leila) >>! In T150545#2810760, @Tbayer wrote: > Depending on the answers, we may find that hashing is useless for the intended purpose of increasing privacy, or that it will disrupt more existi...
[18:25:04] <wikibugs_>	 10Analytics, 06Research-and-Data: Hash IPs on webrequest table - https://phabricator.wikimedia.org/T150545#2811447 (10leila) >>! In T150545#2810727, @Tbayer wrote: >>>! In T150545#2806515, @leila wrote: >> @Tbayer two points to share: 1) I've signed up to reach out to all the people who may be using this data....
[18:33:25] <mforns>	 :]
[18:45:47] <joal>	 !log Launch 0000336-161121120201437-oozie-oozi-C to cover for webrequest-load-text-2016-11-21-13
[18:45:48] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:46:44] <nuria>	 mforns_brb: i am going to deploy the new alarms
[18:49:40] <mforns>	 nuria, hey I'm back
[18:49:45] <mforns>	 nuria, batcave?
[18:50:11] <nuria>	 mforns: Just edited wiki, doing this: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Refinery#How_to_deploy
[18:50:30] <nuria>	 mforns: I am not going to restart jobs cause i think there is a gotcha that i do not quite understand
[18:50:59] <mforns>	 nuria, reading
[18:56:30] <mforns>	 nuria, what is the gotcha?
[19:07:21] <nuria>	 mforns: the jobs that need to be restarted.. let me see if i can figure it out
[19:09:41] <nuria>	 !log deploying improvements to oozie job alarms  to analytics cluster https://gerrit.wikimedia.org/r/#/c/319582/
[19:09:42] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[19:16:22] <nuria>	 mforns: ok deployed now, will be back in a bit and try to see if we can restart jobs
[19:16:32] <mforns>	 nuria, ok
[19:16:33] <nuria>	 cc elukey and joal just as an FYI
[19:35:36] <elukey>	 \o/
[19:35:52] <elukey>	 thanksss
[19:36:39] <elukey>	 I was about to ask for webrequest-load-text-2016-11-21-13 but already taken care, hopefully the last error :)
[19:37:36] <elukey>	 kafka1022 is still churning but hopefully it will settle down soon
[19:50:11] <mforns>	 milimetric, I'll be gone in a couple minutes, do you want to meet tomorrow to work on scala changes?
[19:50:42] <milimetric>	 sure mforns, if joseph's around
[19:50:51] <mforns>	 ok
[19:51:03] <milimetric>	 otherwise I'll keep learning how to make this extension
[20:00:26] <nuria>	 mforns: yt?
[20:01:34] <mforns>	 nuria, yes
[20:01:53] <nuria>	 mforns: what triggers the run of check-sequence-statistics workflow?
[20:02:15] <mforns>	 nuria, I think it's oozie/webrequest/load/bundle.properties
[20:02:19] <mforns>	  the load jobs
[20:05:03] <nuria>	 mforns: let me see
[20:08:30] <nuria>	 mforns: but in order to restart jobs do we ned to restart webrequest or just the sequence statistics ones? seems like we need to restart webrequest
[20:08:36] <nuria>	 ottomata: yt?
[20:09:03] <mforns>	 nuria, I think so, all the parameters needed for the stats workflow are in the load bundle
[20:09:15] <nuria>	 mforns: ok, teh webrequest jobs
[20:09:38] <mforns>	 nuria, we'll need to kill the previous ones no?
[20:10:00] <mforns>	 and there is one for each webrequest_source
[20:10:15] <ottomata>	 Urbanecm:  ya
[20:10:17] <ottomata>	 oops
[20:10:17] <ottomata>	 sorry
[20:10:21] <nuria>	 mforns: ya but wait cause we need to kill them and make sure they do not get stuck which happens if you get the start dates wrong
[20:10:22] <ottomata>	 wrong auto complete ping
[20:10:23] <ottomata>	 nuria:  ya
[20:10:45] <mforns>	 aha
[20:10:55] <nuria>	 ottomata: joal is going to document this so sorry to ask again
[20:10:56] <Urbanecm>	 Should I do something here?
[20:11:09] <nuria>	 ottomata: how do we know since when to restart webrequest jobs?
[20:11:15] <Urbanecm>	 I can see I was mentioned here. 
[20:11:23] <nuria>	 Urbanecm: sorry wrong ping
[20:11:26] <ottomata>	 Urbanecm:  sorry, no, i just started typing nuria's name without the n and autocompleted
[20:11:39] <Urbanecm>	 No problem. 
[20:12:04] <ottomata>	 nuria: we need to look at the last job that ran, not just the next to materialize
[20:12:05] <ottomata>	 so
[20:12:24] <nuria>	 ahem .. what is "materialize?"
[20:12:29] <nuria>	 cc ottomata 
[20:12:35] <mforns>	 :]
[20:12:36] <ottomata>	 materialize means pending execution
[20:12:40] <ottomata>	 it would show up in the oozie ui
[20:12:52] <ottomata>	 but it doesn't mean it has actually run
[20:12:55] <ottomata>	 when we did this last time
[20:13:00] <ottomata>	 we picked the materialized time
[20:13:05] <ottomata>	 and missed a couple of hours
[20:13:19] <ottomata>	 hang on 2 mins, lemme finish something while i have an opsen attention...
[20:13:59] <nuria>	 ottomata: please do
[20:15:31] <ottomata>	 so nuria if we look at https://hue.wikimedia.org/oozie/list_oozie_bundle/0026850-161020124223818-oozie-oozi-B
[20:15:36] <ottomata>	 we can see 'Next materialization'
[20:15:40] <ottomata>	 that's what we used last time
[20:16:05] <ottomata>	 we should look at each of the coords there
[20:16:11] <ottomata>	 to see when we should start the new job
[20:16:18] <nuria>	 ottomata: aham
[20:16:18] <ottomata>	 https://hue.wikimedia.org/oozie/list_oozie_coordinator/0026852-161020124223818-oozie-oozi-C/?bundle_job_id=0026850-161020124223818-oozie-oozi-B
[20:16:21] <ottomata>	 says 20:00
[20:16:37] <ottomata>	 but https://hue.wikimedia.org/oozie/list_oozie_coordinator/0026854-161020124223818-oozie-oozi-C/?bundle_job_id=0026850-161020124223818-oozie-oozi-B and https://hue.wikimedia.org/oozie/list_oozie_coordinator/0026853-161020124223818-oozie-oozi-C/?bundle_job_id=0026850-161020124223818-oozie-oozi-B are running
[20:16:38] <ottomata>	 19:00
[20:16:39] <ottomata>	 right now
[20:16:58] <ottomata>	 so, we should: either wait til those 2 19:00 jobs are done
[20:16:59] <ottomata>	 or.
[20:17:02] <ottomata>	 go ahead and kill now
[20:17:04] <ottomata>	 and start at 19:00
[20:17:13] <ottomata>	 it will rerun the other ones that had already done 19:00
[20:17:14] <ottomata>	 but that is ok
[20:17:44] <nuria>	 ottomata:  i see, this is not so much teh materialization but teh fact that we have 3 partitions and we are starting 3 jobs at once?
[20:17:46] <nuria>	 *the
[20:17:59] <ottomata>	 nuria:  no
[20:18:12] <ottomata>	 materialization, just means that oozie has scheduled the job to be run
[20:18:23] <ottomata>	 its just waiting on data presence to start
[20:18:31] <nuria>	 ottomata: ah ok
[20:19:07] <ottomata>	 so, since often oozie is waiting for data, but has materialized a couple of jobs in the future
[20:19:17] <ottomata>	 if we use materialized time, we'll skip this hours that oozie is currently waiting for
[20:19:32] <ottomata>	 nuria:  i think it is safe to kill now, and re start the load bundle at 19:00
[20:20:54] <nuria>	 ottomata: ok, killed 0026850-161020124223818-oozie-oozi-B
[20:21:18] <ottomata>	 k
[20:21:42] <nuria>	 ottomata: 
[20:21:46] <nuria>	 https://www.irccloud.com/pastebin/fPari37H/
[20:22:20] <ottomata>	 nuria:  +1 looks good
[20:23:04] <nuria>	 !log restarted webrequest jobs 0000454-161121120201437-oozie-oozi-B
[20:23:05] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:25:04] <nuria>	 ottomata:are tehse docs correct? https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie#How_to_deploy_Oozie_production_jobs
[20:25:26] <nuria>	 *these
[20:25:51] <ottomata>	 sure that looks good.  
[20:26:16] <ottomata>	 hmmm
[20:26:20] <ottomata>	 wait
[20:26:22] <ottomata>	    -D refinery_directory=hdfs://analytics-hadoop/wmf/refinery/$REFINERY_VERSION \
[20:26:22] <ottomata>	    -D oozie_directory=hdfs://analytics-hadoop/wmf/refinery/$REFINERY_VERSION/oozie \
[20:26:22] <ottomata>	 ?
[20:26:32] <ottomata>	 oh hm, yeah i guess so.
[20:26:38] <ottomata>	 that variable name is pretty confusing
[20:26:48] <ottomata>	 when i think refinery version, i think refinery-source-version, which is different
[20:26:50] <ottomata>	 but yeah, that's fine
[20:27:06] <ottomata>	 nuria:  maybe put export QUEUE_NAME=produciton
[20:27:11] <ottomata>	 in the example
[20:27:15] <ottomata>	 so its clear that you can change that too
[20:27:22] <nuria>	 ottomata: it is really confusing though
[20:27:40] <nuria>	 ottomata: i could not parse what was supposed to mean
[20:27:47] <ottomata>	 what I just said?
[20:28:30] <nuria>	 ottomata: ah, no, teh wiki, jaja
[20:28:31] <nuria>	 *the
[20:28:48] <nuria>	 I am just going to cut & paste what we just did 
[20:29:32] <ottomata>	 hm ok
[20:29:38] <nuria>	 ottomata: or no?
[20:29:43] <nuria>	 ottomata: you let me know
[20:29:57] <ottomata>	 its kinda got that 'Alternate That Maybe Should replace this section' there
[20:30:04] <ottomata>	 which is kinda like what we did, ja?
[20:31:28] <nuria>	 ottomata: so ahem let's replace it no? idea is that docs should help us do this without me having to ping you on irc
[20:31:40] <nuria>	 ottomata: but now they are real confusing i think
[20:32:02] <mforns>	 thanks ottomata and nuria for deploying this, I was following for next time
[20:32:26] <nuria>	 ottomata: maybe is just me...
[20:33:37] <ottomata>	 nuria: indeed, this page is kinda messy. it mixes an oozie tutorial with one off prod system docs
[20:34:26] <nuria>	 ottomata: ok, just created https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie/Administration
[20:34:45] <ottomata>	 haha, you can't just create it!  you have to put something there if you are going to create it!
[20:34:49] <ottomata>	 :p
[20:35:00] <nuria>	 ottomata: will move somethings 
[20:35:03] <ottomata>	 :)
[20:35:10] <nuria>	 ottomata: and ask team to help improve page
[20:35:59] <ottomata>	 k
[20:36:53] <mforns>	 bye team, see you tomorrow!
[20:42:50] <ottomata>	 bye!
[20:47:42] <nuria>	 ottomata: ok, moved things arround, will ask team to help document gotchas: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie/Administration
[20:51:23] <nuria>	 ottomata: ok, next thing. ahem.... do you want to take a look at the kafka statsd changes, i tested it on your home dir on labs
[20:51:40] <ottomata>	 nuria:  cool, ja will do, we need to talk about that -1 thing, right?
[20:51:55] <ottomata>	 sorry, talking with akosiaris atm, ALMOST got this eventstreams.svc working
[21:00:19] <nuria>	 ottomata: k
[21:01:06] <nuria>	 ottomata: let me know when you have time again
[21:03:08] <ottomata>	 nuria:  k sorry, with you soon i hope
[21:03:22] <nuria>	 ottomata: np at all . really.
[21:03:40] <ottomata>	 its up and working, ut the service alert hasn't fixed itself, so its still showing as down in icinga
[21:18:51] <ottomata>	 nuria:  i dunno what's up with this
[21:18:54] <ottomata>	 let's look at your hting for a bit
[21:19:07] <ottomata>	 i looked at changes, they look good to me
[21:19:18] <ottomata>	 the bootstrap / -1 thing is weird, right?
[21:21:33] <ottomata>	 nuria:  maybe we should add to default filters
[21:22:04] <ottomata>	 to remove partitions.-1 and localhost:9092/bootstrap
[21:22:15] <ottomata>	 hmm or just /bootstream
[21:22:21] <ottomata>	  /bootstrap
[21:23:01] <nuria>	 ottomata: either one, or both
[21:23:17] <nuria>	 ottomata: easy to do, you let me know what is your preference
[21:25:17] <ottomata>	 ok, i think keys in the brokers hash
[21:25:24] <ottomata>	 that end in /bootstrap
[21:25:27] <ottomata>	 should be filtered out
[21:25:28] <ottomata>	 and
[21:25:49] <ottomata>	 anything in a topics.topicName.partitions hash with a -1 key
[21:25:52] <ottomata>	 should be filtered out too
[21:29:36] <nuria>	 ottomata: ok
[21:30:05] <nuria>	 ottomata: i can also change "-1" (which I am not sure what does represent) into something else
[21:30:28] <ottomata>	 no, the -1 in the partition info is some kind of librdkafka artifact, dunno really why its there
[21:30:34] <ottomata>	 none of the info in that object is useful
[21:30:53] <ottomata>	 nuria:  e.g. i'm just talking about the topics.test4.partitions.-1
[21:30:54] <ottomata>	 https://phabricator.wikimedia.org/T145099
[21:30:54] <nuria>	 ottomata: ok, doing those changes now
[21:30:55] <ottomata>	 there
[21:31:13] <ottomata>	 -1 might be an ok gauge value for some metrics
[21:31:23] <ottomata>	 but the info in the partitions.-1 object is useless
[22:10:39] <milimetric>	 ottomata / nuria: just looked at the oozie errors and it looks like an insert query was updated but the table schema is old / not compatible
[22:10:57] <milimetric>	 not sure if that's what you're talking about above, but I can fix it I think, just gotta find what query was changed
[22:11:06] <ottomata>	 Ohhh ja, that table needs update, right? i think the patch changed it
[22:11:20] <milimetric>	 error is: ERROR org.apache.hadoop.hive.ql.Driver  - FAILED: SemanticException [Error 10044]: Line 37:23 Cannot insert into target table because column number/types are different '21': Table insclause-0 has 9 columns, but query has 10 columns.
[22:12:03] <ottomata>	 milimetric:  https://gerrit.wikimedia.org/r/#/c/319582/10/hive/webrequest/create_webrequest_sequence_stats_hourly_table.hql https://gerrit.wikimedia.org/r/#/c/319582/10/hive/webrequest/create_webrequest_sequence_stats_table.hql
[22:12:12] <ottomata>	 need to alter thoes tables and add that field
[22:12:27] <ottomata>	 milimetric:  i can do
[22:12:33] <milimetric>	 k, sure
[22:13:07] <wikibugs>	 10Analytics-Cluster, 06Analytics-Kanban, 06DC-Ops, 06Operations, 10ops-eqiad: Kafka1022 needs a new disk - https://phabricator.wikimedia.org/T151028#2812484 (10Cmjohnson) 05Open>03Resolved
[22:15:09] <ottomata>	 done
[22:15:34] <milimetric>	 so jobs would need to be restarted, right?
[22:16:53] <ottomata>	 rerunning
[22:17:55] <ottomata>	 rerunning just a maps job to make sure
[22:23:23] <ottomata>	 cool that worke,d rerunning all
[22:43:32] <wikibugs_>	 10Analytics, 06Research-and-Data: Hash IPs on webrequest table - https://phabricator.wikimedia.org/T150545#2812585 (10leila) Update: I've sent emails to research-internal, Legal, wmfproduct, and Ops with the following 4 questions: 1) ​Have you (or someone you have been the point of contact for) used raw IP add...
[23:41:53] <wikibugs_>	 10Analytics, 06Research-and-Data: Hash IPs on webrequest table - https://phabricator.wikimedia.org/T150545#2789216 (10bd808) If I understand the goal of this correctly (reducing exposure of IP addresses in the face of a data breach), then I'm not actually convinced that the problem is trivially solvable. If yo...