[00:06:27] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2262211 (kaldari) The meeting with Microsoft was a mixed bag. They basically said that th...
[00:10:24] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2262214 (Earwig) Re point #2, can we argue that CSB is user-initiated on the principle th...
[02:21:32] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2262353 (kaldari) @Tnegrin: Yes, we are still considering Google. If nothing pans out wit...
[05:25:54] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2262499 (Josve05a) >>! In T125459#2262353, @kaldari wrote: > @Tnegrin: Yes, we are still...
[07:32:48] <wikibugs>	 Analytics-General-or-Unknown: gu.wikiquote.org URL Do Not work - https://phabricator.wikimedia.org/T134343#2262627 (YmKavishwar)
[07:33:53] <wikibugs>	 Analytics-General-or-Unknown: gu.wikiquote.org URL Do Not work - https://phabricator.wikimedia.org/T134343#2262640 (YmKavishwar) p:Triage>High
[08:26:07] <elukey>	 o/
[08:26:15] <joal>	 Hi elukey :)
[08:26:24] <joal>	 review morning again for me
[08:26:30] <elukey>	 I was about to say the same
[08:26:36] * joal is trying hard to write correct english :)
[08:26:41] <elukey>	 I have been writing for an hour and still need to write stuff
[08:26:44] <elukey>	 ahhahahaah
[08:27:31] <elukey>	 Today I've read what I wrote yesterday and I thought "did a 5 years old child hacked my laptop tonight?"
[08:27:43] <joal>	 huhuhu :)
[08:29:28] <elukey>	 stats.wikimedia.org, datasets.w.o, etc.. are all working with HTTP/2
[08:30:08] <elukey>	 nginx has been updated yesterday, really really great
[09:38:28] <grrrit-wm>	 (CR) Joal: "Weeeeeird ! To me the separator still shows up as a space !" [analytics/refinery] - https://gerrit.wikimedia.org/r/286672 (https://phabricator.wikimedia.org/T134279) (owner: Nuria)
[11:25:36] <wikibugs>	 Analytics-Kanban, DC-Ops, EventBus, MediaWiki-Cache, and 5 others: setup kafka2001 & kafka2002 - https://phabricator.wikimedia.org/T121558#2263404 (elukey) LVS configuration set up on lvs200[36]:  curl http://eventbus.svc.codfw.wmnet:8085/v1/topics  Pending verification, but the work should be do...
[11:54:00] * elukey lunch + afk for a bit
[11:56:34] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Create basic/high-level Kibana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2263484 (Qgil)
[11:56:39] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2263482 (Qgil)
[11:56:59] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Mismatch between numbers for code merges per organization - https://phabricator.wikimedia.org/T129910#2263494 (Qgil)
[11:57:05] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Play with Bitergia's Kabana UI (which might potential replace our current UI on korma.wmflabs.org) - https://phabricator.wikimedia.org/T127078#2263498 (Qgil)
[11:57:24] <wikibugs>	 Analytics-Tech-community-metrics, Differential, Developer-Relations (Apr-Jun-2016): Make MetricsGrimoire/korma support gathering Code Review statistics from Phabricator's Differential - https://phabricator.wikimedia.org/T118753#2263506 (Qgil)
[13:01:39] <wikibugs>	 Analytics-Kanban, DC-Ops, EventBus, MediaWiki-Cache, and 5 others: setup kafka2001 & kafka2002 - https://phabricator.wikimedia.org/T121558#2263744 (elukey)
[13:03:12] <wikibugs>	 Analytics-Kanban, Operations, Patch-For-Review: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#2263748 (elukey)
[13:03:29] <wikibugs>	 Analytics-Kanban, Operations, Patch-For-Review: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#798408 (elukey)
[15:17:56] <mforns>	 elukey, ottomata, yt?
[15:18:24] <ottomata>	 jaa hi
[15:18:34] <mforns>	 hi ottomata :]
[15:19:11] <mforns>	 can you help me troubleshoot why vital-signs.wmflabs.org and browser-reports.wmflabs.org are 502ing?
[15:19:32] <ottomata>	 surre
[15:19:36] <mforns>	 oh ottomata... it seems not any more...
[15:19:42] <ottomata>	 haha
[15:19:44] <mforns>	 ok...
[15:19:45] <ottomata>	 you are welcome!
[15:19:48] <mforns>	 hehe
[15:20:06] <joal>	 ottomata: you didn't even to `hmmm` to make it work again !
[15:20:16] * joal is impressed
[15:20:25] <ottomata>	 haha
[15:20:31] <ottomata>	 my power grows...
[15:21:00] <mforns>	 xDDDD
[15:21:24] <mforns>	 I think it's the sole mention of his name that does the thing
[15:30:11] <joal>	 mforns: When Chuck Norris invented computers, ottomata had already sent him two emails
[15:30:20] <mforns>	 xDDDDD
[15:33:48] <elukey>	 nuria_: https://phabricator.wikimedia.org/T96848
[15:34:12] <elukey>	 joal: hahahahah
[15:47:17] <nuria_>	 elukey: some respect for ema: https://github.com/wikimedia/operations-puppet/blob/production/modules/tlsproxy/files/utils/h2_spdy_stats.stp
[15:47:22] <nuria_>	 bow
[15:48:00] <elukey>	 nuria_: I know, ema++ :)
[15:48:12] <nuria_>	 elukey: indeed.
[15:49:10] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, scap, Patch-For-Review, Scap3 (Scap3-Adoption-Phase1): Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#2264457 (Ottomata) Yes, it is mostly performance team for the deployment to hafnium. See T131977
[15:49:55] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, scap, Patch-For-Review, Scap3 (Scap3-Adoption-Phase1): Use scap3 to deploy eventlogging/eventlogging - https://phabricator.wikimedia.org/T118772#2264461 (Ottomata) Actually T110903 is more descriptive.
[15:52:00] <elukey>	 ottomata: do you have a minute?
[15:52:22] <ottomata>	 yup
[15:52:26] <ottomata>	 what's up?
[15:53:03] <elukey>	 I sent https://gerrit.wikimedia.org/r/#/c/286845/ and  tested in the puppet compiler, should be good, but before proceeding I'd need a basic step: where can I find the analytics1003 root db pass?
[15:53:13] <elukey>	 I need to create hue's schema :)
[15:53:19] <elukey>	 didn't find it in the private repo
[15:55:49] <ottomata>	 elukey: , no root pass
[15:55:54] <ottomata>	 you just need to be root user
[15:56:01] <ottomata>	 sudo mysql
[15:56:03] <ottomata>	 will do it
[15:56:45] * elukey is confused but will try
[15:57:08] <elukey>	 joal: ottomata scares me sometimes
[15:57:14] <elukey>	 :P
[15:57:26] <elukey>	 ahahhahaha it wooork
[15:57:28] <joal>	 huhu
[15:57:37] <elukey>	 now DROP TABLE
[15:57:39] <elukey>	 err no
[16:00:33] <nuria_>	 a-team: standddupppp
[16:01:16] <nuria_>	 mforns,  joal , elukey , madhuvishy : standuppppp
[16:01:22] <mforns>	 nuria_, going
[16:01:27] <joal>	 nuria_: in meeting, sent email
[16:01:52] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Fix Dashiki's metrics-by-project breakdown - https://phabricator.wikimedia.org/T133944#2264529 (Nuria)
[16:01:58] <elukey>	 mforns: get up stand up, stand up for your rights
[16:02:01] <elukey>	 :P
[16:02:10] <mforns>	 xDDD
[16:03:38] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Client values inbound in X-analytics header  (pageview and preview) are reflected in outbound X-Analytics on varnish - https://phabricator.wikimedia.org/T133204#2264537 (Nuria)
[16:06:00] <madhuvishy>	 i cannot get into batcave
[16:15:18] <wikibugs>	 Analytics-Kanban, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2264602 (Ottomata)
[16:15:31] <wikibugs>	 Analytics-Kanban, DNS, Operations, Traffic: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2197243 (Ottomata) a:Ottomata
[16:30:34] <mforns>	 nuria_, do you want to look at the dashiki bug?
[16:30:44] <nuria_>	 mforns: yes, let's go back to batcave
[16:30:51] <mforns>	 ok
[16:38:57] <travis-ci>	 wikimedia/mediawiki-extensions-EventLogging#553 (REL1_27 - 67dedc9 : Translation updater bot): The build failed.
[16:38:57] <travis-ci>	 Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/compare/REL1_27
[16:38:57] <travis-ci>	 Build details : https://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/127843551
[16:40:12] <ottomata>	 elukey:  mobrovac fyi i'm running a throughput test in codfw
[16:43:13] <elukey>	 ottomata: I am going to stop hue, dump the db && copy the sqlite as prep steps for the migration
[16:43:42] <elukey>	 !log stopping hue on analytics1027 to dump its databse
[16:44:11] <ottomata>	 ok cool
[17:00:03] <elukey>	 ottomata: just saw your comments on the hue code review, I thought about adding everything to hiera too. Just amended it, puppet compiler looks good https://puppet-compiler.wmflabs.org/2666/analytics1027.eqiad.wmnet/
[17:02:46] <elukey>	 anything against me merging?
[17:03:15] <ottomata>	 nope!  +1  this will break hue until it is done though, ja?
[17:03:36] <elukey>	 yeppa
[17:04:01] <elukey>	 but I am going to migrate the db now
[17:06:09] <grrrit-wm>	 (PS2) Mforns: Fix issues in metrics-by-project breakdown patterns [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[17:08:35] <ottomata>	 ok
[17:09:36] <elukey>	 ah snap
[17:16:30] <elukey>	 !log migrating the hue database
[17:20:28] <grrrit-wm>	 (PS3) Mforns: Fix issues in metrics-by-project breakdown patterns [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[17:22:59] <grrrit-wm>	 (PS4) Mforns: Fix issues in metrics-by-project breakdown patterns [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[17:25:23] <mforns>	 nuria_, it's ready for review: https://gerrit.wikimedia.org/r/#/c/286755
[17:25:36] <nuria_>	 mforns: on meeting will talk in abit
[17:25:43] <mforns>	 sure
[17:28:56] <elukey>	 ottomata: migration completed!
[17:29:05] <elukey>	 do you mind to double check that everything is working fine?
[17:31:19] <ottomata>	 elukey:  i think it looks good!
[17:31:19] <ottomata>	 nice!
[17:31:29] <elukey>	 \o/
[17:31:55] <elukey>	 I had to follow the procedure for InnoDB though
[17:32:06] <elukey>	 because there were some foreign keys to clear
[17:32:09] <elukey>	 and then restore
[17:32:11] <elukey>	 a bit weird
[17:33:08] <ottomata>	 hm
[17:33:33] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Use MySQL as Hue data backend store - https://phabricator.wikimedia.org/T127990#2264972 (elukey) Executed the migration steps, everything worked nicely and hue looks good now.
[17:35:05] <elukey>	 ottomata: I suspect that the database engine should have been set to "mariadb"
[17:35:08] <elukey>	 not mysql
[17:36:15] <elukey>	 mmmm no the docs says mysql
[17:36:18] <elukey>	 http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_ig_hue_database.html#concept_id1_wkj_zj__cdh_ig_hue_database_mariadb
[17:36:45] <elukey>	 anyhooowww
[17:37:00] <elukey>	 I am going to log-off and then double check later that everything is fine
[17:37:03] <elukey>	 byeee a-team!
[17:39:28] <ottomata>	 ook laters!
[17:39:39] <mforns>	 bye elukey !
[17:40:40] <ottomata>	 a-team	 fyi, ellery just noticed that spark jobs aren't being accepted by yarn
[17:40:44] <ottomata>	 trying to troubleshoot
[17:40:53] <ottomata>	 have we noticed any problems with prod spark jobs?
[17:41:04] <madhuvishy>	 ottomata: oh at all? i thought just temp resource allocation issue
[17:41:13] <ottomata>	 i mean, maybe it is
[17:41:20] <ottomata>	 but ellery has noticed since yesterday
[17:41:27] <madhuvishy>	 hmmm
[17:41:27] <ottomata>	 hm, will try in higher level queue
[17:52:20] <ottomata>	 yeah madhuvishy it never gets accepted
[17:52:34] <madhuvishy>	 ottomata: hmmm
[17:52:37] <madhuvishy>	 weird
[17:53:08] <ottomata>	 yeah not good, i assume oozie submitted jobs are going to fail too :/
[17:53:15] <ottomata>	 trying to figure out why
[17:53:20] <ottomata>	 i find errors in logs, but nothing really useful
[17:53:28] <madhuvishy>	 checking prod jobs
[17:54:44] <madhuvishy>	 ottomata: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0000116-160420145651441-oozie-oozi-C/
[17:55:04] <madhuvishy>	 everything upto an hour ago seems to have run
[17:55:42] <madhuvishy>	 not sure about the current job though
[17:55:50] <ottomata>	 hm
[17:56:52] <madhuvishy>	 ottomata: http://localhost:8088/proxy/application_1458129362008_163704/
[17:57:00] <madhuvishy>	 if you're proxying already
[17:57:02] <madhuvishy>	 it's running
[17:57:23] <ottomata>	 HMM madhuvishy!
[17:57:29] <ottomata>	 --deploy-mode cluster works
[17:57:36] <madhuvishy>	 ottomata: oh interesting
[17:57:42] <madhuvishy>	 but not local?
[17:58:07] <ottomata>	 not client, yeah
[17:58:12] <madhuvishy>	 hmmm
[17:58:20] <ottomata>	 double checking
[17:58:35] <ottomata>	 ewulczyn_____:  oh you are here
[17:58:39] <ottomata>	 so many underscores!
[17:59:26] <ottomata>	 ok, so prod jobs won't die
[17:59:28] <ottomata>	 hm
[18:00:36] <madhuvishy>	 yeah prod jobs look fine
[18:04:42] <ewulczyn_____>	 I tried  --deploy-mode cluster and the job still did not get going
[18:05:07] <ottomata>	 ewulczyn_____:  you are doing spark-submit?
[18:05:12] <ottomata>	 can you paste full command to launch job?
[18:05:13] <ewulczyn_____>	 yes
[18:05:39] <ewulczyn_____>	 spark-submit \
[18:05:39] <ewulczyn_____>	     --driver-memory 5g \
[18:05:39] <ewulczyn_____>	     --master yarn \
[18:05:39] <ewulczyn_____>	     --deploy-mode cluster \
[18:05:39] <ewulczyn_____>	     --num-executors 4 \
[18:05:39] <ewulczyn_____>	     --executor-memory 5g \
[18:05:39] <ewulczyn_____>	     --executor-cores 4 \
[18:05:40] <ewulczyn_____>	     --priority
[18:05:40] <ewulczyn_____>	 hash_trace_ips.py \
[18:05:41] <ewulczyn_____>	     --name test
[18:07:09] <madhuvishy>	 ewulczyn_____: hmmm woah 5g driver memory?
[18:10:21] <ottomata>	 hmm, i'm trying with ewulczyn_____'s stuff, haven't tried a python example
[18:10:27] <ottomata>	 cluster seems also to not work for python
[18:11:10] <ottomata>	 event without 5g driver
[18:11:18] <ottomata>	 i'm runing a simpler python spark job with spark-submit
[18:11:20] <ottomata>	 not working :(
[18:12:48] <grrrit-wm>	 (PS8) Amire80: Add sorted errors [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228
[18:13:53] <grrrit-wm>	 (CR) Amire80: Add sorted errors (1 comment) [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228 (owner: Amire80)
[18:15:22] <ottomata>	 bahhhh
[18:15:25] <ottomata>	 OH
[18:15:28] <ottomata>	 ewulczyn_____:  i take it back
[18:15:33] <ottomata>	 i got it to run in cluster mode
[18:15:37] <ottomata>	 my script was just funky
[18:15:53] <ottomata>	 hm
[18:16:02] <ottomata>	 so, it isn't a python problem
[18:16:03] <ottomata>	 hm
[18:16:15] <madhuvishy>	 ottomata: may be it cant get 5g driver memory
[18:16:44] <ottomata>	 maybe, but it doesn't work in client mode either
[18:16:46] <ottomata>	 which it should
[18:16:48] <madhuvishy>	 right
[18:17:48] <ottomata>	 when i couldn't get it to run before in cluster mode, my python script had a syntax error in it
[18:17:57] <ottomata>	 which did not show up in cluster mode
[18:18:01] <ottomata>	 but when I ran in client it showed me
[18:18:56] <ottomata>	 madhuvishy:  i got my cluster mode script to run with 5g driver memory, trying the same settings ellery had
[18:18:57] <ottomata>	 ...
[18:19:32] <ewulczyn_____>	 note it should be --queue priority not --priority
[18:19:37] <ottomata>	 aye
[18:19:40] <ottomata>	 i'm running in default queue right now
[18:19:57] <ottomata>	 ja ellery i can make both python and scala run in cluster mode
[18:20:04] <ottomata>	 you sure yours doesn't ever run in cluster mode?
[18:20:26] <ottomata>	 that's not your full command, is it  your script takes args
[18:20:33] <ottomata>	 can you paste full command with absolute paths in a gist or something?
[18:20:38] <ottomata>	 or pastebin?
[18:22:21] <ewulczyn_____>	 the job oscillates between state:ACCEPTED and state: RUNNING
[18:22:36] <ottomata>	 oscillates?
[18:22:42] <ottomata>	 weird
[18:23:08] <ewulczyn_____>	 you were saying that if there is abug in my code and I run in cluster mode, I would not see the stacktrace?
[18:23:22] <ottomata>	 ewulczyn_____:  ewulczyn_____ not sure, i just noticed that when there was one in mine
[18:23:31] <ottomata>	 can you run it in local mode on a small dataset?
[18:24:03] <ottomata>	 also, ewulczyn_____  please paste or email me a full command I can run with your script (with absolute paths so I can find it :) ) i want to try it
[18:25:21] <ewulczyn_____>	 spark-submit \
[18:25:21] <ewulczyn_____>	     --driver-memory 1g \
[18:25:21] <ewulczyn_____>	     --master yarn \
[18:25:21] <ewulczyn_____>	     --deploy-mode cluster \
[18:25:21] <ewulczyn_____>	     --num-executors 4 \
[18:25:21] <ewulczyn_____>	     --executor-memory 5g \
[18:25:21] <ewulczyn_____>	     --executor-cores 4 \
[18:25:22] <ewulczyn_____>	     --queue priority \
[18:25:22] <ewulczyn_____>	     --name test \
[18:25:23] <ewulczyn_____>	     --key dummy_key
[18:25:40] <ewulczyn_____>	 I'll email it
[18:26:50] <ewulczyn_____>	 sent
[18:28:19] <madhuvishy>	 ewulczyn_____: hmmm i've seen the oscillation if it runs out of memory and keeps rescheduling the job
[18:28:31] <madhuvishy>	 ottomata: ^
[18:30:24] <ottomata>	 hm
[18:30:41] <ottomata>	 yeah this does look like a different problem.  at least, a diffferent exit ode
[18:30:42] <ottomata>	 hm
[18:33:07] <ottomata>	 yeah if it get sto running and then back to accepted it means that something is failing during the job execution , pretty sure
[18:33:17] <madhuvishy>	 yup
[18:37:46] <ottomata>	 ewulczyn_____:  for yours
[18:37:47] <ottomata>	 i get
[18:37:51] <ottomata>	 : org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://analytics-hadoop/user/ellery/user/hive/warehouse/traces.db/test
[18:38:13] <ottomata>	 hmm, wait is that my fault?
[18:38:14] <ottomata>	 hang on
[18:38:34] <ottomata>	 no, yeah
[18:38:35] <ottomata>	 ellery
[18:38:37] <ottomata>	 your script has a bug! :)
[18:38:41] <ottomata>	     input_dir = 'user/hive/warehouse/traces.db/' + args.name
[18:40:41] <ottomata>	 ja ewulczyn_____ i was able to run your job in cluster mode succesfully
[18:41:41] <ewulczyn_____>	 yup, me to, but not in client mode
[18:42:23] <ewulczyn_____>	 sorry for adding a red herring
[18:42:57] <ottomata>	 yeah client mode is still a problem
[18:43:02] <ottomata>	 for spark-submit i'd usually just use cluster anyway
[18:43:09] <ottomata>	 but, yeah for shells you can't do that
[18:50:33] <ottomata>	 ewulczyn_____:  this is def a problem
[18:50:38] <ottomata>	 are you unblocked for now?
[18:50:40] <ottomata>	 i'm going to file a ticket
[18:50:42] <ottomata>	 for the client mode stuff
[18:52:24] <ewulczyn_____>	 hmm, can you give me some advice on how to debug my scripts in cluster mode?
[18:52:54] <ewulczyn_____>	 client mode gives me the error
[18:54:51] <ewulczyn_____>	 ^ ottomata
[18:55:40] <ottomata>	 ewulczyn_____:  can you run them in local mode instead of yarn?
[18:55:55] <ottomata>	 --master local
[18:56:18] <ottomata>	 and make sure you use a small local input and and output, with small values for executors and drivers and memory
[18:56:20] <ottomata>	 probably defaults are ok
[18:56:26] <ewulczyn_____>	 also, I'm used to running using python packages that are not installed on the lcuster in the driver
[18:56:46] <ottomata>	 hm
[18:56:53] <ottomata>	 you can ship them with the job somehow, ja?
[18:57:04] <ottomata>	 --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place
[18:57:05] <ottomata>	                               on the PYTHONPATH for Python apps.
[18:58:26] <ewulczyn_____>	 I might just work on something else until cient mode os working again. I've tried to ship packages before with limited success
[18:59:49] <ottomata>	 ewulczyn_____:  what packages are you missing?
[18:59:56] <ottomata>	 a small list of debianized packages? :D
[18:59:59] <ottomata>	 perhaps?
[19:00:01] <ottomata>	 probalby not..
[19:00:30] <ewulczyn_____>	 :) no, probably not.
[19:00:51] <ottomata>	 ya this is pretty annoying
[19:00:59] <ewulczyn_____>	 its ok, I can work on other stuff for a while
[19:01:26] <ewulczyn_____>	 any ideas on what happened to client mode?
[19:02:07] <wikibugs>	 Analytics, Analytics-Cluster: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265192 (Ottomata)
[19:02:10] <ottomata>	 ewulczyn_____:  no idea
[19:02:17] <ottomata>	 are you sure this only started a day ago?
[19:03:02] <ewulczyn_____>	 no, but it definitely worked last week
[19:03:25] <ottomata>	 hm ok
[19:03:52] <wikibugs>	 Analytics, Analytics-Cluster: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265219 (Ottomata) We aren't sure when or why this started happening, but Ellery says he was able to use client mode just fine last week.
[19:17:31] <grrrit-wm>	 (PS5) Nuria: Fix issues in metrics-by-project breakdown patterns [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944)
[19:19:53] <grrrit-wm>	 (CR) Nuria: Fix issues in metrics-by-project breakdown patterns (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[19:22:58] <ottomata>	 ewulczyn_____:  i'd recommend running in local mode to debug your stuff, as for shipping things, and if you need help shipping python stuff lemme know, maybe we can figure it out
[19:23:17] <ottomata>	 i gotta work on some other stuff before my work day finishes, not sure when we will figure out the client mode thing
[19:23:20] <ottomata>	 hopefully not too long
[19:23:24] <ottomata>	 keep poking at us about it
[19:23:27] <joal>	 ottomata, ewulczyn_____ : Was backfilling onto the thing
[19:23:51] <ottomata>	 ?
[19:24:02] <joal>	 ottomata: have we changed anything recently on the cluster that might have made a working stuff go to non-working ?
[19:24:15] <ottomata>	 not that i can remember
[19:24:22] <joal>	 ottomata: backlogging IRC sorry :)
[19:24:25] <ottomata>	 ah
[19:24:29] <joal>	 ottomata: any java update?
[19:24:32] <ottomata>	 elukey:  switched hue stuff around today
[19:24:40] <ottomata>	 but that should be unrelated
[19:24:42] <joal>	 hmm
[19:24:43] <ottomata>	 also not that i know fo
[19:25:10] <joal>	 I recall a global restart for java update from elukey, but I think it was before last week
[19:25:55] <ottomata>	 and ewulczyn_____ said this didn't work for him yesterday too
[19:25:58] <ottomata>	 and hue mysql stuff just ahppened today
[19:26:05] <joal>	 right
[19:28:00] <elukey>	 aloha :)
[19:28:10] <joal>	 Eya elukey
[19:28:27] <joal>	 experiencing weird spark behavior since a few days
[19:28:41] <joal>	 Have we done anything to java on the cluster last weeke ?
[19:28:43] <ottomata>	 elukey:  sorry for the ping!
[19:28:45] <joal>	 elukey: --^
[19:28:53] <elukey>	 no worries! I was just checking hue :P
[19:28:56] <wikibugs>	 Analytics, Commons, Multimedia, Wikidata, and 3 others: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426#2265305 (Yurik)
[19:29:30] <elukey>	 so Moritz has rolled out some upgrades to non java daemons in the past two days (ssh, ntp, etc..) due to openssl upgrades
[19:30:19] <elukey>	 but that's it, not much that could explain spark failures
[19:30:30] <elukey>	 what are the symptoms?
[19:30:43] <elukey>	 I didn't investigate after the email sorry :(
[19:31:28] <joal>	 np elukey : Spark doesn't stop in client mode mode anymore
[19:31:30] <ottomata>	 elukey:  https://phabricator.wikimedia.org/T134422
[19:32:07] <wikibugs>	 Analytics, Analytics-Cluster: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265321 (Ottomata)
[19:32:08] <joal>	 elukey: client mode means that there is a local app that gets results back from the application master
[19:32:44] <ottomata>	 ja elukey don't worry about it now it is late there!
[19:33:09] <elukey>	 all right, will try to follow up tomorrow :)
[19:33:14] <elukey>	 sorry!
[19:33:20] <joal>	 Ohhhhhh, I recall now: there have been changes on the firewall for stat1002 !!!!
[19:33:26] <joal>	 Must be that !
[19:33:53] <elukey>	 !!!
[19:34:27] <elukey>	 29/04 - 07:54 moritzm: enabled base::firewall on stat1002
[19:34:48] <elukey>	 joal: --^
[19:34:59] <elukey>	 that is last Friday
[19:35:04] * joal loves when its brain works :)
[19:35:15] <ottomata>	 !!!!!
[19:35:17] <ottomata>	 !!!
[19:35:19] <ottomata>	 hmmmm
[19:35:23] <ottomata>	 hm
[19:35:25] <ottomata>	 YES
[19:35:41] <ottomata>	 because it starts a listener on a port, and it isn't going to know where the app master is gonna be
[19:35:46] <ottomata>	 so the app master has to contact the local client
[19:35:51] <ottomata>	 HMMM
[19:36:15] <elukey>	 all right, good work :)
[19:36:20] <elukey>	 talk with you tomorrow!
[19:36:21] <elukey>	 byyeeee
[19:36:37] <joal>	 Bye elukey !
[19:36:45] <joal>	 Thanks having been my ruberduck !
[19:37:11] <joal>	 ottomata: I* really think that could be it
[19:41:03] <ottomata>	 yeah i htink so too
[19:41:06] <ottomata>	 here i will stop ferm for a sec
[19:41:07] <ottomata>	 and try
[19:41:13] <ottomata>	 (am doing like 5 thigns at once...)
[19:41:55] <joal>	 huhuh ottomata :)
[19:42:23] <ottomata>	 yup, totally it!
[19:42:42] <ottomata>	 will try to make a patch to fix shortly...not really sure what i'm gonna do yet
[19:42:45] <ottomata>	 dealing with analytics.wm.org first
[19:54:03] <ottomata>	 thanks for thinking of that joal
[19:57:58] <joal>	 ottomata: no prob :)
[19:58:05] <joal>	 ottomata: cool that it was that !
[19:58:51] <joal>	 a-team: logging off for tonight, see you tomorrow
[20:00:02] <wikibugs>	 Analytics, Commons, Multimedia, Wikidata, and 3 others: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426#2265432 (matmarex) Surely this doesn't need all these tags and all these people. Please pay attention when creating subtasks :(
[20:00:17] <ottomata>	 laters joal!
[20:02:46] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, and 4 others: schedule a daily run of ContentTranslation analytics scripts - https://phabricator.wikimedia.org/T122479#2265442 (Amire80)
[20:02:50] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2265439 (Amire80) Resolved>Open @ellery says that I will definitely need stat1002 t...
[20:07:10] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2265466 (Ottomata) Please clarify how you will run these queries. If MySQL, then you only n...
[20:08:13] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2265469 (Krenair) Yes, can we please start talking in terms of groups instead of hosts?
[20:21:01] <wikibugs>	 Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2265512 (Ottomata)
[20:23:21] <wikibugs>	 Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2265532 (Krenair) We need to figure out the correct criteria for wikis to be added to this list. I noticed that there are only 4 entries containing `.wikimedia`, that can't possibly be...
[20:23:26] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2265533 (ellery) @Amire80 needs to run hive queries to count the number times users navigat...
[20:26:56] <wikibugs>	 Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2265540 (Nuria) Just updated wiki: https://wikitech.wikimedia.org/wiki/Add_a_wiki#Analytics  @Krenair: it is correct, only "knowledge public wikis " are on that list
[20:28:51] <wikibugs>	 Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2265548 (Krenair) So non-private, non-wikimania, non-chapter, and none of these? loginwiki outreachwiki qualitywiki strategywiki usabilitywiki
[20:29:56] <ottomata>	 nuria_:  yeehaw!
[20:29:57] <ottomata>	 https://analytics.wikimedia.org/
[20:30:17] <nuria_>	 ottomata: you da ma!
[20:30:19] <nuria_>	 *man
[20:30:23] <ottomata>	 be there 1 min
[20:30:25] <nuria_>	 k
[21:14:33] <wikibugs>	 Analytics-Kanban, Operations, Patch-For-Review: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#2265713 (ezachte) Issues fixed. Thanks, elukey, and others.
[21:20:19] <grrrit-wm>	 (CR) Mforns: Fix issues in metrics-by-project breakdown patterns (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[21:21:00] <grrrit-wm>	 (CR) Mforns: "I tried the newest patch (5) and it has a problem, see comments in patch 4." [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[21:21:44] <mforns>	 nuria_, if yt I looked at the newest patch and it has a problem, I commented in gerrit, but if you want we can chat here or batcave
[21:23:06] <wikibugs>	 Analytics, Analytics-Cluster, Patch-For-Review: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265751 (Ottomata) Ah  ha! Of course, Moritz applied base::firewall a few days ago.  https://gerrit.wikimedia.org/r/287001 should fix.
[21:23:51] <wikibugs>	 Analytics, Analytics-Cluster, Patch-For-Review: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265753 (Ottomata) @MoritzMuehlenhoff
[21:25:47] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265761 (Ottomata) a:Ottomata
[21:26:43] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2265768 (Dzahn) a:Dzahn>None
[21:29:10] <wikibugs>	 Analytics, ContentTranslation-Analytics, MediaWiki-extensions-ContentTranslation, Operations, Ops-Access-Requests: access for amire80 to stat1002.eqiad.wmnet - https://phabricator.wikimedia.org/T122524#2265769 (Dzahn) it will be handled, i'm just giving it back to the pool because access requ...
[21:33:58] <grrrit-wm>	 (CR) Mforns: Fix issues in metrics-by-project breakdown patterns (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[21:41:13] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2265814 (tom29739) >>! In T125459#2262499, @Josve05a wrote: >>>! In T125459#2262353, @kal...
[21:54:58] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Spark yarn in client mode is never moved from ACCEPTED to RUNNING - https://phabricator.wikimedia.org/T134422#2265930 (Ottomata)
[21:55:07] <wikibugs>	 Analytics-Kanban, DNS, Operations, Traffic, Patch-For-Review: Create analytics.wikimedia.org - https://phabricator.wikimedia.org/T132407#2265931 (Ottomata)
[22:02:22] <grrrit-wm>	 (PS6) Mforns: Fix issues in metrics-by-project breakdown patterns [analytics/dashiki] - https://gerrit.wikimedia.org/r/286755 (https://phabricator.wikimedia.org/T133944) (owner: Nuria)
[22:17:56] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2266027 (Tnegrin) It's not the money per se - it's just a budget formality of putting the...
[22:50:31] <nuria_>	 mforns: looking
[22:53:39] <nuria_>	 mforns: ah right, you mention that earlier today, I see that
[22:57:56] <wikibugs>	 Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2266089 (Nuria) @krenair: correct, those are omitted on purpose so their pageviews are not counted. to be clear we still get their requests and we process them we just do not aggregate...
[23:01:25] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Add JJB support for Jenkins Maven Release Plugin {hawk} - https://phabricator.wikimedia.org/T132175#2190696 (hashar) That patch looks entirely fine to me :)  Can you propose it to upstream Gerrit? They are rather friendly and I have approval rights there to help move i...
[23:01:50] <wikibugs>	 Analytics-Kanban, Continuous-Integration-Config, Patch-For-Review: Add JJB support for Jenkins Maven Release Plugin {hawk} - https://phabricator.wikimedia.org/T132175#2266092 (hashar)
[23:01:50] <nuria_>	 mforns: updating
[23:01:54] <mforns>	 nuria_, ok
[23:02:18] <wikibugs>	 Analytics-Kanban, Continuous-Integration-Config, Patch-For-Review: Add JJB support for Jenkins Maven Release Plugin {hawk} - https://phabricator.wikimedia.org/T132175#2190696 (hashar)
[23:02:24] <nuria_>	 mforns: i think it will be easy to not empty breakdowns if metric has not changed on my patch and that makes code a lot simpler no?
[23:02:51] <mforns>	 nuria_, should we store the previous metric?
[23:03:24] <mforns>	 I don't like this very much
[23:03:44] <nuria_>	 mforns: no need, right? we can subscribe to changes we need
[23:05:21] <mforns>	 nuria_, I see, the only change we can subscribe that is close that what we need, is the breakdownState
[23:05:46] <mforns>	 nuria_, but the breakdown state also changes when a breakdown option is toggled
[23:06:37] <mforns>	 nuria_, we can not clean up the patterns when a breakdown option is toggled, otherwise we'll see the problem we were talking about
[23:07:08] <wikibugs>	 Analytics: Augment 'Add a Wiki' process to include Analytics pageview whitelist - https://phabricator.wikimedia.org/T134433#2266095 (Krenair) Okay. I think this is done and we can resolve it, unless anyone disagrees?
[23:07:13] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, Community-Tech-Sprint: Investigation: Can we find a new search API for CorenSearchBot and Copyvio Detector tool? - https://phabricator.wikimedia.org/T125459#2266096 (Majora) >>! In T125459#2265814, @tom29739 wrote: >>>! In T125459#2262499, @Josve...
[23:16:07] <wikibugs>	 Analytics, Commons, Multimedia, Wikidata, and 3 others: Review shared data namespace (tabular data) implementation - https://phabricator.wikimedia.org/T134426#2266122 (Yurik) My apologies to anyone who is not interested in the implementation review of this feature - please unsubscribe. I am hopin...