[00:35:29] <wikibugs>	 (03PS2) 10Sturmkrahe: Darken "subdued" text and lighten footer text for WCAG AA compliance [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419958 (https://phabricator.wikimedia.org/T185533)
[00:49:48] <wikibugs>	 (03CR) 10Milimetric: "Nice, looks good overall.  One general question about media queries that we should talk about, and otherwise just nits." (0326 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345) (owner: 10Fdans)
[00:51:10] <milimetric>	 ebernhardson: I very vaguely remember we serve those static files from somewhere... lemme dig around but it'll probably have to wait for opsy folks tomorrow
[00:55:53] <ebernhardson>	 milimetric: its no worry, i can get the same from hue i just tend to use yarn.wikimedia.org
[00:55:59] <ebernhardson>	 hue is slow :P
[00:57:54] <milimetric>	 oh man, is it ever
[00:58:00] <milimetric>	 but it did used to be a lot slower
[01:37:52] <HaeB>	 milimetric: i use https://yarn.wikimedia.org/cluster/scheduler too :/
[06:59:15] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#4074944 (10Joe)
[07:58:56] <wikibugs>	 10Analytics, 10DBA, 10EventBus, 10MediaWiki-Database, and 7 others: High (2-3x) write and connection load on enwiki databases - https://phabricator.wikimedia.org/T189204#4075020 (10jcrespo) From my point of view, the problem has gone: https://grafana-admin.wikimedia.org/dashboard/db/mysql-aggregated?panelI...
[08:09:51] <wikibugs>	 10Analytics, 10DBA, 10EventBus, 10MediaWiki-Database, and 7 others: High (2-3x) write and connection load on enwiki databases - https://phabricator.wikimedia.org/T189204#4075026 (10jcrespo) I would even dare to say the baseline is lower: https://grafana.wikimedia.org/dashboard/db/jobqueue-eventbus?orgId=1&...
[08:25:54] <joal>	 Hi dsaez - Didn't manage to get to you yesterday :)
[08:45:34] <wikibugs>	 10Analytics, 10User-Elukey: latest varnishkafka fails to build on Debian - https://phabricator.wikimedia.org/T186250#4075111 (10elukey) >>! In T186250#4061550, @Jrdnch wrote: > I can submit a patch. Thanks for the man page, that's helpful. >  > My patch would probably just remove the `_BSD_SOURCE` macro, since...
[09:39:18] <wikibugs>	 (03PS10) 10Fdans: Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345)
[09:39:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345) (owner: 10Fdans)
[09:44:03] <wikibugs>	 (03PS11) 10Fdans: Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345)
[09:52:19] <wikibugs>	 (03CR) 10Fdans: "> This change does not render for me on desktop or mobile, dashboard" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[10:12:18] <elukey>	 this morning I started like this https://gerrit.wikimedia.org/r/#/c/421490/
[10:12:21] <elukey>	 not sure why :D
[10:14:44] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10User-Elukey: Produce webrequests from varnishkafka to Kafka with Kafka message timestamp set to configurable content field - https://phabricator.wikimedia.org/T166833#4075219 (10elukey) Before even starting it would be awesome to merge https://gerrit.wikimedia.org/r/#/c/42...
[11:06:42] <elukey>	 a-team: going to force a manual failover from an1001 to an1002 for jvm restarts
[11:07:09] <joal>	 Hi elukey - Watching with you :)
[11:14:33] <wikibugs>	 (03PS12) 10Fdans: Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345)
[11:15:38] <elukey>	 1001 done, daemons up and stable
[11:15:41] <elukey>	 proceeding with 1002
[11:19:08] <elukey>	 all done :)
[11:20:39] <elukey>	 this morning the yarn ui is weird for me "This page works best with javascript enabled."
[11:20:56] <joal>	 elukey: Erik noticed that yesterday as well
[11:21:01] <joal>	 It is weird as well for me
[11:21:32] <joal>	 elukey: Could it be that some static files used by yarn are not served anymore by our front-end proxy?
[11:23:00] <elukey>	 yeah I was checking the same, the jquery static files seems not available anymore
[11:23:08] <joal>	 Meh
[11:23:19] <elukey>	 https://yarn.wikimedia.org/static/jquery/jquery-1.8.2.min.js -> 404
[11:25:43] <elukey>	 in theory it is jetty saying "I don't have them"
[11:25:50] <elukey>	 so they should be on an1001 right?
[11:26:05] <joal>	 I assume so elukey ?
[11:27:52] <fdans>	 elukey: wanna delete sum data?
[11:30:24] <elukey>	 fdans: after lunch would be better! :)
[11:44:32] <elukey>	 joal: I have no idea where those js files are
[11:44:41] <elukey>	 I can find them for HDFS
[11:44:48] <elukey>	 but not for yarn on an1001
[11:44:54] <elukey>	 maybe they are in the jars?
[11:45:41] <joal>	 Possible elukey!
[11:55:17] <elukey>	 so joal it started yesterday right?
[11:55:23] <joal>	 I think so elukey y
[11:55:32] <joal>	 Have we done anthing yesterday?
[11:55:39] <elukey>	 not that I know of
[11:56:50] <elukey>	 except deploy the new openjdk-8
[11:59:09] <milimetric>	 btw, WOW!!! jquery 1.8?!!!!  2012 called and they want their JS back
[11:59:43] <joal>	 milimetric: You've seen yarn UI - I think even 2012 doesn't want to know about it
[12:00:22] <milimetric>	 it's so funny seeing such an old version of jquery
[12:01:32] <elukey>	 joal: yarn UI is also not displaying any job
[12:01:47] <joal>	 elukey: works for me
[12:02:14] <milimetric>	 elukey: I'm reading the yarn source to figure out how/where they deploy this thing
[12:02:33] <elukey>	 milimetric: thanks :)
[12:02:40] <elukey>	 joal: so you see jobs in RUNNING? 
[12:02:53] <joal>	 elukey: I look at the scheduler
[12:03:04] <joal>	 elukey: I never look at the RUNNING page
[12:04:28] <elukey>	 mmmm
[12:04:41] <elukey>	 if I grep in yarn's logs on an1001 for jquery I can see this 
[12:04:42] <elukey>	 INFO org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is accessing unchecked http://analytics1054.eqiad.wmnet:41117/static/jquery/jquery-1.8.2.min.js which is the app master GUI of application_1520532368078_47226 owned by ironholds
[12:04:47] <elukey>	 but it makes 0 sense
[12:05:56] <elukey>	 probably not related at all
[12:06:44] <elukey>	 joal: out of curiosity, in the scheduler page there is also a way to check the various application_* running or do you use the cli?
[12:07:03] <milimetric>	 elukey: uh, got lost, is there a cloudera-specific version of yarn or can I look at the generic hadoop code?
[12:07:49] <elukey>	 milimetric: no cloudera specific available, I was looking into https://github.com/apache/hadoop/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui
[12:08:28] <elukey>	 but not sure if it is the right one
[12:08:30] <milimetric>	 elukey: yeah but that's trunk, so it looks like they're using jquery 2.1 and a different build
[12:08:35] <elukey>	 yeah 
[12:08:58] <elukey>	 we should use 2.6.0 modified IIRC
[12:09:08] <milimetric>	 I see from here https://yarn.wikimedia.org/cluster/cluster that we're using 2.6.0, but is that the cdh version or does it also correspond to hadoop 2.6? https://github.com/apache/hadoop/tree/release-2.6.3
[12:09:42] <elukey>	 I think that is a cdh version patched
[12:09:53] <elukey>	 so 2.6 with probably some fixes backported
[12:09:56] <milimetric>	 sweet, checking that then
[12:11:29] <elukey>	 going out for a quick lunch!
[12:32:01] <milimetric>	 it's been like 11 years since I looked at Java webapps, but I think this whole thing's being packaged in a webapp jar and serving from this static directory: https://github.com/apache/hadoop/tree/release-2.6.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static
[12:37:46] <milimetric>	 so is this maybe a login problem?  I notice that we're all logged in as "dr. who", right?
[12:38:36] <milimetric>	 or is the login thing broken because there's no js?
[12:40:52] <joal>	 milimetric: to me, it's as if the webapp didn't manage to serve one if its files - Couold be because of permissions?
[12:41:12] <milimetric>	 joal: but upper right do you see "Logged in as dr. who"?
[12:42:52] <milimetric>	 it's not just one, it's all static assets, and they should be served by the webapp from the same jar that the java is bundled in, so yeah, permissions make sense
[12:43:29] <milimetric>	 but were we always all logged in as dr.who or is that something new?
[12:43:58] <milimetric>	 'cause I think Oliver really liked Dr. Who, and so that's Oliver's user?!
[12:46:58] <elukey>	 milimetric: back :)
[12:47:28] <elukey>	 so dr.who is the user that we all share, we have basic auth in front of the UI but we don't do any other auth
[12:48:04] <elukey>	 do you have any idea where https://github.com/apache/hadoop/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui could be on the file system? into a jar?
[12:48:21] <elukey>	 I am trying to figure out where to look on an1001
[12:49:52] <milimetric>	 elukey: I'm not 100% sure but it should be in a hadoop-yarn or hadoop-yarn-common or hadoop-yarn-common-webapp jar or something like that
[12:50:33] <milimetric>	 and I'm not at all sure where these things go on the filesystem, that'd be in the debian package, right?
[12:50:43] <elukey>	 yeah, lemme check
[12:51:37] <joal>	 elukey: I'm having remote-host-change for an1001 - Is that expected?
[12:52:06] <milimetric>	 he did a failover this morning, right? to an1002, so maybe yea
[12:56:27] <elukey>	 joal: do you mean ssh ?
[12:56:31] <joal>	 yessir
[12:56:35] <elukey>	 whattt
[12:57:09] <milimetric>	 elukey: isn't an1002 sitting in place of an1001, just based on what you said this morning
[12:57:16] <milimetric>	 so then doesn't the host-change make sense?
[12:57:31] <milimetric>	 or did the failover thing mean something eles
[12:58:22] <elukey>	 milimetric: so I did a temp failover during the restart but now everything is back to normal.. Joseph is getting a new sha while doing ssh to an1001, that is weird
[12:58:40] <milimetric>	 oh weird
[12:58:51] <milimetric>	 lemme try it to
[13:03:03] <wikibugs>	 (03PS5) 10Joal: [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449)
[13:05:34] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) (owner: 10Joal)
[13:08:22] <elukey>	 all good :)
[13:15:39] <elukey>	 fdans: busy now?
[13:15:49] <fdans>	 elukey: I was just writing you :)
[13:15:51] <fdans>	 cave?
[13:15:58] <elukey>	 sure!
[13:20:41] <joal>	 dsaez: Are we gonna manage to spot each other?
[13:30:18] <milimetric>	 so https://yarn.wikimedia.org/static says 403
[13:30:38] <milimetric>	 which sounds like something's up and working there, to enforce some kind of browsing permissions
[13:31:18] <elukey>	 fdans: it should take ~2h to run I think
[13:32:34] <fdans>	 thank youuu elukey 
[13:33:25] <fdans>	 milimetric: I left a couple questions in the CR you did yesterday, thanks for that :)
[13:33:35] <elukey>	 milimetric: that should be Jetty, embedded in the resource manager
[13:33:35] <fdans>	 other than those I applied your changes and nuria_ 's
[13:34:04] <milimetric>	 looking fdans 
[13:35:39] <milimetric>	 fdans: you maybe didn't reply with the questions?  are they still drafts
[13:36:13] <milimetric>	 elukey: I see, maybe... restart jetty?
[13:36:36] <milimetric>	 is that a separate process with its own logs?
[13:37:06] <elukey>	 nope, I think it is part of the  yarn rm one
[13:37:14] <fdans>	 milimetric: so for some mysterious reason I lost all my drafts/replies
[13:37:33] * fdans is seconds away for throwing his monitor through the window
[13:38:35] <milimetric>	 fdans: you have to first smash it with your keyboard
[13:38:52] <milimetric>	 there's an order to these things, sheesh
[13:39:32] <milimetric>	 (referencing https://makeagif.com/i/lx7mXr of course)
[13:40:09] <fdans>	 this is such bullshit
[13:40:14] <fdans>	 milimetric: rewriting replies
[13:41:06] <milimetric>	 fdans: sorry man, that really sucks, sounds like maybe they rebooted the server or something while you were drafting...  but normally those should be saved in the db - very weird
[13:41:12] <fdans>	 nono
[13:41:14] <fdans>	 they're here
[13:41:46] <fdans>	 but I won't see drafts or signs thereof unless I'm pointing at the exact patch set the code review is at
[13:42:17] <wikibugs>	 (03CR) 10Fdans: Responsive Wikistats 2 UI (0318 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345) (owner: 10Fdans)
[13:42:27] <fdans>	 milimetric: ^
[13:42:29] <milimetric>	 k
[13:44:40] <ottomata>	 elukey:  anything you know going on with namenodes? https://phabricator.wikimedia.org/T188945#4073436
[13:46:50] <elukey>	 ottomata: morning! lemme check
[13:47:05] <elukey>	 we have a problem using the yarn ui after the openjdk upgrade (apparently)
[13:47:29] <elukey>	 standby? Yesterday?
[13:48:18] <elukey>	 let's see logs around that time
[13:48:58] <ottomata>	 yesterday, and he said it happened again?
[13:49:04] <ottomata>	 today?
[13:49:59] <elukey>	 so today I've restarted them due to openjdk-8 upgrades
[13:50:03] <elukey>	 and that makes sense
[13:50:11] <elukey>	 (failover + manual restarts)
[13:50:26] <ottomata>	 ok that makes sense then
[13:52:14] <elukey>	 but yesterday (my evening) is weird
[13:52:39] <elukey>	 ottomata: now we have an interesting issue if you are already caffeinated
[13:52:46] <ottomata>	 sure!
[13:53:04] <elukey>	 https://yarn.wikimedia.org/cluster/apps/RUNNING -> seems that the static js files yields to 404 (apparently) from yesterday
[13:53:36] <elukey>	 dan discovered that they are in /usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.10.0.jar
[13:53:55] <ottomata>	 since yesterday because of reboots?
[13:54:36] <elukey>	 I've done the worker nodes only 
[13:54:40] <elukey>	 (jvm restarts)
[13:55:40] <ottomata>	 oh thought you said you restarted namenodes today, no?
[13:55:47] <wikibugs>	 (03CR) 10Milimetric: Responsive Wikistats 2 UI (034 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345) (owner: 10Fdans)
[13:56:14] <elukey>	 ottomata: yep yesterday (EU afternoon) worker nodes, today (morning EU time) namenodes
[13:56:40] <milimetric>	 fdans: https://gerrit.wikimedia.org/r/#/c/416999/9..12//COMMIT_MSG
[13:56:51] <milimetric>	 (so you can see both my PS9 and PS12 comments)
[13:57:21] <milimetric>	 only thing remaining is to talk about the vue-mq thing
[13:57:26] <elukey>	 milimetric: forgot to tell you that yesterday the jvm was restarted on all the worker nodes, that offer a HTTP API now that Andrew made me think about it
[13:57:43] <milimetric>	 lemme know if you wanna brainbounce
[13:57:48] <ottomata>	 ok
[13:58:01] <ottomata>	 but this req is going to namenode only
[13:58:22] <elukey>	 resource manager right ?
[13:59:13] <ottomata>	 sorry yes
[13:59:34] <elukey>	 anyway, yes, but I was trying to think if the RM UI did any weird request to each of the NM on the worker nodes to get indo
[13:59:37] <elukey>	 *info
[13:59:57] <elukey>	 the first occurrence of the issue seems to be from yesterday (Erik)
[14:00:13] <ottomata>	 v strange
[14:00:46] <fdans>	 mmmmm I'm confuzzled by your suggested approach milimetric 
[14:00:56] <milimetric>	 cave?
[14:01:01] <fdans>	 yep
[14:01:01] <milimetric>	 maybe I'm confuzzled too
[14:02:39] <ottomata>	 elukey:  afaict with tcpdump, the req doesn't leave an01
[14:02:43] <ottomata>	 it isn't proxied to a worker
[14:02:46] <ottomata>	 404 is just returned
[14:04:18] <elukey>	 yep yep you are right, I was only trying to speculate :)
[14:04:50] <ottomata>	 yeah no this is v weird
[14:04:57] <ottomata>	 doesn't really corrleate with your reboots if it started yesterday
[14:05:28] <ottomata>	 wow elukey in googling stuff just found
[14:05:28] <ottomata>	 https://knox.apache.org/
[14:05:30] <ottomata>	 interesting!
[14:06:32] <elukey>	 nice! 
[14:06:45] <elukey>	 https://yarn.wikimedia.org/static/jquery/themes-1.9.1/base/jquery-ui.css returns 200
[14:06:55] <ottomata>	 would allow us to get rid of host specific things an01, an03, etc. and maybe even do ha for web uis?  dunno!
[14:06:58] <ottomata>	 really!?!
[14:07:16] <elukey>	 yeah and it is in the jar
[14:07:21] <ottomata>	 elukey:  did you just randomonly find one that works?
[14:07:31] <elukey>	 yeah :D
[14:07:33] <elukey>	 so
[14:07:34] <elukey>	 elukey@analytics1001:~$ jar tf /usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.10.0.jar | grep /static/jquery/jquery-ui-1.9.1.custom.min.js
[14:07:37] <elukey>	 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
[14:07:39] <elukey>	 webapps/static/jquery/jquery-ui-1.9.1.custom.min.js.gz
[14:07:41] <elukey>	 elukey@analytics1001:~$ jar tf /usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.10.0.jar | grep jquery-ui.css
[14:07:44] <elukey>	 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
[14:07:47] <elukey>	 webapps/static/jquery/themes-1.9.1/base/jquery-ui.css
[14:07:51] <elukey>	 first yield to 404
[14:07:54] <elukey>	 second one to 200
[14:08:05] <elukey>	 note the .gz
[14:08:33] <elukey>	 same thing for 
[14:08:33] <elukey>	 elukey@analytics1001:~$ jar tf /usr/lib/hadoop-yarn/hadoop-yarn-common-2.6.0-cdh5.10.0.jar | grep jquery.dataTables.min.js
[14:08:36] <elukey>	 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
[14:08:39] <elukey>	 (returns 404, .gz)
[14:08:41] <elukey>	 webapps/static/dt-1.9.4/js/jquery.dataTables.min.js.gz
[14:09:30] <ottomata>	  /static/dt-1.9.4/js/jquery.dataTables.min.js.gz works
[14:10:13] <elukey>	 lemme check, might have been confusing urls then
[14:10:34] <elukey>	 it works indeed
[14:10:46] <elukey>	 ah sorry using gz directly in the URL
[14:10:48] <elukey>	 got you
[14:12:03] <elukey>	 I think I got it! 
[14:12:07] <elukey>	 I have a suspicion! 
[14:12:41] <elukey>	 ottomata: https://gerrit.wikimedia.org/r/#/c/419228/
[14:12:46] <wikibugs>	 (03PS13) 10Fdans: Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345)
[14:13:10] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345) (owner: 10Fdans)
[14:13:14] <elukey>	 it happened yesterday
[14:15:38] <ottomata>	 huh....
[14:15:44] <wikibugs>	 (03Merged) 10jenkins-bot: Responsive Wikistats 2 UI [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/416999 (https://phabricator.wikimedia.org/T187345) (owner: 10Fdans)
[14:16:13] <fdans>	 ah yes milimetric, the bug that nuria_ reported was related to the pixelratio division returning Infinity, I corrected that this morning
[14:16:23] <fdans>	 too much testing with only Apple devices
[14:16:26] <milimetric>	 sweet
[14:16:33] <ottomata>	 elukey:  but i get 404 outside of varnish :/
[14:16:40] <ottomata>	 curl analytics1001.eqiad.wmnet:8088/static/jquery/jquery-ui-1.9.1.custom.min.js
[14:17:39] <elukey>	 ottomata: curl analytics1001.eqiad.wmnet:8088/static/jquery/jquery-ui-1.9.1.custom.min.js --header "Accept-Encoding: gzip" -I :)
[14:18:00] <ottomata>	 HUH!
[14:18:01] <ottomata>	 interesting
[14:18:29] <elukey>	 so I bet that the suggested httpd proxy config is to use mod_deflate
[14:24:56] <ottomata>	 hm
[14:26:14] <elukey>	 or Jetty is somehow misconfigured to serve pre-compressed content
[14:33:09] <elukey>	 so I am checking what myself of the past wrong the in the yarn.w.o vhost config
[14:33:12] <elukey>	 and there is SetOutputFilter INFLATE;proxy-html;DEFLATE
[14:39:55] <joal>	 elukey: looks like it's back :)
[14:41:15] <elukey>	 joal: what do you mean?
[14:41:18] <joal>	 elukey: except when click the app-master link
[14:41:42] <joal>	 elukey: I mean yarn ui seems to have gotten the js
[14:41:59] <joal>	 But, app-master link broken (leading the analytics1001......)
[14:42:13] <elukey>	 I commented the SetOutputFilter INFLATE;proxy-html;DEFLATE (that breaks that) buuut I don't see the js loaded
[14:47:50] <elukey>	 super interesting: if I tweak chrome to avoid sending the Content-Encoding header, I get an error (browser not able to decode)
[14:53:11] <wikibugs>	 (03PS4) 10Fdans: Metrics carousel [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440)
[15:03:05] <elukey>	 joal: now it seems to work
[15:05:32] <joal>	 elukey: correct
[15:05:41] <joal>	 Need to drop to catch Lino - Later !
[15:07:19] <ottomata>	 elukey:  FYI i renamed some of the eventlogging dashboards :)
[15:07:27] <ottomata>	 you ok if i delete this one?
[15:07:28] <ottomata>	 https://grafana-admin.wikimedia.org/dashboard/db/eventlogging-to-be-deleted?from=now-3h&to=now
[15:08:29] <elukey>	 yep! 
[15:08:38] <elukey>	 ottomata: we also need to move the el alarms to prometheus
[15:08:43] <elukey>	 I keep forgetting tem
[15:08:45] <elukey>	 *them
[15:09:00] <wikibugs>	 (03PS1) 10Fdans: Corrects problem in previous rebase [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421545
[15:09:17] <ottomata>	 oh
[15:10:13] <fdans>	 milimetric: mind giving me a +2 here? I did a thing wrong in one file when rebasing the Responsive UI change
[15:10:13] <fdans>	 https://gerrit.wikimedia.org/r/#/c/421545/
[15:11:31] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Corrects problem in previous rebase [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421545 (owner: 10Fdans)
[15:14:03] <wikibugs>	 (03Merged) 10jenkins-bot: Corrects problem in previous rebase [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421545 (owner: 10Fdans)
[15:17:48] <elukey>	 a-team: yarn.w.o should be working ok now
[15:18:05] <ottomata>	 woot, thanks elukey
[15:18:06] <milimetric>	 elukey: what was it?
[15:18:42] <elukey>	 milimetric: https://gerrit.wikimedia.org/r/421547
[15:18:58] <elukey>	 this is a hack more than a permanent fix but it was the quickest one
[15:19:23] <milimetric>	 hm, so jetty changed defaults or something?
[15:19:45] <milimetric>	 nice job, thanks for the fix
[15:20:44] <elukey>	 milimetric: nope, I think that varnish stopped sending the Accept-Encoding: gzip and Jetty (with its actual settings) does not work, since it only serves pre-compressed js files
[15:20:50] <elukey>	 that is incredibly crazy
[15:22:44] <milimetric>	 maintaining this stuff is definitely crazy, so many moving parts that all depend on each other
[15:32:25] <elukey>	 fdans: if you have time we can do the second update
[15:49:27] <fdans>	 elukey: sorry, was out for a very late lunch
[15:49:34] <fdans>	 let's do it?
[15:49:49] <elukey>	 fdans: sure
[16:01:20] <wikibugs>	 (03PS5) 10Fdans: Metrics carousel [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440)
[16:01:46] <milimetric>	 fdans joal: standup?
[16:16:54] <wikibugs>	 10Analytics, 10Research: geowiki data for Global Innovation Index - 2017 - https://phabricator.wikimedia.org/T178183#4076189 (10Milimetric) I'm really sorry, I had written this update here months ago, and I guess there was some error submitting it, but just for the record, I did try :(  @leila: just a heads up...
[16:19:06] <wikibugs>	 10Analytics: Productionize job for Global Innovation Index from Hadoop Geowiki data - https://phabricator.wikimedia.org/T190535#4076210 (10Milimetric) p:05Triage>03Normal
[16:19:08] <wikibugs>	 10Analytics, 10Research: geowiki data for Global Innovation Index - 2017 - https://phabricator.wikimedia.org/T178183#4076222 (10leila) Thanks @Milimetric. For this year, I'll do it manually given that the deadline is in any minute (or past minutes. ;)
[16:28:36] <wikibugs>	 10Analytics, 10Operations, 10Wikimedia-Stream, 10hardware-requests, and 2 others: decommission rcs100[12] - https://phabricator.wikimedia.org/T170157#4076259 (10Cmjohnson)
[16:29:00] <wikibugs>	 10Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#4076261 (10Cmjohnson)
[16:29:06] <wikibugs>	 10Analytics, 10Operations, 10Wikimedia-Stream, 10hardware-requests, and 2 others: decommission rcs100[12] - https://phabricator.wikimedia.org/T170157#3420970 (10Cmjohnson) 05Open>03Resolved
[16:34:57] <joal>	 Heya team - Sorry, time conflict again
[16:35:10] <joal>	 I promise next week I'll be back to normal
[16:36:08] <elukey>	 going off earlier! 
[16:36:09] <elukey>	 byyee
[16:36:18] <joal>	 Bye elukey - Thanks gain for yarn!
[16:40:50] <chelsyx>	 nuria_: Could you tell me more about the outage on March 12, 13 for ios piwik dashboard? Is it the archiver again?
[16:41:46] <elukey>	 chelsyx: hi :)
[16:41:53] <elukey>	 are you seeing another hole?
[16:42:18] <chelsyx>	 elukey: yes, on March 12, 13
[16:42:29] <elukey>	 the infra on which bohrium/piwik runs now is more stable (the sre team fixed the underlying issue) but we might have to fix some days
[16:43:03] <elukey>	 chelsyx: ack, I am going to invalidate the data for those days
[16:43:21] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#3813007 (10Amitjoki) @mforns would displaying the description on the tooltip be enough? Currently, the tooltip that goes like, "Click this to get...
[16:43:45] <chelsyx>	 elukey: thx! so we don't need to do anything from our end? nuria_ contact me a few days ago telling me that we may want to downsampling the data
[16:44:35] <elukey>	 !log invalidated 2018-03-12/13 for iOS data in piwik to force a re-run of the archiver
[16:44:36] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:44:50] <nuria_>	 chelsyx: ya, that would be best as  piwik is really meant for microsites and this volume of events  is higher than what it can handle
[16:45:18] <elukey>	 (off for real now :)
[16:47:42] <chelsyx>	 nuria_: I'm wondering if making the Dispatch Interval would be helpful. the tracker now dispatch events every 60 seconds.
[16:48:50] <chelsyx>	 nuria_: the data has already sampled at a 1:10 rate
[16:49:27] <nuria_>	 chelsyx: right, but still the volume event seems to be too much for piwik, thus the holes you see
[16:50:01] <nuria_>	 chelsyx: i am surprised that it can even handle this much, it is set to do I order of magnitude less
[17:19:26] <joal>	 dsaez: plip plop?
[17:19:51] <dsaez>	 plip plop? 
[17:19:59] <joal>	 means hello are you here :)
[17:20:02] <joal>	 dsaez: --^
[17:20:14] <joal>	 Yay ! I finally find you dsaez :)
[17:20:25] <joal>	 dsaez: https://gist.github.com/jobar/a8bf503934ff1942e469a504b45062fa
[17:20:43] <joal>	 dsaez: Data available here: hdfs://user/joal/enwiki_links
[17:20:57] <joal>	 dsaez: I have info for you regarding the job etc
[17:21:35] <dsaez>	 joal> great! thanks
[17:22:20] <joal>	 dsaez: The mean problem we had was with the regex - It was not eagerly getting links, therefore returning huge portions of text, therefore breaking stuff
[17:22:43] <joal>	 dsaez: With correct regex, it takes time and memory, but works fine 
[17:23:42] <dsaez>	 amazing
[17:24:37] <dsaez>	 thank you
[17:25:04] <dsaez>	 joal: and in April we will have the new parquets, true?
[17:25:12] <joal>	 dsaez: no problem, I'd have liked to get back to you faster
[17:25:18] <joal>	 dsaez: not productionized
[17:26:21] <joal>	 dsaez: I'll download new data in April - And productionising data import is a stretch goal for me
[17:27:31] <ottomata>	 elukey:  i'm trying to use burrow http api
[17:27:37] <ottomata>	 8700 is for jumbo, ya?
[17:27:41] <ottomata>	 curl -v kafkamon1001.eqiad.wmnet:8700/v3/kafka
[17:27:41] <ottomata>	 says
[17:27:52] <joal>	 ottomata: elukey's gone in weekend ;_
[17:27:56] <ottomata>	 ohhh
[17:27:57] <ottomata>	 k
[17:28:02] <joal>	 ;)
[17:30:21] <ottomata>	 (nm also i figured it out :) )
[17:30:30] <joal>	 cool elukey :)
[17:30:36] <joal>	 ottomata soory
[17:30:42] <joal>	 ottomata: any news onthe torre side?
[17:50:37] <ottomata>	 joal:  naw, not really, i think we need to bulid it for our version of spark?  or scala?  there were some problems i couldn't get it to work
[17:50:40] <ottomata>	 i tried building but also failed
[17:51:46] <joal>	 ok ottomata - thanks for the feeback
[18:13:04] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#4076683 (10mforns) @Amitjoki I agree that the tooltip is a bit redundant there. I think what the average Wikistats2 user will expect is a 2/3 par...
[18:16:20] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#4076713 (10Amitjoki) @mforns, if that's the case, we could go one step ahead and do this: Make the first para visible in the tooltip and truncate...
[18:52:08] <wikibugs>	 (03PS1) 10Amitjoki: Converted initial mixed cases to regular cases. Will do more. [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421605
[19:01:09] <wikibugs>	 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4076895 (10Jdlrobson)
[19:51:45] <wikibugs>	 10Analytics, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4077070 (10chelsyx)
[20:16:51] <wikibugs>	 (03PS1) 10Milimetric: [WIP] Add geowiki_monthly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421616 (https://phabricator.wikimedia.org/T190409)
[20:21:07] <wikibugs>	 10Analytics: Puppet failing on k4-1.analytics.eqiad.wmflabs and k4-2.analytics.eqiad.wmflabs - https://phabricator.wikimedia.org/T190571#4077200 (10Andrew)
[20:28:47] <wikibugs>	 10Analytics, 10DBA, 10EventBus, 10MediaWiki-Database, and 6 others: High (2-3x) write and connection load on enwiki databases - https://phabricator.wikimedia.org/T189204#4077242 (10mobrovac) 05Open>03Resolved a:03Pchelolo I agree that all of the issues have been fixed, but to my understanding the sco...
[20:37:03] <wikibugs>	 10Analytics, 10Patch-For-Review: Puppet failing on k4-1.analytics.eqiad.wmflabs and k4-2.analytics.eqiad.wmflabs - https://phabricator.wikimedia.org/T190571#4077274 (10Ottomata) 05Open>03Resolved
[20:49:41] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4077308 (10Volker_E) @Nuria, the domain is live. Would you provide us with an instance?
[21:16:53] <joal>	 Gone for tonight team
[21:39:31] <wikibugs>	 10Analytics, 10Patch-For-Review: Puppet failing on k4-1.analytics.eqiad.wmflabs and k4-2.analytics.eqiad.wmflabs - https://phabricator.wikimedia.org/T190571#4077408 (10Andrew) Thank you!
[21:42:51] <wikibugs>	 10Analytics: Correct documentation for the referer_class field in pageview_hourly - https://phabricator.wikimedia.org/T190579#4077428 (10Tbayer)
[22:46:14] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#3813007 (10Tbayer) Frankly, we have, over the last decade or so, been quite bad at maintaining and updating analytics documentation such as this,...