[00:50:16] <wikibugs>	 10Analytics, 10SEO: Make Google API Python Client Library available on stat* machines - https://phabricator.wikimedia.org/T190767#4083192 (10mpopov) p:05Triage>03Normal
[01:15:59] <wikibugs>	 10Analytics: Notebook machine to double as RStudio Server? - https://phabricator.wikimedia.org/T190769#4083241 (10mpopov)
[01:45:27] <wikibugs>	 (03PS1) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422088 (https://phabricator.wikimedia.org/T190435)
[01:50:58] <wikibugs>	 (03PS1) 10Amitjoki: Expand topic explorer by default and fix T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089
[04:30:20] <ebernhardson>	 stat1005 is oom'ing :(
[05:42:19] <wikibugs>	 (03CR) 10Sahil505: "@Amitjoki There is no need to create a new patch for the same bug, you could have updated the previous patch or rebased it if you know how" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[06:40:32] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4083490 (10Volker_E) @Nuria Thanks for the quick action and the snippet. Just to clarify, these two entities are for different teams, I'm suspicious if having them unde...
[06:46:32] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4083495 (10Volker_E) For an explanation for the “image tracker” above, only with addition of `rec=1` parameter, non-JS users are tracked too. https://matomo.org/faq/how...
[06:56:12] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4083505 (10sahil505)
[07:37:35] <elukey>	 joal: o/
[07:37:38] <elukey>	 morninggg
[07:38:07] <elukey>	 FYI I am attemping (3rd time) to upgrade cassassandra on aqs1004 to allow the deploy of the jmx exporter
[07:40:43] <joal>	 np elukey - Please let me know if you want me to help
[07:41:34] <elukey>	 joal: --verbose list of things that I am doing (as FYI + sanity check as always :)
[07:41:46] <joal>	 sure
[07:42:49] <elukey>	 1) merged https://gerrit.wikimedia.org/r/#/c/421241, that basically is a workaround to allow only aqs to deploy cassandra 2.2.6-wmf3. The only relevant diff is that in the cassandra bash script launcher there is no weird check for JAVA_OPTS that causes any -javaagent to get stuck when loaded.
[07:43:04] <elukey>	 2) deploy it on aqs1004 and test that it works correctly (currently doing it)
[07:43:21] <elukey>	 3) merge the change to enable the jmx agent, and apply it again on aqs1004 only 
[07:43:35] <elukey>	 4) if everything works as expected, proceed with the rest :)
[07:43:55] <elukey>	 in the process I'll also reboot the nodes for kernel upgrades
[07:44:03] <joal>	 awesome elukey :)
[07:56:25] <elukey>	 the jmx agent seems to work! 
[07:56:35] <joal>	 elukey: Yay !!! Woooooot
[07:57:17] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4083565 (10Volker_E) For documentation purposes, the further adapted ([[ https://matomo.org/blog/2017/04/different-ways-embedding-piwik-tracking-code-faster-website-per...
[08:03:03] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4083568 (10Volker_E) PR at https://github.com/wikimedia/WikimediaUI-Style-Guide/pull/90
[08:03:17] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide, 10Patch-For-Review: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4083569 (10Volker_E)
[08:04:30] <elukey>	 ok so aqs1004 is running with: new kernel, new openjdk, new cassandra, jmx agent. Nothing exploding so far :)
[08:04:55] <joal>	 elukey: Yay bis !
[08:09:58] <wikibugs>	 (03PS6) 10Fdans: Metrics carousel [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440)
[08:10:51] <elukey>	 ok proceeding with aqs1005!
[08:22:58] <wikibugs>	 10Analytics-Kanban: Zooming in or out in wikistats shouldn't alter the number of metrics shown - https://phabricator.wikimedia.org/T190782#4083610 (10fdans)
[08:30:46] <wikibugs>	 (03PS1) 10Fdans: Prevents zooming in site from changing the number of metrics shown per area [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422102 (https://phabricator.wikimedia.org/T190782)
[08:43:18] <elukey>	 upgrading aqs1006!
[08:52:46] <elukey>	 ah snap I just realized that the openjdk-8 package is not upgraded 
[08:53:20] <elukey>	 moritzm: hi! :)
[08:53:45] <elukey>	 so aqs100[456] came up nicely with new cassandra + kernel + jmx agent
[08:54:08] <elukey>	 I still need to do aqs100[789], so I guess we could add the new openjdk in there 
[08:55:35] <elukey>	 or I can finish the upgrade/reboots and do the restarts tomorrow 
[08:55:42] <elukey>	 better probably, already too many changes in flight :)
[09:05:47] <elukey>	 joal: so half way through cassandra metrics are looking good, didn't spot anything weird
[09:05:58] <elukey>	 if you are ok I'd proceed with the last three nodes
[09:06:31] <elukey>	 nice consequence about this round of changes: we'll be able to drop all the aqs:: specific grafana dashboard, to re-use https://grafana.wikimedia.org/dashboard/db/cassandra?orgId=1
[09:06:38] <elukey>	 that is shared with services
[09:06:50] <elukey>	 so we'll get all the new changes/improvements/etc..
[09:06:58] <elukey>	 rather than lagging behind
[09:10:09] <elukey>	 (doing 1007 now)
[09:22:32] <elukey>	 ok I didn't notice https://grafana.wikimedia.org/dashboard/db/aqs-elukey?orgId=1&panelId=7&fullscreen&from=now-6h&to=now
[09:22:44] <elukey>	 there are a bit of 504s ?
[09:25:38] <elukey>	 ah no wait these are probably due to the pool/depools/reboots etc..
[09:25:50] <elukey>	 very low amount 
[09:25:53] <elukey>	 okok nothing on fire
[09:28:07] <elukey>	 proceeding with 1008
[09:45:03] <elukey>	 aaand 1009
[09:48:57] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reboot all Analytics hosts for Kernel upgrade - https://phabricator.wikimedia.org/T188594#4083729 (10elukey)
[09:49:04] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reboot all Analytics hosts for Kernel upgrade - https://phabricator.wikimedia.org/T188594#4012985 (10elukey)
[09:50:31] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx agent to AQS Cassandra - https://phabricator.wikimedia.org/T184795#4083753 (10elukey)
[09:56:00] <elukey>	 done!
[09:57:41] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx agent to AQS Cassandra - https://phabricator.wikimedia.org/T184795#4083785 (10elukey)
[09:59:02] <elukey>	 ah so after reading https://phabricator.wikimedia.org/T186567#3993452 I am not so happy anymore about the shared analytics/services dashboard anymore :(
[10:14:49] <elukey>	 sooo aqs looks good, rebooted and with the jmx agent
[10:15:21] <elukey>	 I am going to do the roll restart for the jvm updates in a couple of days, just to make sure that the new settings do not cause any regression
[10:15:37] * elukey afk for a bit
[10:31:19] <moritzm>	 elukey: good morning, starting later today, younger son can't go to day care today. We can upgrade OpenJDK, but we first should verify that Cassandra 2 isn't hit by the bug which prevents us to use the new Java with Cassandra 3.11
[10:31:35] <moritzm>	 Eric said it's unlikely, but still let's better test it
[10:31:56] <moritzm>	 is there an AQS test setup in labs or can we test only in production?
[10:40:57] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4083956 (10sahil505)
[10:47:13] <wikibugs>	 (03PS7) 10Fdans: Metrics carousel [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440)
[10:54:01] <elukey>	 moritzm: hi! yes there is in deployment prep
[10:54:26] <elukey>	 should be deployment-aqs01.deployment-prep.eqiad.wmflabs
[10:54:56] <elukey>	 and I need to upgrade cassandra in there too
[10:55:57] <elukey>	 ah nice 8u162-b12-1~bpo8+1 is already deployed
[10:56:01] <elukey>	 restarting cassandra
[10:57:45] <moritzm>	 the magic of unattended-upgrades
[10:59:28] <elukey>	 moritzm: seems working fine :)
[11:02:41] <moritzm>	 great, shall I upgrade all of AQS with the new version?
[11:03:44] <elukey>	 moritzm: yep, if you are ok I'd wait a couple of days before doing the rolling restart to check metrics after today's reboots/changes
[11:08:01] <moritzm>	 sure, Java packages have been upgraded
[11:10:21] <moritzm>	 BTW, I checked missing restarts on the hadoop cluster and there's four hosts missing a restart of yarn (1060, 1040, 1062, 1030)
[11:11:07] <elukey>	 moritzm: weird, are those long running jvms by any chance? (checking)
[11:11:55] <moritzm>	 the one on 1030 is running since March 6
[11:13:06] <elukey>	 ah yes it is a spark job
[11:13:27] <elukey>	 org.wikimedia.analytics.refinery.job.druid.BannerImpressionsStream
[11:14:13] <elukey>	 this one is a known job that pulls from kafka and sends to druid realtime, stays up unless explicitly killed/respawned
[11:14:50] <elukey>	 something to keep in mind for the next restart
[11:14:52] <elukey>	 check the others
[11:15:31] <elukey>	 spark again 
[11:15:32] <elukey>	 lovely :)
[11:16:21] <elukey>	 on 1040 is again banner impression
[11:16:47] <elukey>	 !log kill banner impression job to force a respawn (still using an old jvm)
[11:16:48] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[11:19:01] <elukey>	 moritzm: I've run lsof -Xd DEL across the workers and they seem clean
[11:21:47] <moritzm>	 ack, confirmed :-)
[11:22:16] <moritzm>	 analytics1003 isn't updated yet, BTW
[11:22:26] <moritzm>	 the rest of analytics* is
[11:24:17] <elukey>	 moritzm: can we update it as well? I was planning to check since restarting daemons in there is a bit delicate (requires stopping hive etc..)
[11:25:51] <elukey>	 and I guess also druid*
[11:26:08] <elukey>	 (maybe we can start with druid1002)
[11:30:46] <moritzm>	 ack, 1003 updated
[11:31:12] <moritzm>	 shall I already upgrade all of druid* or only 1002 first?
[11:38:26] <elukey>	 you can do all
[11:38:40] <joal>	 Hey team - I needed to leave to catch Naé from creche again - I'll be on and off the rest of the ay
[11:39:21] <elukey>	 ack!
[11:41:20] <moritzm>	 druid* upgraded
[11:41:41] <elukey>	 thanksssss
[12:21:58] <wikibugs>	 (03CR) 10Mforns: Expand topic explorer by default and fix T186335 (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[12:23:08] <wikibugs>	 (03Abandoned) 10Mforns: Expand topic explorer by default as in T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 (owner: 10Amitjoki)
[12:25:55] <wikibugs>	 (03PS2) 10Mforns: Expand topic explorer by default and fix T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[12:28:33] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Expand topic explorer by default and fix T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[12:35:42] <wikibugs>	 (03CR) 10Mforns: [C: 04-1] "Hm, some tests are failing..." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[12:39:34] <wikibugs>	 (03PS2) 10Mforns: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435) (owner: 10Amitjoki)
[12:45:39] <wikibugs>	 (03Restored) 10Mforns: Expand topic explorer by default as in T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 (owner: 10Amitjoki)
[13:08:31] <elukey>	 morning ottomata !
[13:09:47] <ottomata>	 morning!
[13:23:37] <ottomata>	 elukey:  am thikning about using https://github.com/williamsjj/kafka_health/blob/master/nagios/check_kafka_consumer_group.py for burrow -> icinga notifications
[13:23:52] <ottomata>	 might be easier to configure for mirror marker instances
[13:23:58] <ottomata>	 would be nice to make those alerts visible in icigna
[13:24:25] <ottomata>	 we can update it if/when we switch to newer burrow pretty easily i thikn
[13:25:12] <elukey>	 ottomata: makes sense yes! 
[13:26:29] <wikibugs>	 (03CR) 10Amitjoki: "> @Amitjoki There is no need to create a new patch for the same bug," [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[13:28:58] <wikibugs>	 (03Abandoned) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422088 (https://phabricator.wikimedia.org/T190435) (owner: 10Amitjoki)
[13:30:11] <wikibugs>	 (03Abandoned) 10Amitjoki: Expand topic explorer by default and fix T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422089 (owner: 10Amitjoki)
[13:56:49] <wikibugs>	 (03PS3) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435)
[13:59:14] <wikibugs>	 (03PS4) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435)
[14:03:01] <elukey>	 !log consolidate all the zookeeper definition in one 'main-eqiad' one in Horizon -> Project-Analytics
[14:03:03] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:03:05] <elukey>	 ottomata: --^
[14:03:42] <wikibugs>	 (03PS2) 10Amitjoki: Expand topic explorer by default as in T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711
[14:03:51] <elukey>	 doing some refactor in there since 1) I didn't get that we have a per project global hiera (the usual me discovering things in time) 2) it will be good for testing my zookeper  cluster expansion
[14:06:46] <elukey>	 yeah also my hadoop hiera config was horrible
[14:07:07] * elukey looks for jobs in something not related to computers
[14:07:57] <ottomata>	 ok
[14:08:01] <ottomata>	 hah
[14:08:15] <ottomata>	 elukey:  might need a brain bounce about how best to do this monitoring check
[14:08:21] <ottomata>	 puppet is making me do crazy things
[14:08:31] <ottomata>	 i'm trying to reuse the profile::kafka::monitoring::config you have set up already
[14:08:45] <ottomata>	 to auto declare teh checks
[14:09:40] <elukey>	 ottomata: sure! 
[14:10:17] <ottomata>	 gimme a few... still tihnking
[14:23:11] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] "Looks good! Merging :]" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 (owner: 10Amitjoki)
[14:40:29] <wikibugs>	 (03CR) 10Mforns: [C: 04-1] "Cool that the stacking problems were solved :]" (036 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435) (owner: 10Amitjoki)
[14:43:56] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Zooming in or out in wikistats shouldn't alter the number of metrics shown - https://phabricator.wikimedia.org/T190782#4084556 (10Nuria) >Funny thing is, this only happens in every other zoom step. If we zoom in again the width goes back to 1024px, and so on. Right, i sa...
[14:46:00] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide, 10Patch-For-Review: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4084578 (10Nuria) >Thanks for the quick action and the snippet. Just to clarify, these two entities are for different teams, I'm suspicious if hav...
[14:49:09] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Zooming in or out in wikistats shouldn't alter the number of metrics shown - https://phabricator.wikimedia.org/T190782#4084589 (10fdans) @Nuria but the issue you reported was caused by computed width of the app container switching between 1024 and 1023.99. Even though th...
[14:52:32] <ottomata>	 alright elukey i got something, want to talk about it though
[14:52:33] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/422163/
[14:52:41] <ottomata>	 happy to chat for few mins before standup if you wann :)
[14:52:44] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Refresh zookeeper nodes on eqiad - https://phabricator.wikimedia.org/T182924#4084598 (10elukey)
[14:52:58] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Refresh zookeeper nodes on eqiad - https://phabricator.wikimedia.org/T182924#3838887 (10elukey)
[14:53:02] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#4084601 (10elukey)
[14:54:18] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Refresh zookeeper nodes in eqiad - https://phabricator.wikimedia.org/T182924#3838887 (10elukey) p:05Triage>03High
[14:56:27] <elukey>	 ottomata: sure
[14:56:44] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4084609 (10Milimetric) @Nuria I see the same thing, yeah, but that's good.  When you zoom you limit the number of pixels you see on your screen, which triggers media queries.  Even if there...
[14:57:06] <wikibugs>	 (03PS5) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435)
[14:59:58] <fdans>	 milimetric: take a look at https://phabricator.wikimedia.org/T190782
[15:00:10] <fdans>	 for nuria_ 's reported issue
[15:00:12] <ottomata>	 ack elukey sorry
[15:00:13] <ottomata>	  missed it
[15:00:14] <ottomata>	 haha
[15:01:19] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Zooming in or out in wikistats shouldn't alter the number of metrics shown - https://phabricator.wikimedia.org/T190782#4083610 (10Milimetric) oh! good investigation, my bad then
[15:02:23] <wikibugs>	 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review: public data sets for the WDCM Biases Dashboard - https://phabricator.wikimedia.org/T189653#4084635 (10Nuria) 05Open>03Resolved
[15:30:19] <wikibugs>	 (03PS6) 10Amitjoki: Correct mixed case in Wikistats2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435)
[15:33:03] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Find out if Kibana/Elasticsearch allows queries based on the results of other queries - https://phabricator.wikimedia.org/T189903#4084932 (10Aklapper) 05Open>03Resolved Investigation done, hence closing this task. Again thanks to Eri...
[15:36:59] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435) (owner: 10Amitjoki)
[15:37:20] <wikibugs>	 (03PS3) 10Mforns: Expand topic explorer by default as in T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 (owner: 10Amitjoki)
[15:37:29] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] Expand topic explorer by default as in T186335 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 (owner: 10Amitjoki)
[15:56:02] <wikibugs>	 10Analytics: Define battery of smoke tests  to run by hand before realease - https://phabricator.wikimedia.org/T190837#4085060 (10Nuria)
[16:03:51] <wikibugs>	 10Analytics: Spike: Quantify how many EventLogging requests we get from non-wiki* hostnames or apps - https://phabricator.wikimedia.org/T190840#4085113 (10Milimetric)
[16:04:35] <milimetric>	 elukey: ^ that's the thing
[16:05:35] <elukey>	 ack thanks!
[16:11:22] <milimetric>	 mforns: I added another job like the reportupdater-queries one, for the interlanguage metrics
[16:11:29] <milimetric>	 and it's throwing a permission problem
[16:11:33] <milimetric>	 but not being very specific
[16:11:45] <milimetric>	 does that ring a bell, did you have to create / chown a directory somewhere or something?
[16:12:12] <milimetric>	 it got installed on stat1005:/srv/reportupdater/jobs/limn-language-data/
[16:12:29] <milimetric>	 configured here: https://github.com/wikimedia/puppet/blob/2905f1305eb8fb4d51da14355e435efa8ddbd379/modules/profile/manifests/reportupdater/jobs/hadoop.pp#L46
[16:14:46] <mforns>	 milimetric, it happened before
[16:15:20] <ottomata>	 milimetric:  looking too
[16:15:36] <mforns>	 milimetric, did you run it by hand at some point?
[16:15:49] <milimetric>	 no mforns
[16:15:54] <mforns>	 ok
[16:16:00] <milimetric>	 but the output where it's supposed to go isn't created
[16:16:02] <milimetric>	 tail /srv/reportupdater/limn-language-data-interlanguage.log
[16:16:20] <ottomata>	 where is output supposed to go?
[16:16:35] <milimetric>	 sorry, that should be tail /srv/reportupdater/log/limn-language-data-interlanguage.log
[16:17:00] <milimetric>	 output is supposed to be in /srv/reportupdater/output/metrics/interlanguage according to the config
[16:17:07] <mforns>	 aha
[16:17:09] <milimetric>	 which I thought puppet creates when it instantiates the job
[16:17:20] <ottomata>	 the script should have perms to create it
[16:17:50] <mforns>	 milimetric, I don't think that puppet creates that, rather update_reports.py created that when writing
[16:19:30] <milimetric>	 oh ok, but it's running as the hdfs user and those directories are all owned by hdfs/root or hdfs/wikidev, maybe there's a mismatch in there somewhere?
[16:19:53] <mforns>	 hm
[16:20:22] <milimetric>	 it's ok if you don't know, just checking
[16:20:28] <milimetric>	 I'll run the command manually and see what's up
[16:20:31] <milimetric>	 python /srv/reportupdater/reportupdater/update_reports.py -l info /srv/reportupdater/jobs/limn-language-data/interlanguage /srv    /reportupdater/output/metrics/interlanguage >> /srv/reportupdater/log/limn-language-data-interlanguage.log 2>&1
[16:20:53] <mforns>	 yes, that's what happened the other time, but it was because someone executed update_reports.py by hand with another user first
[16:21:15] <mforns>	 milimetric, use sudo -u hdfs no?
[16:21:15] <milimetric>	 oh ok, weird, I'll run it with debug and see
[16:21:17] <milimetric>	 thanks
[16:21:18] <milimetric>	 yeah
[16:21:26] <elukey>	 nuria_: if you want we can discuss https://gerrit.wikimedia.org/r/#/c/422135/, or I can read the comments tomorrow :)
[16:21:45] <nuria_>	 elukey: on meeting, let's do tomorrow?
[16:22:37] <nuria_>	 elukey: posted some comments but let's talk tomorrow
[16:23:51] <wikibugs>	 10Analytics: Define battery of smoke tests  to run by hand before realease - https://phabricator.wikimedia.org/T190837#4085198 (10Nuria)
[16:24:09] <elukey>	 nuria_: sure, I'll answer to those comments
[16:27:50] <mforns>	 milimetric, I don't see any weirdness in the code you gave me... should work well theoretically!
[16:28:11] <ottomata>	 elukey:  ok, i think https://gerrit.wikimedia.org/r/#/c/422163/ is gonna work
[16:28:16] <ottomata>	 got a sec for a quick review?
[16:28:42] <ottomata>	 happy to bikeshed name of 'nagios_check' param and check define, etc.
[16:30:36] <nuria_>	 milimetric: the EL events  from russian site are hitting us from : https://ru.wikipedia.kim/beacon/event?
[16:30:48] <nuria_>	 milimetric: which means WE Are hosting that site
[16:31:00] <elukey>	 ottomata: sure, review coming in 2 mins
[16:31:05] <nuria_>	 milimetric: unless they proxy them to us server side
[16:32:39] <ottomata>	 nuria_:  i think it is proxied
[16:32:44] <ottomata>	 somehow...
[16:33:22] <nuria_>	 ottomata: it has to be cause as far as i can see it is a russian clone of wikipedia with adds
[16:36:18] <ottomata>	 its not just ru
[16:36:24] <ottomata>	 https://en.wikipedia.kim/wiki/Main_Page
[16:37:38] <elukey>	 ottomata: looks good to me!
[16:37:54] <elukey>	 the only doubt that I have is adding the consumer_group without email, but should be fine
[16:38:05] <elukey>	 (burrow should not get upset)
[16:38:13] <nuria_>	 ottomata: why would they proxy those in their end  it is a mistery
[16:39:57] <ottomata>	 asking in mw sec about this... 
[16:40:11] <ottomata>	 elukey:  it should be fine, since you could also configure it to use http notifier
[16:41:53] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843#4085283 (10Nuria)
[16:42:24] <nuria_>	 ottomata: ticket created (cc milimetric ) more than anything to explain brandon this is happening
[16:43:06] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843#4085301 (10Nuria)
[16:43:36] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843#4085283 (10Nuria) Adding folks from traffic.
[16:43:52] <milimetric>	 nuria_ / ottomata: wait... that's *.kim*, what is that?  Aren't we always .org?
[16:44:48] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843#4085324 (10Nuria) Seems like this is an "unofficial mirror". Looping legal
[16:46:59] <ottomata>	 yes
[16:47:16] <ottomata>	 nuria_:  they asked me to send an email to legal, will mention that ticket
[16:47:40] <nuria_>	 ottomata: ok, ya, as i was looking at what they do it was seeming more and more sketchy
[16:51:40] <wikibugs>	 10Analytics, 10SEO: Make Google API Python Client Library available on stat* machines - https://phabricator.wikimedia.org/T190767#4085346 (10mpopov) p:05Normal>03High
[16:55:55] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843#4085283 (10Jdlrobson) Note there is also https://ru.wiki.ng/ doing exactly the same thing (different host)
[16:59:02] <nuria_>	 joal: let's have our meeting 5 minutes  later, need to change rooms
[16:59:16] <joal>	 no problem nuria_ 
[17:19:46] <wikibugs>	 (03PS1) 10Milimetric: Quote dates [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422188
[17:19:59] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Quote dates [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422188 (owner: 10Milimetric)
[17:38:55] <wikibugs>	 (03PS1) 10Milimetric: Qualify table names [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422191
[17:39:04] <wikibugs>	 (03CR) 10Milimetric: [V: 032 C: 032] Qualify table names [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422191 (owner: 10Milimetric)
[17:39:28] * elukey off!
[17:41:27] <wikibugs>	 10Analytics: Upgrade main Kafka clusters to 1.0 - https://phabricator.wikimedia.org/T190853#4085559 (10mforns)
[17:53:25] <wikibugs>	 10Analytics: Enable automatic ingestion from eventlogging into druid for some schemas - https://phabricator.wikimedia.org/T190855#4085639 (10mforns)
[17:54:44] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Zooming in or out in wikistats shouldn't alter the number of metrics shown - https://phabricator.wikimedia.org/T190782#4085650 (10Nuria) >Even though the pixel ration was changing too (as it should), that wasn't causing the threshold jump. the point i wanted to make was...
[17:55:40] <wikibugs>	 10Analytics: Archive old geowiki data (editors per country) and make it easily available at WMF - https://phabricator.wikimedia.org/T190856#4085654 (10mforns)
[17:55:43] <nuria_>	 mforns, milimetric , fdans : let's please make sure any changeset we CR in wikistats UI we also take time to look at in our desktop visually please, that way we have one more pair of eyes in every changeset. 
[17:56:02] <mforns>	 sure
[17:56:35] <wikibugs>	 (03PS1) 10Milimetric: Use the right date format [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422198
[17:56:46] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Use the right date format [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422198 (owner: 10Milimetric)
[17:57:24] <milimetric>	 nuria_: that's a good guideline, though I did run the code in that last case, I just didn't zoom in, I only changed the window size.  That's my fault
[17:57:46] <nuria_>	 milimetric:  looking for solutions no guilty parties
[17:58:20] <milimetric>	 I know, I agree with fdans that we should have a checklist of what exactly to check, 'cause it's easy to forget the more nuanced tests like zoom / hover
[17:58:20] <nuria_>	 milimetric: it is a matter of  acommon understanding and we all agree that regressions should not happen so let's start with that cc mforns fdans 
[17:59:08] <mforns>	 +1 milimetric
[17:59:42] <wikibugs>	 10Analytics, 10EventBus, 10ORES, 10Reading-Infrastructure-Team-Backlog, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#3319947 (10mforns) Moving this to Incoming column, because it appears in the goals for next quarter: https://www...
[18:00:50] <nuria_>	 milimetric: started list https://wikitech.wikimedia.org/wiki/Analytics/Wikistats_2#Smoke_tests
[18:00:56] <nuria_>	 cc fdans mforns 
[18:02:55] <wikibugs>	 10Analytics: Build prototype for MediaWiki content processing - https://phabricator.wikimedia.org/T190858#4085692 (10mforns)
[18:04:30] <milimetric>	 nuria_: added some thoughts, thanks for writing
[18:06:34] <wikibugs>	 10Analytics, 10Research: [Open question] Improve bot identification at scale - https://phabricator.wikimedia.org/T138207#4085724 (10mforns) I linked this task from Analytics goals (https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q4#Program_7._Smart_tools_for_better_data), as it has been the o...
[18:11:00] <nuria_>	 milimetric: ADDED more ou yeah
[18:17:38] <milimetric>	 mforns: I figured it out and made it a little more explicit in the docs: https://wikitech.wikimedia.org/w/index.php?title=Analytics%2FSystems%2FReportupdater&type=revision&diff=1786910&oldid=1777966
[18:17:50] <mforns>	 milimetric, reading
[18:17:51] <milimetric>	 wow that took forever to find, I feel slow :)
[18:18:06] <mforns>	 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaah
[18:18:11] <mforns>	 boy...
[18:18:40] <mforns>	 milimetric, was it a new install?
[18:18:50] <milimetric>	 mforns: new install of what?
[18:18:52] <wikibugs>	 (03PS1) 10Milimetric: Make scripts executable [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422205
[18:18:55] <mforns>	 reportupdater?
[18:19:11] <mforns>	 what script was oh! the bash script?
[18:19:13] <milimetric>	 mforns: no, the files with the hive commands were wrong
[18:19:18] <mforns>	 ok ok 
[18:19:23] <mforns>	 I understand
[18:19:33] <milimetric>	 mforns: I read this amazing thing the other day
[18:19:47] <milimetric>	 that Roman concrete was super strong, and we couldn't figure out how they made it for a really long time
[18:19:58] <milimetric>	 even though we had the recipe, and followed it exactly
[18:20:03] <milimetric>	 until someone thought to use salt water
[18:20:19] <nuria_>	 ottomata: our eventlogging machine is eventlog1002 (triple checking), correct?
[18:20:19] <milimetric>	 to the Romans it was obvious, because of course you wouldn't waste clean drinkable water
[18:20:27] <milimetric>	 to us, the opposite
[18:20:28] <mforns>	 O.o
[18:20:46] <nuria_>	 milimetric: i LOVE that story
[18:20:52] <nuria_>	 context is everything
[18:20:56] <milimetric>	 cool, right? :)
[18:21:51] <milimetric>	 my point regarding the scripts is that to me, nothing is obvious :)
[18:21:54] <nuria_>	 milimetric: if ever we needed an example of why documentation w/o communication  will never work
[18:22:13] <nuria_>	 milimetric: what a good one, the same word means something completely different
[18:22:29] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Make scripts executable [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422205 (owner: 10Milimetric)
[18:24:07] <milimetric>	 btw, too, "tree -puf /srv/reportupdater/jobs/" helped me figure out the problem ultimately, 'cause the reportupdater scripts were green and the others were not
[18:25:14] <wikibugs>	 10Analytics, 10Analytics-Kanban: Issues with page deleted dates on data lake - https://phabricator.wikimedia.org/T190434#4085830 (10Nettrom) I've spent a bit of time looking at this, and as far as I can find, the `revision_deleted_timestamp` is consistently incorrect. Using a sample dataset of creations from f...
[18:30:15] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4085842 (10JMinor)
[18:32:00] <wikibugs>	 (03PS1) 10Milimetric: Fix funnel configuration [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422207
[18:32:09] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Fix funnel configuration [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/422207 (owner: 10Milimetric)
[18:34:33] <dsaez>	 Hi Trizek
[18:39:46] <nuria_>	 ping ottomata looked at box and errors on validation and i think we are reday to launch to the next set of wikis  , let me know if you disagree
[18:43:37] <nuria_>	 https://www.irccloud.com/pastebin/JQCoF2TF/
[18:43:41] <nuria_>	 sorry
[18:44:08] <ottomata>	 nuria_:  +1 let's do it
[18:44:15] <nuria_>	 fdans: let's do your next rebase together, there are two rebases that are appearing as merges on git history , see for example this  changeset
[18:44:27] <nuria_>	 fdans: 
[18:44:30] <nuria_>	 https://www.irccloud.com/pastebin/0UQuqbDh/
[18:44:45] <nuria_>	 it a ppears twice cause I think you might be merging rather than rebasing
[18:45:17] <nuria_>	 ottomata: ok, i think disk space in el machine might become a concern for real
[18:45:32] <nuria_>	 ottomata: we might need to logpull logs out at  a higher rfrequency
[18:46:17] <nuria_>	 ottomata: does that make sense?
[18:57:19] <ottomata>	 oh disk space
[18:57:20] <ottomata>	 hmm
[18:57:22] <ottomata>	 yes
[18:57:34] <ottomata>	 nuria_:  not sure
[18:57:38] <ottomata>	 there is 736G avail
[18:58:32] <ottomata>	 looks like we keep last 30 day sof logs on eventlog1002
[18:58:36] <ottomata>	 happy to make that less though
[18:58:40] <ottomata>	 maybe 21
[18:58:51] <nuria_>	 ottomata: our rate of events is going to double which means we might have say another .5 Tb next week and every week after that 
[18:59:09] <nuria_>	 ottomata: let me check numbers on the newly added logs
[19:01:38] <ottomata>	 nuria_:  if it doubles, that'll be another < 100G of logs stored
[19:01:49] <ottomata>	 we currently have about 74G
[19:02:55] <nuria_>	 ottomata: wait , we have about two weeks in /srv/log/eventlogging/archive
[19:04:21] <nuria_>	 ottomata: ah yesyes 
[19:04:25] <nuria_>	 ottomata: i see mi mistake
[19:04:38] <nuria_>	 ottomata: not all logs are compressed, teh recent ones are not
[19:04:41] <nuria_>	 ottomata: SORRY
[19:04:47] <nuria_>	 ottomata: on archive dir
[19:04:57] <nuria_>	 ottomata: FALSE alarm, abort!
[19:05:00] <ottomata>	 :)
[19:17:28] <wikibugs>	 10Analytics, 10EventBus, 10ORES, 10Reading-Infrastructure-Team-Backlog, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4086017 (10Halfak) Great news!  Let me know how I can help.
[19:36:47] <wikibugs>	 (03PS6) 10Joal: [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449)
[19:38:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) (owner: 10Joal)
[20:05:42] <wikibugs>	 (03PS7) 10Joal: [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449)
[20:08:26] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) (owner: 10Joal)
[20:18:23] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Alert for Kafka MirrorMaker lag - https://phabricator.wikimedia.org/T189611#4086291 (10Ottomata) a:03Ottomata
[20:32:03] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Alert for Kafka MirrorMaker lag - https://phabricator.wikimedia.org/T189611#4086338 (10Ottomata) lag alert: - https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=kafkamon1001&service=Kafka+main-eqiad+consumer+group+la...
[20:33:41] <SMalyshev>	 ottomata: is jumbo/mirrormaker fixed now or still unstable?
[20:40:29] <wikibugs>	 (03CR) 10Milimetric: "Feedback on look and feel, sorry if some of it is from previous patch:" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[20:45:24] <wikibugs>	 (03CR) 10Milimetric: "Wait, more visual stuff." (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[20:51:02] <wikibugs>	 (03CR) 10Milimetric: Metrics carousel (035 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[20:52:35] <wikibugs>	 (03PS2) 10Milimetric: Make geowiki job output monthly druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409)
[20:56:57] <wikibugs>	 (03CR) 10Milimetric: "Ok, couple tweaks and it's all good.  This shows active editors (5+) by country broken down by whether they edit anonymously or not: https" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409) (owner: 10Milimetric)
[21:05:40] <wikibugs>	 10Analytics, 10User-Elukey: latest varnishkafka fails to build on Debian - https://phabricator.wikimedia.org/T186250#4086443 (10Jrdnch) Alright, I'll do both of those things.
[21:56:20] <wikibugs>	 (03PS2) 10Nuria: Prevents zooming in site from changing the number of metrics shown per area [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422102 (https://phabricator.wikimedia.org/T190782) (owner: 10Fdans)
[21:56:56] <wikibugs>	 (03CR) 10Nuria: Prevents zooming in site from changing the number of metrics shown per area (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422102 (https://phabricator.wikimedia.org/T190782) (owner: 10Fdans)
[21:57:35] <wikibugs>	 (03PS3) 10Nuria: Prevents zooming from changing the number of metrics shown per area [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422102 (https://phabricator.wikimedia.org/T190782) (owner: 10Fdans)
[22:07:13] <wikibugs>	 (03PS4) 10Nuria: Prevents zooming from changing the number of metrics shown per area [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/422102 (https://phabricator.wikimedia.org/T190782) (owner: 10Fdans)
[22:12:52] <wikibugs>	 (03CR) 10Nuria: "I think is probably worth moving the change to master rather than having to do merge by hand." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421041 (https://phabricator.wikimedia.org/T188953) (owner: 10Fdans)
[22:38:28] <wikibugs>	 (03PS8) 10Nuria: Metrics carousel [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[22:38:40] <wikibugs>	 (03CR) 10Nuria: "Some warnings on console " Vue warn]: Failed to resolve directive: touch" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[23:30:42] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Refresh SWAP notebook hardware - https://phabricator.wikimedia.org/T183145#4086760 (10Tbayer) Small note for the record: I'm getting "Warning: JupyterHub seems to be served over an unsecured HTTP connection. We strongly recommend enabling HTTPS for JupyterHub" at the log...