[00:03:46] <wikibugs>	 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10SNowick_WMF) a:05mpop...
[00:08:37] <wikibugs>	 10Analytics, 10Product-Analytics (Kanban): Unique Devices Data for Mobile Apps in Wikistats - https://phabricator.wikimedia.org/T257998 (10SNowick_WMF) Thank you for the reminder @Nuria, I've moved the assignment of [[ https://phabricator.wikimedia.org/T202664 | T202664 ]] to myself, will make sure that this c...
[00:08:48] <wikibugs>	 10Analytics, 10Product-Analytics (Kanban): Unique Devices Data for Mobile Apps in Wikistats - https://phabricator.wikimedia.org/T257998 (10SNowick_WMF) 05Open→03Resolved a:03SNowick_WMF
[06:25:43] <elukey>	 good morning :)
[06:45:12] <RhinosF1>	 hi
[07:43:18] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10Marostegui) >>! In T234826#6307751, @elukey wrote: > Side note - in T257412 I am investigating a failover plan to write down in case a...
[07:49:57] * elukey afk for ~1h, doctor appointment
[07:52:33] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Repurpose db1108 as generic Analytics db replica - https://phabricator.wikimedia.org/T234826 (10elukey) @Marostegui yep yep it was only an idea to avoid changing the port in puppet when doing the failover (and just update the CNAM...
[09:28:26] * elukey back
[09:28:37] <elukey>	 so druid 0.18.1 includes hadoop-client 2.8.5
[09:28:43] <elukey>	 that of course doesn't work for us
[09:29:02] <elukey>	 I tried to switch to the cdh hadoop-client (via puppet config), as we used to do, and it doesn't work either :D
[09:38:17] <wikibugs>	 (03PS5) 10Fdans: [wip] Allow more than one dimension to be filtered in Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/612574 (https://phabricator.wikimedia.org/T255757)
[09:38:19] <wikibugs>	 (03PS4) 10Fdans: [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/607768 (https://phabricator.wikimedia.org/T249758)
[09:39:48] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [wip] Allow more than one dimension to be filtered in Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/612574 (https://phabricator.wikimedia.org/T255757) (owner: 10Fdans)
[09:39:50] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/607768 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans)
[09:40:28] <wikibugs>	 (03PS6) 10Fdans: [wip] Allow more than one dimension to be filtered in Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/612574 (https://phabricator.wikimedia.org/T255757)
[09:41:23] <wikibugs>	 (03PS5) 10Fdans: [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/607768 (https://phabricator.wikimedia.org/T249758)
[09:41:52] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [wip] Allow more than one dimension to be filtered in Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/612574 (https://phabricator.wikimedia.org/T255757) (owner: 10Fdans)
[09:42:46] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/607768 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans)
[10:08:47] <wikibugs>	 (03Abandoned) 10Fdans: [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/607768 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans)
[10:15:55] <wikibugs>	 (03PS1) 10Fdans: [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758)
[10:17:32] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [wip] Add filter/split component to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/613114 (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans)
[10:26:12] * elukey quick lunch
[11:13:16] <wikibugs>	 10Analytics, 10Operations: Move yarn.wikimedia.org to a separate Buster VM - https://phabricator.wikimedia.org/T258152 (10MoritzMuehlenhoff)
[11:15:54] <elukey>	 so the current problem is the same as https://github.com/apache/druid/issues/5763
[11:16:02] <elukey>	 I forgot past me already doing this
[11:26:27] <joal>	 Hi team
[11:28:04] <joal>	 elukey: it'll be great to have bigtop - At least we'll be on-track version wise for more things
[11:28:15] <elukey>	 yep
[11:33:36] <elukey>	 we use the binary release of Druid, but the druid-hdfs-storage extension now uses hadoop 2.8.x libraries, so they suggest to recompile it if needed
[11:34:10] <joal>	 makes sense - do you want help with that elukey ?
[11:34:52] <elukey>	 nono thanks
[11:35:13] <elukey>	 I hoped to make it work without this but it seems not working
[11:35:18] <joal>	 ok - let me know if there's anything I can do
[11:35:28] <wikibugs>	 (03CR) 10Andrew-WMDE: [C: 03+2] Change metric for TwoColConflict disables [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/611279 (https://phabricator.wikimedia.org/T257577) (owner: 10Awight)
[11:35:58] <wikibugs>	 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10JAllemandou) Thing to...
[11:36:01] <wikibugs>	 (03Merged) 10jenkins-bot: Change metric for TwoColConflict disables [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/611279 (https://phabricator.wikimedia.org/T257577) (owner: 10Awight)
[11:45:10] <elukey>	 joal: the main issue in this way is that we'll also need to (probably) upgrade druid as well right after hadoop
[11:45:44] <joal>	 elukey: only the hdfs-connector right?
[11:46:08] <elukey>	 joal: yes but it is a new version of the debian package etc..
[11:46:16] <joal>	 right
[11:46:19] <joal>	 :(
[11:46:21] <elukey>	 and upstream suggests to run mvn package after changing the main pom.xml
[11:46:28] <elukey>	 (of druid'ssource code)
[11:46:45] <elukey>	 let's see how it goes :)
[11:47:06] <elukey>	 I'll upgrade to bigtop when you are on holidays
[11:47:11] <elukey>	 just to have more fun :D
[11:47:54] <joal>	 I must say it would make me disapointed to not be part of that great adventure :)
[11:52:37] <joal>	 fdans: almost there! https://stats.wikimedia.org/#/all-wikipedia-projects/reading/unique-devices/normal|line|2018-03-16~2020-07-15|~total|daily
[11:53:00] <fdans>	 niiiiiiice!
[12:26:17] <elukey>	  Could not find artifact org.apache.hadoop:hadoop-hdfs-client:jar:2.6.0 in apache.snapshots
[12:26:46] <elukey>	 https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs-client
[12:26:49] * elukey cries in a corner
[12:29:43] <elukey>	 so hadoop-client is available in a lot of flavors https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client
[12:29:47] <elukey>	 but not hdfs-client?
[12:30:00] <mforns>	 hi team!
[12:30:08] <mforns>	 sorry to see you cry elukey :[
[12:30:48] <elukey>	 mforns: never a joy
[12:34:17] <joal>	 hi mforns :)
[12:44:28] <wikibugs>	 10Analytics-Radar, 10Operations, 10Product-Analytics, 10SRE-Access-Requests: Hive access for Sam Patton - https://phabricator.wikimedia.org/T248097 (10CDanis) 05Stalled→03Resolved @spatton I'm going to optimistically close this assuming that Turnilo access has been sufficient for you, please do reopen...
[14:08:17] <elukey>	 there is something that I don't understand
[14:08:48] <elukey>	 druid's pom.xml lists hadoop-hdfs-client among the dependencies, using ${hadoop.compile.version}
[14:09:22] <elukey>	 but in https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-hdfs-client/ I see only 3.x.x versions
[14:09:32] <elukey>	 and on maven central, the oldest is 2.8.0
[14:09:43] <elukey>	 so of course the build fails
[14:10:01] <elukey>	 the interesting thing is that in the cloudera repo, hadoop-hdfs has all versions
[14:13:48] <ottomata>	 elukey:  i'm not sure, but iirc there was some custom stuff we had to do to add hadoop and other libs to the druid deb?
[14:14:32] <ottomata>	 oh hm, no they are provided in the druid dist tarball
[14:14:35] <ottomata>	 hadoop-dependencies
[14:14:36] <elukey>	 ottomata: I already tried all, this time the druid extension for hdfs has been compiled with hadoop 2.8.5 libs and upstream suggests to re-build druid
[14:14:44] <ottomata>	 ah
[14:14:46] <ottomata>	 grr
[14:14:48] <ottomata>	 i see
[14:14:50] <elukey>	 yeah :(
[14:15:21] <elukey>	 I tried to re-add the cdh deps, to use all the parameters listed in https://druid.apache.org/docs/latest/operations/other-hadoop.html
[14:15:26] <elukey>	 but I keep getting errors
[14:15:48] <elukey>	 so I was hoping to test a custom binary release with cdh dependencies
[14:16:03] <elukey>	 but maybe there is another trick to make indexations workign
[14:16:30] <elukey>	 I haven't tried yet to set stuff like "mapreduce.job.classloader": "true" in the indexation json configs for example
[14:16:39] <elukey>	 (only tuned the middle manager's config file)
[14:18:34] <elukey>	 brb
[14:58:07] <wikibugs>	 (03CR) 10Mforns: "LGTM! Very Java :]" (034 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[15:15:42] <wikibugs>	 (03PS5) 10Ottomata: Overloaded methods to make working with default Refine related classes easier [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788
[15:23:19] <wikibugs>	 (03CR) 10Ottomata: Overloaded methods to make working with default Refine related classes easier (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788 (owner: 10Ottomata)
[15:37:31] <wikibugs>	 10Analytics, 10Structured Data Engineering, 10Structured-Data-Backlog (Current Work): Instrument MediaSearch results page - https://phabricator.wikimedia.org/T258183 (10Cparle)
[15:42:04] <wikibugs>	 10Analytics, 10Structured Data Engineering, 10Structured-Data-Backlog (Current Work): Instrument MediaSearch results page - https://phabricator.wikimedia.org/T258183 (10Cparle)
[15:50:13] <wikibugs>	 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey)
[16:00:04] <wikibugs>	 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10EBernhardson) This isn't only ranking models, but also general updates to the search indices that flow from analytics. This inclu...
[16:09:23] <wikibugs>	 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10elukey)
[16:23:18] * elukey off!
[16:37:23] <joal>	 Leaving that here in case it talks to anyone - https://twitter.com/steveklabnik/status/1283402702056230912
[16:43:31] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[16:43:47] <wikibugs>	 (03PS20) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609)
[16:44:48] <wikibugs>	 (03CR) 10Ottomata: Add classes to use EventStreamConfig with EventSchemaLoader to aide in event ingestion tasks (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/603582 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata)
[17:38:54] <wikibugs>	 10Analytics-Clusters, 10Discovery-Search, 10Operations, 10vm-requests: VM request for Analytics -> Elastic Search ML models update - https://phabricator.wikimedia.org/T258189 (10EBernhardson) I also just remembered while considering this, we need to have an instance per datacenter. The current applications...
[18:19:24] <wikibugs>	 10Analytics, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10Ottomata)
[18:19:45] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10Ottomata)
[18:56:58] <wikibugs>	 (03CR) 10MNeisler: Add the new VisualEditorFeatureUse fields to eventlogging whitelist (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/607309 (https://phabricator.wikimedia.org/T256048) (owner: 10MNeisler)
[19:11:59] <cdanis>	 elukey: any chance you're still around for the day? no worries if not, just a question about creating kerberos principals
[19:12:20] <joal>	 hi cdanis - he signed-off a few hours ago :)
[19:12:37] <cdanis>	 thanks! I'll find him tomorrow
[19:40:42] <wikibugs>	 (03PS1) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818)
[19:42:40] <wikibugs_>	 (03PS2) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818)
[19:56:59] <wikibugs>	 (03PS6) 10Ottomata: Overloaded methods to make working with default Refine related classes easier [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/607788
[19:57:01] <wikibugs>	 (03PS3) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818)
[20:25:31] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10Ottomata) Actually, I think https://gerrit.wikimedia.org/r/613251 should just work, at least for `_schema`.    For non EventLogging metawiki schemas, we ca...
[20:25:53] <wikibugs>	 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10SNowick_WMF) @JAlleman...
[20:27:44] <wikibugs>	 (03PS4) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818)
[20:34:24] <wikibugs>	 (03PS5) 10Ottomata: Refine - Don't merge Hive schema by default when reading input data [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/613251 (https://phabricator.wikimedia.org/T255818)
[21:20:30] <wikibugs>	 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10Nuria) @SNowick_WMF I...