[02:54:58] <icinga-wm>	 PROBLEM - Webrequests Varnishkafka log producer on cp5006 is CRITICAL: connect to address 10.132.0.106 port 5666: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[06:34:55] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Implement inequality metrics for WikiStats - https://phabricator.wikimedia.org/T248964 (10Quasipodo) Thank you @Milimetric and @MGerlach, I appreciate that. Let's see how everything evolves, I'm currently deeply involved in finishing up my master's thesis.
[06:48:47] <fdans>	 morning!
[07:03:47] <elukey>	 o/
[07:44:44] <wikibugs>	 (03PS2) 10Awight: Move data to semi-permanent path [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596436
[07:45:33] <wikibugs>	 (03PS3) 10Awight: Move data to semi-permanent path [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596436
[07:48:37] <wikibugs>	 (03PS2) 10Awight: Refresh notebooks with April 2020 data [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596480 (https://phabricator.wikimedia.org/T252507)
[07:49:16] <wikibugs>	 (03PS6) 10Awight: Analyze row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507)
[08:23:03] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10fdans)
[08:28:12] <wikibugs>	 10Analytics, 10Analytics-Kanban: Create job that backfills Pagecounts-EZ (2011 - 2016) data via hadoop correcting issues - https://phabricator.wikimedia.org/T252857 (10fdans)
[08:28:31] <wikibugs>	 10Analytics, 10Analytics-Kanban: Create job that backfills Pagecounts-EZ (2011 - 2016) data via hadoop correcting issues - https://phabricator.wikimedia.org/T252857 (10fdans) a:05Milimetric→03fdans
[08:29:01] <wikibugs>	 (03PS1) 10Fdans: Add special explode UDTF that turns EZ-style hourly strings into rows [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/596605 (https://phabricator.wikimedia.org/T252857)
[08:39:43] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10fdans) @Isaac thank you for the insight!  After a bunch of discussion among the team, we see how the addition of the page ID would add a lot of value...
[08:55:37] <wikibugs>	 (03PS40) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777)
[08:55:59] <wikibugs>	 (03PS41) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777)
[08:56:12] <wikibugs>	 (03CR) 10Fdans: "Just added page id" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) (owner: 10Fdans)
[09:05:14] <icinga-wm>	 PROBLEM - aqs endpoints health on aqs1007 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:05:14] <icinga-wm>	 PROBLEM - aqs endpoints health on aqs1008 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:05:18] <icinga-wm>	 PROBLEM - aqs endpoints health on aqs1009 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:05:22] <icinga-wm>	 PROBLEM - aqs endpoints health on aqs1006 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:05:30] <icinga-wm>	 PROBLEM - aqs endpoints health on aqs1004 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:05:46] <icinga-wm>	 PROBLEM - aqs endpoints health on aqs1005 is CRITICAL: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:07:47] <fdans>	 loading wikistats, seems druid metrics are not responding
[09:08:10] <fdans>	 cc elukey 
[09:08:15] <elukey>	 yeah I know what's happening
[09:08:53] <elukey>	 !log restart druid brokers on Analytics Public
[09:08:55] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[09:09:41] <elukey>	 it is an occurrence of T226035
[09:09:42] <stashbot>	 T226035: Dropping data from druid takes down aqs hosts  - https://phabricator.wikimedia.org/T226035
[09:09:54] <elukey>	 fdans: wikistats should work now
[09:10:15] <fdans>	 oh interesting
[09:10:19] <fdans>	 thank you elukey 
[09:10:38] <icinga-wm>	 RECOVERY - aqs endpoints health on aqs1007 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:10:40] <icinga-wm>	 RECOVERY - aqs endpoints health on aqs1008 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:10:44] <icinga-wm>	 RECOVERY - aqs endpoints health on aqs1009 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:10:48] <icinga-wm>	 RECOVERY - aqs endpoints health on aqs1006 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:10:58] <icinga-wm>	 RECOVERY - aqs endpoints health on aqs1004 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:11:12] <icinga-wm>	 RECOVERY - aqs endpoints health on aqs1005 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/aqs
[09:11:54] <elukey>	 the fix is to upgrade to a newer version of druid
[09:12:07] <elukey>	 I'll try to work on this next month
[09:12:13] <elukey>	 it has become a big problem
[09:18:32] <elukey>	 I am wondering if we are missing a key metric that could unveil what the problem is
[09:27:57] * elukey brb
[09:48:16] <elukey>	 upstream wrote some docs, maybe it can help https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#connection-pool-guidelines
[10:25:03] <elukey>	 had a look into metrics again, but still a mistery
[10:25:32] <elukey>	 at around 9UTC the procedure to drop the snapshot started
[10:25:49] <elukey>	 from the historical point of view, it seems as if no more connection is processed (query I mean)
[10:26:04] <elukey>	 from the broker, connections are slowly piling up
[10:26:35] <elukey>	 then when brokers are restarted, everything restart working
[11:14:34] <joal>	 this is bizarre :(
[11:14:54] <joal>	 as if connection piling was unable to unpile
[11:16:26] <elukey>	 joal: I went back to https://phabricator.wikimedia.org/T226035#5804481, and the BLOCKED threads are around 20-ish, that would explan the mess
[11:16:51] <elukey>	 after lunch I'll try to follow the code and see if I can get anything out of it
[11:17:06] <elukey>	 for $reasons I didn't do it last time, will try this one
[11:17:23] <elukey>	 but in any case, I'll prioritize druid 0.18
[11:17:36] <joal>	 ack elukey - this is still very not cool :(
[11:17:59] <elukey>	 we are lagging a lot from upstream, I am pretty sure that this will disappear when we'll upgrade
[11:18:15] <joal>	 very possi
[11:18:25] <joal>	 elukey: just looked at a locked thread
[11:18:57] <joal>	 elukey: the lock happen on a Cache object
[11:19:53] <joal>	 meh nevermind - need to investigate befroe talkiung
[11:19:55] <joal>	 sorry
[11:19:59] <elukey>	 yeah, during io.druid.server.QueryResource.doPost
[11:21:17] <elukey>	 sorry, query lifecycle
[11:21:18] <elukey>	 https://www.javadoc.io/doc/io.druid/druid-server/0.12.2/io/druid/server/QueryLifecycle.html
[11:21:37] <elukey>	 that says
[11:21:37] <elukey>	 Class that helps a Druid server (broker, historical, etc) manage the lifecycle of a query that it is handling. It ensures that a query goes through the following stages, in the proper order:
[11:21:41] <elukey>	 Initialization (initialize(Query))
[11:21:44] <elukey>	 Authorization (authorize(HttpServletRequest)
[11:21:46] <elukey>	 Execution (execute()
[11:21:49] <elukey>	 Logging (emitLogsAndMetrics(Throwable, String, long)
[11:22:16] <elukey>	 and the blocked thread enters io.druid.server.QueryLifecycle.execute(QueryLifecycle.java:254)
[11:22:46] <elukey>	 it must be a stupid bug already resolved :(
[11:22:52] <elukey>	 anyway, going afk for a bit :)
[11:22:53] <elukey>	 o/
[11:23:13] <joal>	 \o
[11:32:58] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10CristianCantoro) My 2 cents:   - Redirects are very useful and something to be taken into account when working on Wikipedia - the paper linked by Isaa...
[11:36:53] <wikibugs>	 (03PS1) 10Awight: Query: nojs vs new user [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596625
[11:44:14] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10CristianCantoro) >>! In T251777#6140199, @CristianCantoro wrote: > My fear is that selecting a page by id would not be exactly equivalent to select it...
[12:00:58] <wikibugs>	 (03PS6) 10Awight: Clean up table persistence [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225
[12:16:31] <elukey>	 joal: need to do an invasive change for Druid, do you have a min to assist? :)
[12:16:38] <joal>	 ksure!
[12:16:50] <joal>	 elukey: tell me more - invasive changes look fun :)
[12:17:19] <milimetric>	 joal: not sure there's a way to do exactly what I'm saying in the comment here:
[12:17:20] <milimetric>	 https://www.irccloud.com/pastebin/WtOWuBBH/
[12:17:35] <milimetric>	 because -1 MONTH will get me back to 2020-02-01
[12:18:19] <milimetric>	 but I can't think of a way to do 2020-03-01, or is that offset(0, "MONTH")?
[12:18:34] <joal>	 milimetric: fun question :)
[12:18:38] <joal>	 milimetric: I don't know :)
[12:18:43] <milimetric>	 ok, I'm just gonna test :)
[12:18:59] <milimetric>	 life is like a box of oozie datasets... you never know what you gonna get
[12:19:09] <elukey>	 joal: context in https://phabricator.wikimedia.org/T252771
[12:19:18] <joal>	 milimetric: something to know about tests - if you go for dryrun instead of run, It'll write you a bunch of XML containing dataset dependency details
[12:19:30] <milimetric>	 oh!
[12:19:34] <milimetric>	 I saw that but never tried it
[12:19:35] <elukey>	 since I need to roll restart druid anyway, I am planning also to swap the /var/lib/druid location
[12:19:36] <milimetric>	 ok, cool
[12:19:38] <elukey>	 to /srv
[12:19:53] <joal>	 elukey: reading
[12:20:11] <wikibugs>	 (03PS4) 10Milimetric: Use new page move incremental updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773)
[12:21:06] <elukey>	 related change is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/596633/
[12:21:46] <joal>	 elukey: just for my understanding - for raid10-partman reasons, we want big SSDs to be mounted on /srv
[12:21:59] <joal>	 So we need to move druid data content folders
[12:22:05] <joal>	 elukey: about right?
[12:22:26] <joal>	 also milimetric - I'd try offset(0, "MONTH") first - feels reasonable ;)
[12:22:35] <milimetric>	 yep, that's what I'm doing
[12:23:04] <elukey>	 joal: yes the new hosts will follow a more standard partman recipe, so in order to keep mental sanity in puppet we'd need to move to the new scheme also with the "old" hosts. 
[12:23:49] <joal>	 maeks sense elukey, even I doubt mental sanity has ever be something we can count on for any of us ;)
[12:24:04] <elukey>	 joal: that's another good point too long to discuss :D
[12:24:40] <elukey>	 joal: does the puppet change make sense?
[12:24:49] <joal>	 ok let's go elukey  - How do you want to proceed? Trying to make a plan - Stop druid, unmount/remount in different location using partman, update druid config, restart druid?
[12:25:05] <elukey>	 yes it is the one listed in the taskk
[12:25:08] <elukey>	 one node at the time
[12:25:19] <joal>	 of course
[12:25:28] * joal should have read the tasks-comments
[12:26:22] <joal>	 question about puppet elukey - Should we also move druid.request.logging.dir?
[12:27:04] <joal>	 elukey: change looks good, but I have no knowledge of how many places we should bump
[12:27:45] <elukey>	 I think that druid.request.logging.dir is good for now
[12:27:52] <elukey>	 let's do the minimal change
[12:28:07] <elukey>	 so in the gerrit change I pasted a grep of /etc/druid
[12:28:11] <joal>	 elukey: again a dumb q then: why indexer log if not the other one?
[12:28:29] <elukey>	 joal: I am basically following what we have now
[12:28:53] <elukey>	 in the future we can switch from that location
[12:29:22] <elukey>	 I changed only the places in which we have /var/lib/druid in the config
[12:29:33] <elukey>	 does it make sense?
[12:29:39] <joal>	 ok I didn't get that - makes sense indeed
[12:30:04] <elukey>	 basically I changed these
[12:30:04] <elukey>	 elukey@druid1001:~$ grep -rni /var/lib/druid /etc/druid/
[12:30:04] <elukey>	 /etc/druid/common.runtime.properties:11:druid.indexer.logs.directory=/var/lib/druid/indexing-logs
[12:30:07] <elukey>	 /etc/druid/historical/runtime.properties:13:druid.segmentCache.locations=[{"path":"/var/lib/druid/segment-cache","maxSize"\:2748779069440}]
[12:30:10] <elukey>	 /etc/druid/middlemanager/runtime.properties:9:druid.indexer.task.baseTaskDir=/var/lib/druid/task
[12:30:25] <elukey>	 afaics those 3 are the only ones that we need to change
[12:30:51] <joal>	 I follow you now - thanks for the explanations - it helps :)
[12:30:59] <joal>	 Ready elukey :)
[12:31:38] <elukey>	 !log move superset config to druid1002 (was druid1003) to ease maintenance
[12:31:41] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[12:34:40] <milimetric>	 joal: heh, amazingly, -1 and 0 with "MONTH" result in the same set of URIs, both starting from 2020-03-02T00... which is not what we want :)
[12:34:43] <milimetric>	 so... yeah
[12:34:56] <joal>	 hm milimetric 
[12:35:01] <joal>	 Bizarre !
[12:35:04] <milimetric>	 I know
[12:35:12] <milimetric>	 I checked it twice 'cause it made no sense
[12:35:29] <milimetric>	 I'll eat breakfast and think about it some more, sorry, was trying to get you a decent code review earlier than later
[12:36:41] <joal>	 np milimetric 
[12:37:04] <joal>	 elukey: do you want us to batcave, so that I can follow?
[12:37:49] <elukey>	 joal: ah sorry didn't see it, 1003 done
[12:37:57] <joal>	 np elukey :)
[12:38:00] <joal>	 all good so far?
[12:38:07] <elukey>	 Loading segment cache file [13180/21874][/srv/druid/segment-cache
[12:38:10] <elukey>	 seems so :)
[12:38:13] <joal>	 \o/
[12:39:25] <joal>	 elukey: cluster views 2 nodes still
[12:40:33] <elukey>	 from the coord?
[12:40:44] <joal>	 yes elukey 
[12:40:54] <joal>	 3 nodes now :)
[12:40:59] <elukey>	 the historical is still loading, should be good in a sec (hopefully)
[12:41:02] <elukey>	 :)
[12:41:31] <elukey>	 ok I'll leave it for ~10/15 mins before proceeding with the others
[12:41:32] <joal>	 3 nodes look good
[12:41:35] <joal>	 ack!
[12:41:43] <elukey>	 thanks!
[12:41:53] <joal>	 elukey: np, not done anything actually :)
[12:42:04] <shrini>	 hello all
[12:42:07] <elukey>	 joal: pebcak prevention scheme, it is a lot of work :)
[12:42:15] <joal>	 :)
[12:42:29] <shrini>	 I am looking the page https://stats.wikimedia.org/#/ta.wikisource.org/contributing/edits/normal|bar|2-year|page_type~content*non-content|monthly
[12:42:35] <elukey>	 when all nodes are done, we could be good to add the new nodes joal 
[12:42:45] <shrini>	 how to get the counts for proofread and validated pages?
[12:42:47] <joal>	 WOW :) Great!
[12:43:30] <shrini>	 I just built a dashboard using prometheus and grafana for indic wikipedia sites here
[12:43:33] <shrini>	 http://139.59.47.5:3000/d/kx1Pb36Zz/indic-wiki-stats
[12:43:36] <joal>	 Hi shrini - I think t he dimensions you ask for are not present in wikistats, not in the underlying dataset we use
[12:43:49] <shrini>	 like to build the same for wikisource
[12:43:55] <shrini>	 oh ok joal 
[12:44:14] <joal>	 shrini: how do you get the values you are looking for?
[12:44:29] <shrini>	 in current wikistats, can we have comparision charts like in the grafana dashboard I sent?
[12:44:43] <joal>	 shrini: I actually don't know where to find 'proofread' and 'validated' :)
[12:44:55] <shrini>	 joal: ok.
[12:45:03] <shrini>	 Here is another query
[12:45:23] <shrini>	 can we have comparision graphs with current wikistats?
[12:45:33] <shrini>	 check this dashboard - http://139.59.47.5:3000/d/kx1Pb36Zz/indic-wiki-stats
[12:45:40] <joal>	 shrini: We don't yet have wikis comparison in wikistats :( It'll come for sure but it is not yet here
[12:45:49] <shrini>	 ok. thanks
[12:46:10] <shrini>	 just asked to decide whether to continue with this prometheus/grafana idea or not
[12:46:29] <joal>	 ack shrini
[12:46:47] <shrini>	 friends said wikistats has all features and why yet another tool ?
[12:47:01] <shrini>	 I think this will be useful to get some more insights
[12:50:32] <fdans>	 shrini: the use case you're showing in grafana matches exactly part of the functionality we'll be adding to Wikistats, but this is still being designed
[12:52:01] <fdans>	 joal: I'm getting a weird feeling seeing that 2014 pagecounts-ez dumps are about twice as heavy as current
[12:52:09] <fdans>	 longer long tail i guess?
[12:52:31] <joal>	 hm - why would it fdans? same data, no?
[12:52:41] <shrini>	 fdans: thanks for the info
[12:53:01] <joal>	 ah excuse me fdans - current data would be lighter than 2014?
[12:53:07] <fdans>	 joal: yep
[12:53:07] <shrini>	 Is there any volunteering task for these, for beginners ?
[12:53:16] <joal>	 hm
[12:53:25] <fdans>	 https://usercontent.irccloud-cdn.com/file/yrBsKC0b/joal
[12:53:46] <fdans>	 shrini I'm sure we can find something if you want to help!
[12:54:01] <joal>	 fdans: try with equivalent dates - Jan-1st is not a heavy day in traffic ;)
[12:54:34] <wikibugs>	 (03PS7) 10Awight: Analyze row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507)
[12:54:36] <shrini>	 fdans: sure. will do.
[12:54:44] <fdans>	 joal: yea but 1/4th uncompressed?
[12:58:16] <joal>	 hm
[12:58:26] <joal>	 fdans: bots?
[12:59:09] <fdans>	 joal: will look further. Time for lunchi for me :D
[12:59:15] <joal>	 o/
[12:59:59] <fdans>	 shrini: I'll take a good look at easy tasks, in the meantime you can scroll through the Wikistats column and see if there's something that you would like to work on https://phabricator.wikimedia.org/tag/analytics/
[13:02:19] <shrini>	 fdans: ok. will explore there
[13:02:57] <joal>	 elukey: I've been doing a relatively heavy query to duid using turnilo - all good on my side: )
[13:03:29] <elukey>	 joal: yep! proceeding with druid1001, will change turnilo's config for a moment
[13:03:35] <joal>	 ack
[13:03:42] <elukey>	 !log move turnilo config to druid1002 to ease druid maintenance
[13:03:44] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:12:04] <elukey>	 1001 is up
[13:15:34] <elukey>	 !log turnilo back to druid1001
[13:15:36] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:23:09] <elukey>	 !log roll restart of the Druid analytics cluster to pick up new openjdk + /srv completed
[13:23:12] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:25:00] <joal>	 \o/ elukey 
[13:25:25] <joal>	 Once again, /me bow to elukey, master of analytics-prod
[13:25:29] <cdanis>	 elukey: how soon until netflow data is fresh again? :)
[13:26:45] <elukey>	 cdanis: in a sec
[13:27:55] <elukey>	 cdanis: is it needed now?
[13:28:08] <cdanis>	 oh no, nothing urgent
[13:28:21] <elukey>	 ack ack thanks
[13:28:40] <cdanis>	 just had been in the middle of messing with something (recv buffer size on the samplicator that passes out data to nafcctd + fastnetmon)
[13:28:49] <elukey>	 ahh snap bad timing
[13:28:59] <elukey>	 I am completing some invasive restart + maintenance
[13:29:01] <elukey>	 will finish soon
[13:29:06] <cdanis>	 np!
[13:29:09] <elukey>	 cdanis: also two more nodes coming!
[13:29:12] <cdanis>	 \o/
[13:30:40] <elukey>	 cdanis: it is weird, two out of three indexers are running, do you see a drop in data?
[13:31:03] <cdanis>	 elukey: 'past 1h' in netflow is 30 minutes stale
[13:31:12] <cdanis>	 https://w.wiki/Quu
[13:31:25] <elukey>	 I think it is in a weird state, will kill/restart
[13:31:41] <joal>	 sounds a good idea elukey 
[13:34:22] <wikibugs>	 (03PS8) 10Awight: Analyze row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507)
[13:43:16] <cdanis>	 elukey: still seems stuck
[13:44:13] <elukey>	 cdanis: yes there is a little problem, the indexers are not liking the new /srv location, but I have to understand why
[13:44:36] <cdanis>	 okay, let me know if i can help :)
[13:54:15] <elukey>	 -Djava.io.tmpdir=/var/lib/druid/tmp
[13:54:18] * elukey cries in a corner
[13:56:11] <elukey>	 where the hell it is set
[13:57:21] <joal>	 elukey: do you need me or can I leave for shopping?
[13:57:31] <joal>	 elukey: I can leave later if you want me to help
[13:58:26] <elukey>	 nono please go
[13:58:34] <joal>	 ack elukey
[14:13:09] <elukey>	 cdanis: should be working now
[14:13:52] <cdanis>	 yeah, looks like it is catching up :)
[14:13:57] <cdanis>	 thanks Luca!
[14:14:48] <elukey>	 so it seems that druid by defaults uses a java option that points to a location on disk that I don't want, and I am not sure where it gets set sigh
[14:14:58] <elukey>	 I added a symlink for the moment
[14:17:01] <wikibugs>	 10Analytics: Add new Druid nodes to analytics and public clusters - https://phabricator.wikimedia.org/T252771 (10elukey) druid100[1-3] have been ported to the new scheme, all good except the following bit that I didn't know: all druid daemons are executed with `-Djava.io.tmpdir=/var/lib/druid/tmp`, and I don't k...
[14:20:57] <cdanis>	 lol, charming
[14:26:34] <elukey>	 # Default JVM opts for all druid processes
[14:26:35] <elukey>	 DRUID_JVM_OPTS=${DRUID_JVM_OPTS:-"-Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/var/lib/druid/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"}
[14:26:40] * elukey cries in a corner - part 2
[14:27:23] <elukey>	 ok so DRUID_EXTRA_JVM_OPTS are appended afterwards to DRUID_JVM_OPTS, might be easy
[14:28:52] * elukey afk for a bit!
[14:34:32] <wikibugs>	 (03PS1) 10Awight: Start investigating same-revision conflicts [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596653 (https://phabricator.wikimedia.org/T246440)
[14:56:47] <wikibugs>	 (03CR) 10Andrew-WMDE: [C: 03+2] Query: nojs vs new user [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596625 (owner: 10Awight)
[15:00:30] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Dropping data from druid takes down aqs hosts - https://phabricator.wikimedia.org/T226035 (10Nuria) +1 to upgrade
[15:10:05] <fdans>	 joal: hmmmm it seems that in old pagecouonts-ez there's both En.z and en.z
[15:10:14] <fdans>	 weeeeird
[15:10:21] <joal>	 fdans: that would be so uncool
[15:10:31] <fdans>	 joal: indeed
[15:10:54] <fdans>	 joal: a lot of wikis seem to have both
[15:11:04] <fdans>	 the answer is probably buried in a 3000 line perl script
[15:11:55] <joal>	 fdans: a forgotten toLower?
[15:12:01] <fdans>	 omg joal there's DE.z, De.z and de.z...
[15:12:14] <fdans>	 I don't know, they might mean something
[15:12:27] <fdans>	 but that's the most logical answer
[15:12:40] <fdans>	 hmmm actually no
[15:12:52] <fdans>	 because most wikis are Titlecase
[15:13:05] <fdans>	 this is all unique wiki values for a 2014 dump:
[15:13:13] <fdans>	 https://www.irccloud.com/pastebin/Yk4ohBnE/
[15:41:16] <wikibugs>	 (03PS7) 10Andrew-WMDE: Clean up table persistence [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 (owner: 10Awight)
[15:46:04] <wikibugs>	 (03CR) 10Andrew-WMDE: [C: 03+1] "Looks good to me. I just made a few slight stylistic changes. Feel free to merge it but please make sure I didn't break anything first." [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 (owner: 10Awight)
[16:10:31] <joal>	 Hi milimetric - Would you have some time now to brainbounce on mediawiki-history-reduced?
[16:11:10] <elukey>	 joal: on Monday we'll be able to add the new druid nodes :)
[16:11:19] <joal>	 awesome :)
[16:12:38] <elukey>	 in theory there shouldn't be any pre-requisite to start a node no?
[16:13:21] <joal>	 hm - I don't think so - when new node comes-in, coordinator will realize and ask it to handle some data currently handled by others, is all
[16:14:20] <elukey>	 yeah
[16:14:32] <elukey>	 I am tempted to add them now
[16:15:00] <joal>	 I understand elukey - we nontheless are late friday :S
[16:15:38] <elukey>	 yes yes 
[16:15:48] <elukey>	 I said "tempted", will not do it :)
[16:16:00] <joal>	 ok no milimetric around - will leave for now, and write my thouhts :)
[16:16:07] <joal>	 have a good weekend team
[16:17:45] <elukey>	 will be afk for a bit as well, and read later if anything needs me :)
[16:55:35] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10Isaac) > we see how the addition of the page ID would add a lot of value to the dump without treading too much into feature creep territory Thanks! I...
[16:58:32] <wikibugs>	 (03CR) 10Nuria: [C: 04-1] "I have several comments but my main one is that I do not think you can setup "generically" a sparse timeseries filler this way." (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) (owner: 10Mforns)
[18:16:08] <mforns>	 yes nuria, we can talk about that next standup also
[20:54:00] <wikibugs>	 10Analytics, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T252915 (10srishakatux)
[20:58:58] <wikibugs>	 10Analytics, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T252915 (10Nuria) @srishakatux data for may will be available by june 1st, makes sense? As data for may is complied after the month has finished
[21:52:57] <wikibugs>	 (03PS5) 10Milimetric: Use new page move incremental updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773)
[21:54:43] <wikibugs>	 (03CR) 10Milimetric: "There doesn't seem to be a way to do what we want in Oozie.  I'd be happy to be wrong but I went up and down the docs and there are fancy " (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) (owner: 10Milimetric)
[23:04:40] <wikibugs>	 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10MMiller_WMF) @Ottomata -- I now have the notebook running after logging out and in again, but I'm having trouble running queries via the `wmfdata` package.  I'm using the `mariadb.run` command and getting this:  {F31820466}  M...