[02:05:59] <wikibugs>	 10Analytics, 10MediaWiki-API: There is not an easy way to tag API requests by application for analytics - https://phabricator.wikimedia.org/T181862#3804738 (10Legoktm) Can you not use a unique user-agent for your application?
[02:47:18] <wikibugs>	 (03PS1) 10GoranSMilovanovic: Non-productionized runs 02 Dec 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/394737
[02:47:28] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Non-productionized runs 02 Dec 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/394737 (owner: 10GoranSMilovanovic)
[02:49:01] <wikibugs>	 (03PS2) 10GoranSMilovanovic: Non-productionized runs 02 Dec 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/394737
[02:49:25] <wikibugs>	 (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Non-productionized runs 02 Dec 2017 [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/394737 (owner: 10GoranSMilovanovic)
[03:03:20] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review the WDCM public datasets and allow them to access published datasets on stat1005 - https://phabricator.wikimedia.org/T181871#3805060 (10GoranSMilovanovic)
[03:04:22] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review the WDCM public datasets and allow them to access published datasets on stat1005 - https://phabricator.wikimedia.org/T181871#3805075 (10GoranSMilovanovic)
[03:04:41] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review the WDCM public datasets and allow them to access published datasets on stat1005 - https://phabricator.wikimedia.org/T181871#3805060 (10GoranSMilovanovic)
[03:04:56] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review the WDCM public datasets and allow them to access published datasets on stat1005 - https://phabricator.wikimedia.org/T181871#3805060 (10GoranSMilovanovic)
[03:05:22] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review the WDCM public datasets and allow them to access published datasets on stat1005 - https://phabricator.wikimedia.org/T181871#3805060 (10GoranSMilovanovic)
[03:06:49] <HaeB>	 Cluster has been very slow since yesterday (a daily Hive query that usually takes less than 30 min took more than 6h, a niced query that previously took between 1 and 6h now still running after 27h)
[03:07:03] <HaeB>	 I assume that's because of all the monthly queries running on the 1st
[03:53:58] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Please review the WDCM public datasets and allow them to access published datasets on stat1005 - https://phabricator.wikimedia.org/T181871#3805097 (10GoranSMilovanovic) Here is the README file that would accompany these data sets in a case...
[03:58:18] <Shilad>	 Hey all: I need to run some jobs on the cluster. Looks pretty busy right now. Anybody know if this a back log for some reason? I'm wondering if I should wait or just submit them now.
[04:35:52] <nuria_>	 Shilad: the first 3/4 days of the month cluster is busiest
[04:36:04] <nuria_>	 Shilad: so waiting will be wise
[04:36:12] <Shilad>	 Good to know. I'll hold off until things clear up
[04:36:27] <Shilad>	 Thanks!
[09:17:24] <joal>	 Hi chan, as of this morning the cluster has almost recovered from a super-busy first day of month
[09:17:57] <joal>	 It's still a bit late on some jobs, but has recovered most yesterday's accumulated jobs
[09:29:40] * elukey waves to joal 
[09:48:52] <elukey>	 fdans: https://twitter.com/thomasfuchs/status/933447631514169344
[09:56:51] <wikibugs>	 10Quarry, 10Discovery, 10VPS-Projects, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/  (like Quarry) - https://phabricator.wikimedia.org/T104762#1426314 (10Lucas_Werkmeister_WMDE) >>! In T104762#2635939, @Multichill wrote: > With the current SPARQL setup it's easy to share...
[10:13:04] <elukey>	 joal: I followed up on yesterday's druid realtime issue
[10:13:21] <elukey>	 all the tasks that have failed have been scheduled on druid1001
[10:13:49] <elukey>	 its middlemanager reports errors for index_hadoop_webrequest_2017-12-01T14 and all replicas for index_realtime_banner_activity_minutely_2017-12-01T13
[10:14:05] <elukey>	 I think that equalDistribution played a role in here
[10:15:39] <elukey>	 the overlord tried to schedule equally tasks, but since they were failing quickly I think that it was always seeing all the middle managers as they were "free"
[10:16:09] <elukey>	 so all the tasks ended up on druid1001, that was running probably a "rogue" middlemanager :D
[10:16:26] <elukey>	 when tasks got scheduled on different middlemanagers, all good
[10:17:06] <elukey>	 so I am 99% sure that if we drain a middle manager before stopping it we should not see any issue with realtime indexers
[10:17:40] <joal>	 elukey: Mwahahahah ! Javascript :)b
[10:18:01] <elukey>	 :D
[10:18:32] <joal>	 elukey: We'll try that on monday, when trying to restart the rest of the cluster :)
[10:18:44] <joal>	 Thanks a lot for investigating that elukey, it makes a lot more sense :)
[10:19:01] <elukey>	 I wanted to get your opinion on this, it seems the only plausible explanation
[10:19:34] <joal>	 elukey: maybe not the only one, but at least a very reasonable one given the data you gathered: )
[10:20:00] <elukey>	 :)
[10:20:17] <elukey>	 all right, the other problem could be that from what I gathered the indexer machinery doesn't like zookeeper down
[10:20:30] <elukey>	 but it shouldn't be a problem if one single node goes down
[10:20:40] <elukey>	 anyhoooow
[10:20:48] <elukey>	 sorry for bothering you on a saturday :)
[10:21:09] <elukey>	 have a good weekend! Talk with you on monday (and thanks for checking and reporting the health of the cluster!)
[10:21:11] <joal>	 no bother mate, it's to have a team as ours :)
[10:21:18] <joal>	 later !
[11:47:20] <joal>	 !log Rerun unique_devices-per_project_family-monthly-wf-2017-11
[11:47:21] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:39:31] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10MediaWiki-API, 10Easy, and 2 others: ApiJsonSchema implements ApiBase::getCustomPrinter for no good reason - https://phabricator.wikimedia.org/T91454#3805526 (10Framawiki)
[14:34:17] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations, 10Upstream: Understand difference between author_name and name in gerrit - https://phabricator.wikimedia.org/T177890#3805653 (10Aklapper) Don't think I can find out myself: ``` $:acko\> grep -r user_name --exclude="*.json" . ./grimoirelab/panels/READ...
[14:51:40] <wikibugs>	 10Analytics-Tech-community-metrics, 10Developer-Relations, 10Upstream: Understand difference between author_name and name in gerrit - https://phabricator.wikimedia.org/T177890#3805694 (10Aklapper) Comparison of the three values on the front page: {F11142761}
[23:00:47] <wikibugs>	 10Quarry, 10Discovery, 10VPS-Projects, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/  (like Quarry) - https://phabricator.wikimedia.org/T104762#1426314 (10Smalyshev) > We want to be able to save query results and share them  Tabular data on Commons should be a good place f...
[23:57:23] <wikibugs>	 10Quarry, 10Discovery, 10VPS-Projects, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/  (like Quarry) - https://phabricator.wikimedia.org/T104762#3806292 (10Daniel_Mietchen) >>! In T104762#3806240, @Smalyshev wrote: > Tabular data on Commons should be a good place for it, no...