[18:32:18] <musikanimal>	 hello, anyone know what's up with https://phabricator.wikimedia.org/T150524 ?
[18:32:36] <musikanimal>	 that phab is referring to the dump files, but this also seems to be affecting the API
[18:33:00] <musikanimal>	 I got a few complaints about data missing since 9 November, when typically data for the 10th/11th would have shown up by now
[18:35:12] <wikibugs>	 10Analytics, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 10th November - https://phabricator.wikimedia.org/T150524#2788564 (10MusikAnimal) The issue also seems to be affected the API ([[ https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Cat/...
[19:18:11] <elukey>	 musikanimal: I can see data from 10/11 now in dumps, and the example that you provided in the task seems returning data
[19:18:15] <elukey>	 am I missing something? 
[19:19:00] <musikanimal>	 maybe it's caching, I only have data up to 11/9, and others are reporting the same
[19:19:14] <musikanimal>	 I am very curious how that caching works, not the first time this has come up
[19:19:36] <musikanimal>	 where some users see data and others do not, from the API directly
[19:20:27] <elukey>	 ah you are right sorry
[19:20:42] <elukey>	 (I was indeed missing something :)
[19:22:13] <elukey>	 ah snap the cassandra jobs are waiting for data
[19:22:15] <elukey>	 this is why
[19:23:40] <joal>	 !log Launch 0028421-161020124223818-oozie-oozi-B to cover for webrequest-load hours 19-20 missing on 2016-11-10
[19:23:41] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[19:23:42] <elukey>	 so basically when data is refined (IIRC, I am more focused on ops stuff :) we set a _SUCCESS file in each hour dir 
[19:23:47] <elukey>	 hola joal !
[19:23:49] <elukey>	 he is the man
[19:23:52] <joal>	 Ha !
[19:23:57] <joal>	 Didn't know you were here
[19:23:59] <joal>	 Just the mess
[19:24:08] <elukey>	 I saw the cassandra load jobs waiting
[19:24:14] <joal>	 There are 2 hours missing for last Thursday
[19:24:23] <elukey>	 gneee
[19:24:36] <elukey>	 like this one?
[19:24:49] <elukey>	 sorry, two days ago?
[19:24:58] <elukey>	 because IIRC we deployed the refinery
[19:25:04] <joal>	 When nuria restarted job, 2 hours were lost in translationg
[19:25:16] <elukey>	 all right makes sense
[19:25:39] <elukey>	 and these are reported in the email
[19:25:59] <joal>	 If you backlog chan, she had trouble with oozie, and discussing with ottomata they restarted a job starting at 21:00, while hours 19:00 and 20:00 were not done
[19:26:08] <joal>	 correct elukey 
[19:26:19] <joal>	 Should bge fixed in 2 hours hopefully
[19:26:27] <elukey>	 2016-11-10T19 and 2016-11-10T20
[19:26:29] <elukey>	 snap
[19:26:50] <elukey>	 I am going to update the task
[19:26:53] <elukey>	 thanks joal !
[19:26:55] <joal>	 Classical oozie mistake: you kill the job without having checked which of the materialized jobs had finished
[19:27:15] <elukey>	 (I am waiting my flight for seville, two hours of delay :/)
[19:27:23] <joal>	 Mwarf elukey :()
[19:27:27] <joal>	 Best of luck with that
[19:27:47] <joal>	 I'll arrive tomorrow, around 20:00 (depending on delay and so on)
[19:27:58] <joal>	 If you agree elukey, let's have diner tomorrow together :)
[19:29:55] <wikibugs>	 10Analytics, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 10th November - https://phabricator.wikimedia.org/T150524#2788564 (10elukey) Sorry for the inconvenience!   We deployed the new Analytics refinery on Thursday and one of its task was to kill and restart the Oozie jobs responsib...
[19:30:09] <elukey>	 joal: +2
[19:30:23] <joal>	 Yay elukey :)
[19:30:45] <elukey>	 musikanimal: I commented the task, really really sorry for the incovenience. Joal just fixed it, we should get data back during then next hours
[19:32:50] <elukey>	 also answered to the email on analytics@
[19:34:09] <joal>	 Thanks a lot elukey for having managed communication
[19:34:23] <joal>	 elukey: I sent an email to alert to explain the thing.
[19:36:23] <elukey>	 I missed it yesterday :(
[19:36:28] <elukey>	 what are the materialized jobs?
[19:37:49] <joal>	 elukey: oozie materializes a job in waiting state at a given point in time
[19:38:10] <joal>	 You can look at any webrequest load coordinator, there are jobs in orange, waiting
[19:38:17] <joal>	 Those are materialized but not started jobs
[19:38:20] <musikanimal>	 thanks!
[19:38:33] <joal>	 np musikanimal, sorry for the problem
[19:38:39] <musikanimal>	 all good
[19:40:33] <elukey>	 joal: ahh ok! The jobs waiting! So basically those were not restarted as well
[19:41:02] <joal>	 elukey: you need to set your start time at the point where the jobs were waiting (or were running, if you kill)
[19:43:03] <wikibugs>	 10Analytics, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 10th November - https://phabricator.wikimedia.org/T150524#2790334 (10elukey) p:05Triage>03Normal
[19:45:41] <joal>	 elukey: online checkin, done, boarding passes ready: getting there !
[19:46:31] <elukey>	 :)
[19:51:51] <joal>	 good luck for your flight elukey 
[19:53:02] <elukey>	 thankssss