[00:00:46] ragesoss: those pages that give 404s might have never been visited since July 2015 (until now, obviously) [00:00:52] https://wikitech.wikimedia.org/wiki/Analytics/AQS/Pageviews#Gotchas [00:01:25] musikanimal: why do some pages show 0 views instead? [00:02:33] that's a good point, I see zeros across the board for that page [00:03:42] not sure how to explain that [00:05:26] what is really frustrating is that there's no way to tell the difference between a page with no views and a bad request. [00:16:24] oh, wow, I had no idea that it returns error messages, like {"type":"https://restbase.org/errors/not_found","title":"Not found.","method":"get","detail":"The date(s) you used are valid, but we either do not have data for those date(s), or the project you asked for is not loaded yet. Please check https://wikimedia.org/api/rest_v1/?doc for more information.","uri":"/analytics.wikimedia.org/v1/pageviews/per-article/fr.wikisource/all-access [00:16:24] /user/Voyages%2C_aventures_et_combats%2FChapitre_26/daily/2017041200/2017053100"} [00:16:40] in the browser, it just shows the Firefox page not found message [01:23:27] Hey, the browser analytics dashboard doesn't seem to have updated on 2017-05-28 (it's weekly); is this a known issue/thing or should I file a task? [04:42:14] !log removed some old scap revs for the Analytics refinery on stat1002 to free space [04:42:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:19:55] thanks elukey for scap cleaning :) [08:41:15] !log Restarted last_access_uniques-monthly-coord after bug correction and deploy [08:41:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:53:22] (03PS11) 10Joal: Add unique devices project-wide oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352181 (https://phabricator.wikimedia.org/T143928) [11:24:27] (03PS1) 10Joal: Rename last_access_uniques for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 [11:57:39] (03PS12) 10Joal: Add unique devices project-wide oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352181 (https://phabricator.wikimedia.org/T143928) [11:59:09] (03PS2) 10Joal: Rename last_access_uniques for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 [12:10:28] (03PS1) 10Joal: Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 [12:11:22] taking a break a-team - later [14:48:16] !log Restarted webrequest-load-bundle after deploy [14:48:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:59:08] (03PS25) 10Ottomata: EventLogging JSON -> Hive [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [14:59:32] (03CR) 10Ottomata: EventLogging JSON -> Hive (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [15:01:52] ping joal elukey standuuppppp :] [15:02:13] (03CR) 10Ottomata: "Joal, I still need to figure out the _SUCCESS file stuff, we should talk about that. I also might need to clean up a bit, and add more co" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/346291 (https://phabricator.wikimedia.org/T161924) (owner: 10Joal) [15:04:21] hey all, I'm still at the hospital, might be here for a while [15:06:50] * joal hugs milimetric [15:24:10] * mforns hugs milimetric [15:30:34] thanks yall :) relaying hugs to Steph [15:48:33] how does one find out when a new mediawiki-history monthly snapshot is ready to be queried? [15:49:33] Hi HaeB - Easiest is to track oozie jobs [15:49:52] the "sqoop" thing on yarn? [15:49:57] nope [15:50:06] HaeB: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0044069-170424154741156-oozie-oozi-C/ [15:50:44] HaeB: sqoops on yarn are run to import data from MySQL (1 job per table per wiki, so many of them) [15:51:22] HaeB: Once data is imported, we process it to provide the wmf.mediawiki_history table - This "processing" bit is the oozie I sent you [15:53:30] joal: i see, thanks! might be useful to mention at https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Mediawiki_history [16:00:23] joal: random other question, i tried to use the mediawiki history tables to get per-wiki counts on their min/max/mean # of edits per day since jan 1. The numbers looked mostly reasonable, but there were quite a few missing wikis like dewiki, frwiki, etc. is that intentional? https://phabricator.wikimedia.org/P5383 [16:02:41] HaeB: doc updated - good enough? [16:02:44] Hi ebernhardson [16:03:24] hi :) [16:04:02] the talk of history tables just reminded me of that thing from earlier this month :) [16:04:10] ebernhardson: It's a known issue, we use new labs databases to feed the system, and it;s not yet fully replicated [16:04:47] ok, reasonable enough. good to know [16:05:02] ebernhardson: reasoin for labs first was to ensure puvlic first [16:05:43] ebernhardson: I follow https://phabricator.wikimedia.org/T153743 for updates on new data, and regularly check for new wikis imports [16:21:23] (03CR) 10Mforns: [V: 032 C: 032] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/352181 (https://phabricator.wikimedia.org/T143928) (owner: 10Joal) [18:42:35] joal: that was informative, but doesn't quite answer the question in the same direct way as the link you sent.. [18:42:49] .. i tried to summarize here, is this correct? https://wikitech.wikimedia.org/w/index.php?title=Analytics/Data_Lake/Edits/Mediawiki_history&diff=1761120&oldid=1761113