[01:00:59] AndyRussG: you can do that, sure, but please delete the data when you're done with it. [01:08:50] milimetric: ah cool, thx! will that be a good way to query stuff more quickly (rather than say downloading locally)? [01:09:07] Also, feel like commenting on the query I'm working on? It's at the very end of https://etherpad.wikimedia.org/p/T152122_notes [01:09:16] (something's not working just yet...) [01:09:18] thx!!! [01:16:09] AndyRussG: I don't think you can select * and then other fields, I think you have to list everything one by one. And I'm not sure what the syntax is for creating a table off of a select, your guess is as good as mine. Might be easier to export the data with hive to an hdfs folder under your user /user/andyrussg and then build an external table on top of [01:16:09] that [01:16:17] no idea - and I gotta run, sorry, good luck [01:16:34] milimetric: k thx! aarg found it, missing ' [01:16:43] * AndyRussG plays silly music :) [01:16:47] thx much, have fun!! [09:43:58] !log Restarted yesterday failed oozie webrequest-load jobs (upload, text, misc, hours 21, 22,23) [09:43:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:01:58] thanks joal :) [10:02:05] np elukey :) [10:02:20] elukey: pageviews were stuck on yesterday, no good [10:04:12] :( [12:08:31] 06Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#2841363 (10ArielGlenn) [12:09:53] 06Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#2841163 (10JAllemandou) Known issue - Some jobs have been failing yesterday night, we are currently rerunning them. Data should flow-in today. [12:25:36] * elukey lunch! [14:00:48] (03PS1) 10Addshore: WIP DNM ExactValueQuantityProcessor [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/324906 [14:13:14] joal: should I have just restarted them? I figured we'd have to look into them first [14:13:49] milimetric: we can look at fail jobs afteer yes, but not breaking the flow first [14:13:51] is there a type of alert where I should not try restarting first? Or is that the universal fix? "Sir, have you tried turning it off and on again?" [14:14:16] I guess, can I mess something up by restarting? [14:14:22] 'cause if not, I'll just restart everything I see fail [14:14:40] milimetric: restart first is usually the good thing to do [14:14:58] hm, that begs the question: why don't we restart automatically once? [14:15:09] milimetric: If failures don't happen just after a deploy, it is usually due to cluster overload or machines failures [14:16:13] milimetric: oozie can do that, yes :) [14:16:26] * milimetric is not at all convinced that someone with front-end aesthetics should be touching ops work [14:16:30] :) [14:16:32] :D [14:17:32] like, if something doesn't work in my world I restart it. But that uses like 1mW of power vs. the cluster which probably eats a baby cat every time I push a button in hue [14:30:36] milimetric: I'm looking deeper into the issue you reported and couldn't find any reason for which it happened yesterday night :( [14:38:29] hm [14:47:01] milimetric: it's weird, cluster was not overloaded at the time, and job still failed [14:52:48] gremlins? [14:53:51] milimetric: looking at https://grafana-admin.wikimedia.org/dashboard/db/analytics-hadoop?from=now-7d&to=now [14:54:04] resource manager was strongly loaded yesterday evening [14:56:01] mwarf milimetric -- Having a max-retry=2 or 3 for webrequest load would make sense [14:56:38] maybe 2 for now [14:56:48] because we don't want it to thrash if it's got a real problem [15:15:20] 06Analytics-Kanban, 13Patch-For-Review: Evaluate a unit testing framework and add tests for the formatter function - https://phabricator.wikimedia.org/T147440#2841716 (10elukey) I tried to refactor the code to make varnishkafka's source modular, with separate modules to unit test separately. Some of the work i... [15:15:29] defeat --^ [15:16:27] 06Analytics-Kanban, 13Patch-For-Review: Better Compiler warnings in Makefile - https://phabricator.wikimedia.org/T147436#2841717 (10elukey) Summary in https://phabricator.wikimedia.org/T147440#2841716 [15:19:29] 06Analytics-Kanban, 10Datasets-General-or-Unknown: pageviews files missing since yesterday 1st December - https://phabricator.wikimedia.org/T152193#2841731 (10elukey) 05Open>03Resolved Closing task, please re-open it if data will not be there during the next 24 hours. Sorry for the trouble and thanks for t... [15:21:43] 06Analytics-Kanban, 06Operations, 10Traffic, 13Patch-For-Review: Ganglia varnishkafka python module crashing repeatedly - https://phabricator.wikimedia.org/T152093#2841735 (10elukey) a:03elukey [15:23:19] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2782769 (10elukey) https://github.com/yandex/ClickHouse/tree/master/debian [15:23:32] 06Analytics-Kanban: Puppetize clickhouse - https://phabricator.wikimedia.org/T150343#2841741 (10elukey) a:03elukey [15:31:30] joal: can we talk for a sec? [15:31:38] milimetric: sure [15:31:39] OMW [16:08:30] 10Quarry, 06Labs, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry is allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841828 (10yuvipanda) [16:14:41] 10Quarry, 06Labs, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841853 (10yuvipanda) [16:14:52] AndyRussG: Hi [16:15:13] AndyRussG: Looks like you have 2 pyspakr running on the cluster, but possibly doing nothing [16:18:02] 10Quarry, 06Labs, 10Tool-Labs: Clarify Tool Labs' rules to see if Quarry and PAWS are allowed to be hosted there - https://phabricator.wikimedia.org/T152212#2841856 (10yuvipanda) [16:43:04] ping AndyRussG ! [16:47:07] mforns, milimetric, do we take a few minutes discussing code structure, or next week? [16:57:49] joal, was eating, you mean code review of user/page? [16:57:55] joal: hi! [16:58:06] I was thinking of global code architecture mforns [16:58:11] ok [16:58:56] sure joal, let's talk! milimetric? [16:59:05] joal: just one running now AFIK [16:59:21] omw batcave [16:59:24] just reading thru some doc... I can shut it off meanwhile [16:59:28] AndyRussG: https://yarn.wikimedia.org/cluster/scheduler [17:00:01] mforns: we're in there :) [17:00:26] AndyRussG: I think that when closing pyspark, it sometimes forget to kill it's cluster app -- Haow bad [17:00:53] AndyRussG: I guess we can kill the first app you launched, right? [17:01:33] joal the second one is the one I ctrl-C'd... was just trying (perhaps the wrong way) to check the version [17:01:42] I can kill both and restart one that I'll actually use [17:03:57] AndyRussG: As you wish :) [17:06:03] joal: K all gone I think... I'll restart when I'm ready to do stuff again... thx! [17:06:35] No prob :) Thanks AndyRussG [17:21:26] 10Analytics, 06Security-Team: Statistics on Captcha success/failure rate - https://phabricator.wikimedia.org/T152219#2842060 (10Reedy) [17:23:40] going afk! have a good weekend! [17:45:50] milimetric, hey :] what was the new Dashiki convention on config page urls? Dashiki:Config:Blah or Config:Dashiki:Blah ? [17:46:27] 06Analytics-Kanban, 06Operations: setup/install thorium/wmf4726 as stat1001 replacement - https://phabricator.wikimedia.org/T151816#2842125 (10Cmjohnson) [17:48:37] mforns: oh, no, just use the same one for now [17:48:45] we'll move them all when we deploy the extension [17:48:48] milimetric, ok ok [17:48:50] thx! [17:48:53] I'm not sure what happens when the namespace doesn't exist and you create a page in it [17:49:00] (I think bad things) [17:57:38] 10Analytics, 10MediaWiki-General-or-Unknown: Make aggregated MediaWiki Pingback data publicly available - https://phabricator.wikimedia.org/T152222#2842133 (10Legoktm) [18:03:45] 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, and 2 others: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2842146 (10ellery) @RobH Thank you for the thorough investigation :). Now we know that the stat machines cannot accommodate a top-of-the-line GPU. Tha... [18:09:43] (03PS1) 10Mforns: Change value column name for better coloring [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324946 (https://phabricator.wikimedia.org/T126358) [18:20:00] (03CR) 10Mforns: [C: 032 V: 032] "Self merging minor change." [analytics/limn-ee-data] - 10https://gerrit.wikimedia.org/r/324946 (https://phabricator.wikimedia.org/T126358) (owner: 10Mforns) [18:41:32] 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, and 2 others: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2842281 (10RobH) So the Titan line is right out at 10.5" long. There are others that are half that size in the next series down: http://www.geforce.... [19:17:59] nuria: re: https://phabricator.wikimedia.org/T144431#2838506, what did you have in mind there? in what context would you put that on wikitech? [19:19:15] urandom: it doesn't have to be now, i would add a "pitfalls" documentation so errors are not repeated when it comes to define partitions [19:19:33] urandom: but again, it can wait for resolution and next steps [19:21:05] i see [19:21:43] nuria: yeah, that makes sense [19:21:49] "considerations" [19:21:56] or somesuch [19:22:03] or "antipatterns" [19:22:22] ya [19:22:55] urandom: there might be a better forum than wikitech as it seems to be a cassandra antipattern [19:23:49] nuria: yeah, that was one reason i was asking: if the idea is to generalize, then pushing something upstream would probably be better [19:24:00] urandom: ya, ya agreed [19:24:01] rather than maintain something separately on wikitech [19:24:43] hi! Mmm anyone know offhand what's the easiest way to make a simple plot from a DataFrame on ipython notebooks? [19:24:46] urandom: yes, i think wikitech should pertain to wmf systems as in we might refactor restbase and if so a document that explains why will point to this antipattern doc [19:24:49] nuria: haha: http://cassandra.apache.org/doc/latest/data_modeling/index.html [19:25:01] TODO [19:25:02] urandom: gotta love OS [19:25:11] urandom: ya, it is a "minor item" [19:25:18] urandom: can be documented later [19:25:31] urandom: man ... [19:25:48] nuria: i am "upstream" here, so i'm not going to throw too many stones at this glass house [19:25:55] but, yeah [19:26:07] * urandom doesn't document worth a damn [19:26:32] urandom: ay ay ... [20:01:53] bye team! nice weekend to you! [20:18:38] 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, and 2 others: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2842537 (10RobH) The Dell PowerEdge R730 can add up to two add on GPUs, via their own ordering during the time of system build. We don't have any exp... [20:26:01] neilpquinn: btw, I'm working on sending out an update about the data we were showing you and Helen, so I moved the table [20:26:13] neilpquinn: it's no longer milimetric.mediawiki_history, it's wmf.mediawiki_history [20:26:25] and wmf.mediawiki_user_history and wmf.mediawiki_page_history [20:27:00] I'll explain in the update, but I'm taking a long time writing it, so just a heads up [20:27:09] * milimetric afk for a while [20:38:20] (03PS31) 10Milimetric: Script sqooping mediawiki tables into hdfs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/306292 (https://phabricator.wikimedia.org/T141476) [20:49:58] (03CR) 10Yurik: [C: 04-1] Record sum of all wikis for geo tag counts (032 comments) [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324822 (owner: 10MaxSem) [20:51:19] (03CR) 10MaxSem: Record sum of all wikis for geo tag counts (031 comment) [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324822 (owner: 10MaxSem) [21:00:21] 10Analytics, 06Discovery, 06Discovery-Analysis, 03Interactive-Sprint: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2842692 (10mpopov) a:03mpopov [21:00:32] 10Analytics, 06Discovery, 06Discovery-Analysis (Current work), 03Interactive-Sprint: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2829353 (10mpopov) [21:04:40] (03CR) 10Yurik: Record sum of all wikis for geo tag counts (031 comment) [analytics/discovery-stats] - 10https://gerrit.wikimedia.org/r/324822 (owner: 10MaxSem) [21:05:37] yurik, you realize that my pattern is how we're recording it in abunch places already? [21:05:45] and it's causing no problems [21:15:00] milimetric: thank you! ironically, I'm currently working on thinking through your response about my query, and I'll send a response to that eventually as well :) [21:20:33] MaxSem, my fault - i didn't catch it earlier. The more I learn about grafana, the more "good practices" i develop [21:20:46] MaxSem, i hope we will be able to rename them at some point [21:20:56] yurik, all doesn't match *.* [21:21:24] MaxSem, yes, but as i said - it makes it harder to work with [21:21:48] it was a mistake to put them together IMO [21:22:54] also, MaxSem, at some point we may decide to provide total per project as a series (makes it faster to work wih) - in which case we again should have a new 2nd level key - e.g. pagesperproj [21:23:19] and we wouldn't want to put it together with "all" [21:56:55] 10Analytics-Dashiki, 06Analytics-Kanban: Remove dependency on available-projects.json file hosted in labs - https://phabricator.wikimedia.org/T136120#2323947 (10Nuria) [23:13:43] 10Analytics, 06Discovery, 06Discovery-Analysis (Current work), 03Interactive-Sprint: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2843097 (10mpopov) @Nuria: okie dokie, here's the Hive query that counts up successful tile requests: ```lang=sql ADD JAR hdfs...