[00:11:13] Analytics-Kanban, Editing-Analysis, Documentation: Remove outdated docs regarding dashboard info - https://phabricator.wikimedia.org/T137883#2584684 (Quiddity) @Nuria note: we rarely ever delete pages. Outdated pages should just be tagged with the template {{Historical}}, e.g. `{{TNT|Historical}}` or... [03:09:04] PROBLEM - cassandra CQL 10.64.48.117:9042 on aqs1003 is CRITICAL: Connection refused [03:23:04] RECOVERY - cassandra CQL 10.64.48.117:9042 on aqs1003 is OK: TCP OK - 0.002 second response time on port 9042 [07:43:18] good morning to you cassandra [07:46:35] ERROR [SharedPool-Worker-21] 2016-08-26 03:03:08,936 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to: [07:46:38] java.lang.OutOfMemoryError: Java heap space [07:47:20] happened during SSTable reads [07:47:22] sigh [08:05:12] so I am pretty sure at this point that new traffic is hitting AQS since the 17th (or around that date) query data that spans more SSTables [08:05:17] triggering timeouts [08:05:51] Joseph did some traffic analytics with spark and webrequest logs, but maybe I can do something similar with bare beeline [09:33:16] Analytics, Beta-Cluster-Infrastructure, Services, scap, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#2585345 (elukey) >>! In T116206#2582429, @elukey wrote: > Thanks for reporting, this is my bad since analytics_hadoop_hosts is not in hiera labs. Since this value s... [11:08:43] FYI I am going to restart the JVM daemons on the Hadoop cluster [11:08:58] let me know if you see/incurr in problems! [11:31:13] !log suspended all the oozie bundles via Hue [11:31:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [11:32:50] !log stopped camus on analytics1027 [11:32:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [12:04:22] still waiting for the oozie jobs to finish [12:04:27] I should I have done it sooner [12:04:28] sigh [12:04:33] I always forget [13:09:30] !log oozie, hive-server and hive-metastore restarted for security upgrades [13:09:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [13:27:44] all right now I need to complete the work with analytics100[12] [13:27:48] the masterzzz [13:30:39] !log restarted yarn-resourcemanager on analytics1001 [13:30:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [13:33:47] !log restarted hadoop-hdfs-namenode on analytics1001 [13:33:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [13:35:56] transition from 1001 to 1002 happened correctly [13:35:58] all good [13:36:40] I'll wait 10/20 minutes and then I'll restart 1002's daemons [13:45:56] !log restarted yarn-resourcemanager on analytics1002 (1001 back to active) [13:45:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [13:48:15] !log restarted hadoop-hdfs-namenode on analytics1002 (1001 back to active) [13:48:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [13:52:27] !log re-enabling camus and oozie [13:52:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [13:54:09] all good, I don't see anything exploding [15:31:30] mforns: couple mins late for 1 on 1 [15:31:46] nuria_, oh! I thought you had cancelled it [15:31:52] np, joining now [15:32:11] mforns: i moved it to today 30 mins after satndup [15:32:46] mforns: do you see it on your calendar? [16:01:16] nuria_: You might be interested in evaluating this grant proposal -- https://meta.wikimedia.org/wiki/Grants:Project/Ilya/ScalaWiki_data_processing_toolbox [16:05:17] bd808: Thanks for the ping. reading fast seems like he is trying to ease access to data but some of it (like editing data) cannot be easily extracted until we are done with the project of "mediawiki edit history identity reconstruction". His project seems a better fit for jupyter notebooks and connectors to existing datasources. We already use oozie, spark [16:05:17] and scala extensively, that is itself provides no value in the absence of a datasource. [16:06:28] *nod* IT also sounds like they want to run this on tool labs somehow but I'm not sure how that would actually work [16:07:45] I don't like the stack divergence (flink, kite) either. Makes a prod promotion path more difficult [16:13:20] bd808: and those are too technical [16:13:32] bd808: the proposal should focus on jupyter and data connectors [16:13:43] bd808: to currently existing data sources [16:13:51] this is why I pinged you about it :) [16:13:59] I knew you would have opinions [16:14:05] and domain knowledge [16:14:05] bd808: will comment to that fact and madhuvishy i am sure would be interested [16:14:16] bd808: well... opinions for sure [16:14:39] bd808: madhuvishy can also comment as to teh value proposition of jupyter notebooks she is thought about that quite a bit [16:14:47] I'll poke madhu and yuvi too [16:15:57] bd808: https://meta.wikimedia.org/wiki/Grants_talk:Project/Ilya/ScalaWiki_data_processing_toolbox#Concerns_and_suggestions_for_refining_proposal [16:16:23] bd808: also proposal needs refinement when it comes as to what is the goal [16:16:35] bd808: thanks again for the ping [16:17:17] thank you! Now that we are open to funding grants for software things we need to get good feedback from folks who actually understand the problem domains [16:17:44] or we will accidentally fund things that sound good but can't actually be accomplished [16:17:55] (not that we don't do that internally, but ...) [16:27:06] (PS25) Mforns: [WIP] Refactor Mediawiki History scala code [analytics/refinery/source] - https://gerrit.wikimedia.org/r/301837 (https://phabricator.wikimedia.org/T141548) (owner: Joal) [16:28:05] a-team: going offline, will check later on if the cluster is up and running (even if I don't expect any issue) [16:28:16] bye elukey! nice weekend [16:28:24] I'll be afk for a bit so in case of fire fire please ping me on hangouts :) [16:28:30] ok :] [16:28:30] byyyeeeeeee [16:28:36] nice weekend to you too [18:17:50] (PS1) Mforns: Support passing the exploded values by file path [analytics/reportupdater] - https://gerrit.wikimedia.org/r/306966 (https://phabricator.wikimedia.org/T132481) [18:22:50] (PS1) Mforns: Disable the deprecated option by_wiki [analytics/reportupdater] - https://gerrit.wikimedia.org/r/306968 (https://phabricator.wikimedia.org/T132481) [18:32:49] have a nice weekend team! see ya :] [18:35:02] (CR) Mforns: [C: -1] "We need to deploy other changes before this one." [analytics/reportupdater] - https://gerrit.wikimedia.org/r/306968 (https://phabricator.wikimedia.org/T132481) (owner: Mforns) [20:05:26] Analytics-Dashiki, Analytics-Kanban: Bookmarkable date filters for browser stats dashboard - https://phabricator.wikimedia.org/T143689#2587326 (Nuria) [20:09:02] (PS1) Nuria: [WIP] Bookmark for tab/graph/date [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) [20:19:07] (PS2) Nuria: [WIP] Bookmark for tab/graph/date [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) [21:31:02] (PS3) Nuria: [WIP] Bookmark for browser dashboard reagrding graph and time [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) [21:37:01] (PS4) Nuria: Bookmark for browser dashboard reagrding graph and time [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) [21:37:32] (PS5) Nuria: Bookmark for browser dashboard regarding graph and time [analytics/dashiki] - https://gerrit.wikimedia.org/r/306980 (https://phabricator.wikimedia.org/T143689) [22:13:27] hey nuria_. I shared the proposal for industry ranking with you. [22:13:46] Let's jump on a call so I can tell you where we are with it, before you spending a lot of time on it. :) [22:38:41] (PS4) MaxSem: Import from GitHub [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/306699 (https://phabricator.wikimedia.org/T143048) [23:53:25] (CR) Yurik: [C: 2 V: 2] Import from GitHub [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/306699 (https://phabricator.wikimedia.org/T143048) (owner: MaxSem)