[03:16:19] PROBLEM - Check if active EventStreams endpoint is delivering messages. on scb2002 is CRITICAL: CRITICAL: No EventStreams message was consumed from http://scb2002.codfw.wmnet:8092/v2/stream/recentchange within 10 seconds. [03:46:29] RECOVERY - Check if active EventStreams endpoint is delivering messages. on scb2002 is OK: OK: An EventStreams message was consumed from http://scb2002.codfw.wmnet:8092/v2/stream/recentchange within 10 seconds. [05:56:15] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka eventlogging instances delivery failures - https://phabricator.wikimedia.org/T198070 (10elukey) As reference, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/314336/ is the change that Brandon did a while ago to adjust... [06:21:25] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add interface::add_ip6_mapped { 'main': } to all the Analytics hosts - https://phabricator.wikimedia.org/T199180 (10elukey) [06:29:41] as FYI I am adding static IPv4 to IPv6 mappings to all our hours [06:29:43] *hosts [06:30:25] basically the IPv6 address of a lot of hosts now is subnet + mac address, meanwhile we have some puppet code that "standardize" it using the ipv4 address instead of the MAC [06:30:39] for example, stat1005 becomes 2620:0:861:108:10:64:53:30/64 [06:30:42] etc.. [06:31:03] it shouldn't cause any trouble but let me know otherwise :) [06:39:17] now it is the turn of analytics100[2,3] [06:39:25] 1001 of course uses ip6 for zookeeper :D [06:57:09] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add interface::add_ip6_mapped { 'main': } to all the Analytics hosts - https://phabricator.wikimedia.org/T199180 (10elukey) [06:58:22] 1001 done as well [07:11:31] Hi elukey :) [07:11:49] elukey: This means we're better at v6 compatibility? [07:12:04] hello :) [07:12:18] yes this is also part of the efforts to add the ip6 firewall rules to the routers [07:13:17] makes sense :) [07:14:10] in theory the ip6 addresses are meant to auto configure with the subnet coming from the router (RA packet) and a combination of mac address (48 bits) and some padding [07:14:30] (so basically 64 bits coming from the router, the other 64 from the host) [07:14:39] but since this depends on the NIC installed etc.. [07:14:59] we standardized using the ip4 address as last 64 bits, statically configured [07:15:10] so we can put it in DNS safely etc.. [07:15:23] all our host afaics are ip6 capable, but most of them are autoconfigured [07:15:53] Super clear explanation :) Thanks Luca :) [07:16:43] :) [07:16:57] last bit that is interesting [07:17:27] Arzhel has set up the first ip6 firewall rules on the routers for analytics, but rather than blocking they only log what "escapes" from the rules [07:17:32] so we know what is missing [07:17:49] some of our services are already using ip6 [07:17:53] like hadoop with zookeeper [07:17:55] really? [07:17:57] Wow [07:18:01] I had no idea :) [07:18:15] yeah I think it depends if the AAAA record is present in the DNS for a given host [07:18:35] puppet goes via ip6 too [07:18:51] and it is interesting since it is not currently filtered :P [07:20:02] (brb a min) [07:40:38] elukey: About the event-streams alerts - Have it fixed itself or hqave you done something? [07:43:29] nope self recovered, didn't check yet [07:43:34] k [07:44:03] joal: if you are ok I'd rollout the ip6 changes to all the hadoop workers [07:44:14] works for me :) [07:44:18] ack :) [08:02:45] change is rolling out [08:07:42] firewall rules should be ok [08:20:12] \o/ [08:20:31] !log Update AQS druid backend datasource to 2018-06 [08:20:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:33:46] joal: ip6 changes live on aqs1004, ready to proceed with other nodes [08:34:34] proceeding now [08:34:40] +1 elukey [08:57:03] joal: done! [08:58:06] elukey: minimal test on hosts work fine ;) [08:59:15] super :) [08:59:23] ip6 config rolled out as well [09:09:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add interface::add_ip6_mapped { 'main': } to all the Analytics hosts - https://phabricator.wikimedia.org/T199180 (10elukey) [09:11:03] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) Last changed applied by Arzhel, including merging common-infrastructure4 to analytics-in4 [09:35:07] for some reason stat1004 had some troubles this morning [09:35:29] I thought it was related to ip6 but instead there was a lot of load due to processes hang for /mnt/hdfs [09:37:39] time to enable the /mnt/hdfs readability check everywhere [09:43:36] elukey: I used the mount by mistake, and it was stuck [09:45:49] joal: do you mean now or this morning? [09:46:26] works for me now [09:46:53] (I am rolling out the alarm now) [09:47:17] (it doesn't catch all issues but it is better than nothing) [09:48:58] elukey: I used it his morning, and it was hanging [09:50:35] ah yes okok, it was due to the mount went awol [09:50:46] yup [09:52:58] elukey: yarn UI unresponsive [09:53:03] elukey: anything going on there? [09:56:25] not that I am aware of [09:56:25] weird [09:57:37] elukey: Got an error page [09:57:46] elukey: Request from 78.227.114.48 via cp1058 cp1058, Varnish XID 114285111 [09:57:49] Error: 503, Backend fetch failed at Tue, 10 Jul 2018 09:57:01 GMT [09:58:17] pretty sure it is due to the ip6 change [09:58:32] even if it shouldn't have [09:58:53] all the websites are working, so it shouldn't be thorium [09:59:39] elukey: do I need to ask mu ISP to upgrade?? :-P [10:02:56] joal: yarn is back :) [10:03:20] !log bounce yarn RM on an100[12], some socket errors after the ip6 interface rollout [10:03:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:03:33] Thanks elukey :) [10:03:36] It'll me [10:03:42] It'll HELP me :) [10:04:02] it is indeed back :) [10:04:09] I checked the status of yarn after the change but there were some socket errors [10:04:20] but via tcpdump I didn't see anything flowing [10:04:22] weird [10:04:27] :-( [10:05:07] it is a bit slow though [10:06:31] back timing out [10:11:36] so jobs are flowing, hue works fine afaics [10:15:21] as a test, I am rolling restart the yarn nodemanagers (it is seamless so no issue) [10:15:43] I am wondering if the an1001's UI times out due to unresponsive node managers to http queries (or similar) [10:16:39] elukey: possible - also elukey, cluster is busy :) [10:19:57] the annoying thing is that as always in those logs I can't find anything [10:20:26] elukey: May I help? [10:21:37] if you have any ideas :) [10:30:53] elukey: arh missed the lat message [10:31:06] elukey: I don't have special ideas - I can just help with logs if you want :) [10:35:16] so there's something that makes the yarn ui hang [10:35:22] if you can find any trace in the logs [10:36:39] https://grafana-admin.wikimedia.org/dashboard/db/analytics-hadoop?panelId=14&fullscreen&orgId=1 [10:36:45] could be realted elukey --^ ?? [10:37:02] (nit: grafana-admin will be deprecated soon) [10:37:03] :) [10:37:14] duely noted [10:39:28] so it might be related, but it might also be an unrelated increase in GCs due to activity.. [10:43:56] !log restart map reduce history server on an1001 as attempt to see if related with yarn.w.o unresponsiveness [10:43:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:44:15] 10Analytics, 10MinervaNeue, 10Readers-Web-Backlog, 10Design: Sticky header instrumentation - https://phabricator.wikimedia.org/T199157 (10phuedx) [10:50:39] elukey: looks like yarn UI is disoriented :S [10:55:34] this is really weird [10:58:30] 10Analytics, 10Operations, 10hardware-requests: eqiad: (2) hardware refresh for analytics1003 - https://phabricator.wikimedia.org/T198685 (10MoritzMuehlenhoff) p:05Triage>03Normal [11:01:13] so it must be related to the ip6 change [11:01:14] for sure [11:10:42] joal: from your perspective the cluster is running fine right? [11:21:41] 10Analytics: Piwik user account for Wikimedia.org.il - https://phabricator.wikimedia.org/T199046 (10Itzike) @fdans , yes, I'm talking about piwik.wikimedia.org. We want to use piwiki's stats on our chapter website, and I wonder if instead of installing and maintaining our own piwiki's server, we can use the glo... [11:25:33] going to eat something, brb! [11:26:19] elukey: only concern I have is with UI, the rest is working fine for me indeed [11:34:22] joal: now it seems working fine [11:34:23] whatttt [11:35:17] ok I don't really have a clue [11:36:21] (going afk for a bit) [12:14:50] seems stable now [12:15:00] not sure in what state it was in [12:15:10] maybe long lived connections getting weird? [13:35:58] i/ [13:35:59] o/ [13:36:01] elukey: a thought: [13:36:11] we shoudl probably change the mirror maker alerts to be ops, not analytics [13:36:17] esp for main [13:38:25] 10Analytics: Piwik user account for Wikimedia.org.il - https://phabricator.wikimedia.org/T199046 (10fdans) Thanks for clarifying @Itzike. The subdomain piwik.wikimedia.org is used only for analytics sites (like [[ http://stats.wikimedia.org/v2 | Wikistats ]]). But @Milimetric mentioned yesterday that there is a... [13:45:39] ottomata: o/ [13:45:45] sure, now that they are stable I guess so [13:47:43] gonna add some wikitech docs [13:47:46] think we don't have any mirror maker ones [13:47:55] ottomata: whenever you have time :) https://phabricator.wikimedia.org/T199109 [13:55:38] aye proceed elukey makes sense! [14:06:05] we also have refinery snapshots on hdfs from 2014 [14:06:20] not a big deal but maybe we should clean them up a bit? [14:08:41] o yea a-team, I gotta take Lauren to the airport now so I'm missing standup, but I'll be back later and send an e-scrumptious [14:09:19] elukey: yes clean them up! [14:09:41] there was some backlogged task to automate that, but its annoying because you'd also have to know which versions are still in use by various jobs [14:10:01] oh actually no, we know that pretty well [14:10:16] its just the verisioned files in artifacts that can be annoying [14:10:49] i guess any hdfs snapshot that has versions older than what we have in current artifacts can be removed [14:11:01] e.g. if the oldest version in current is 0.46 [14:11:17] then any hdfs snapshot that has stuff older than 0.46 can be removed [14:12:01] we can even think about something like "keep only the snapshot of this year" [14:14:00] (afk for a bit) [14:19:30] 10Analytics: Make WhitelistSanitization support arrays of structs, maps or other arrays - https://phabricator.wikimedia.org/T199230 (10mforns) [14:35:42] 10Analytics: Piwik user account for Wikimedia.org.il - https://phabricator.wikimedia.org/T199046 (10Nuria) Rather than analytics sites, sites that we maintain. Agreed with recommendation of using piwik community instalation. You could install it on labs if it is not there yet. [14:42:48] 10Analytics: Piwik user account for Wikimedia.org.il - https://phabricator.wikimedia.org/T199046 (10Itzike) I searched under tools (https://tools.wmflabs.org/admin/tools) and the only tool I found was this: https://tools.wmflabs.org/file-reuse-piwik/ If someone can install and manage this service, I'm sure many... [14:46:38] (03PS1) 10Mforns: Add MobileWikiAppiOSUserHistory to EL whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/444894 (https://phabricator.wikimedia.org/T195269) [14:53:21] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Partially purge MobileWikiAppiOSUserHistory eventlogging schema - https://phabricator.wikimedia.org/T195269 (10mforns) Hi all, I added MobileWikiAppiOSUserHistory to EventLogging sanitization white-list. Please review that everythin... [15:04:57] 10Analytics: Bot Identification: Inconsistent data in #all-sites-by-os-and-browser for IE7 - https://phabricator.wikimedia.org/T148461 (10Nuria) Following up on this, our prior version of ua-parser was missclassifying this traffic as IE7, the traffic looks automated in nature but the true classification of the... [15:06:04] ping ottomata milimetric fdans [15:14:27] 10Analytics: Bot Identification: Inconsistent data in #all-sites-by-os-and-browser for IE7 - https://phabricator.wikimedia.org/T148461 (10Tbayer) >>! In T148461#4412178, @Nuria wrote: > Following up on this, our prior version of ua-parser was missclassifying this traffic as IE7, the traffic looks automated in n... [15:26:34] (03CR) 10Mforns: [C: 032] Make bar-chart and line-chart resilient to breakdowns with null values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443640 (https://phabricator.wikimedia.org/T198630) (owner: 10Mforns) [15:37:26] ottomata: I'm struggling with panda data frames and the clickstream data :P [15:37:50] I feel like I threw myself in without learning how the data frames even work [15:38:12] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) [15:39:16] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add interface::add_ip6_mapped { 'main': } to all the Analytics hosts - https://phabricator.wikimedia.org/T199180 (10ayounsi) [15:39:18] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) [15:42:15] (03CR) 10Mforns: [V: 032 C: 032] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443592 (https://phabricator.wikimedia.org/T198183) (owner: 10Sahil505) [15:43:16] (03CR) 10Mforns: [V: 032 C: 032] Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [15:43:23] (03CR) 10jerkins-bot: [V: 04-1] Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [15:45:30] 10Analytics, 10Research: [Open question] Improve bot identification at scale - https://phabricator.wikimedia.org/T138207 (10Nuria) Note to self: considerer referrer as a predictor of whether request is from a bot (no referrer most likely be a direct hit). Completely unrelated but interesting read: https://bit... [15:45:48] (03PS12) 10Mforns: Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [15:46:24] (03CR) 10Mforns: [V: 032 C: 032] Added pagecounts metric [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442257 (https://phabricator.wikimedia.org/T189619) (owner: 10Sahil505) [15:46:39] (03PS5) 10Mforns: Make bar-chart and line-chart resilient to breakdowns with null values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443640 (https://phabricator.wikimedia.org/T198630) [15:46:43] (03CR) 10Mforns: [V: 032 C: 032] Make bar-chart and line-chart resilient to breakdowns with null values [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/443640 (https://phabricator.wikimedia.org/T198630) (owner: 10Mforns) [15:47:38] nuria_, I merged all Wikistat2 changes that can be deployed [15:47:57] mforns: k, will test locally with couple browsers and deploy if proceeds [15:48:34] nuria_, wait, there's another one Sahil pushed but did not add me as a reviewer, looking [15:49:46] addshore: in notebooks? [15:49:55] i can't say i'm an expert either! [15:50:45] nuria_, no that last patch still needs changes, all that can be deployed is merged [15:51:49] ottomata: I'm not even sure where the wmf notebooks stuff is :P [15:51:55] just in a python file right now [15:52:06] currently @ [15:52:08] https://www.irccloud.com/pastebin/kffXaJUH/ [15:52:59] oh, but maybe my issue is not my code, but the fact what im looking for isn't in the data [15:53:08] addshore@stat1005:~$ cat clickstream-enwiki-2018-05.tsv |grep Wikipedia:Bots [15:53:08] addshore@stat1005:~$ [15:53:48] meh [15:57:25] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) I added the IPv6 equivalent of the v4 filter with a default "log+permit" term, so we can see if we missed anything. 3 highlig... [15:58:43] (03CR) 10Nuria: [C: 031] Add MobileWikiAppiOSUserHistory to EL whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/444894 (https://phabricator.wikimedia.org/T195269) (owner: 10Mforns) [15:59:24] addshore: i've actually barely ever used pandas, i just know a lot of analysts like it [15:59:57] addshore: whenver you have time let's discuss https://phabricator.wikimedia.org/T198623#4412383 :) [16:00:01] after my grep I feel like the pandas stuff is working fine, but the clickstream maybe doesn't include what I want :( [16:00:12] (03PS2) 10Mforns: Add MobileWikiAppiOSUserHistory to EL whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/444894 (https://phabricator.wikimedia.org/T195269) [16:00:14] elukey: yup, i just got pinged about it, will look in a sec :) [16:00:15] i also don't know anything about that datase [16:00:17] dataset [16:00:20] maybe joal? [16:03:02] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Partially purge MobileWikiAppiOSUserHistory eventlogging schema - https://phabricator.wikimedia.org/T195269 (10Nuria) >I've seen that since we started this conversation the schema has changed, and a 2-level nested field has been adde... [16:14:29] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10Addshore) The WMDE scripts have requests going to the following places not via the webproxy: - https://noc.wikimedia.org/conf/dblists/... [16:22:46] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) >>! In T198623#4412494, @Addshore wrote: > The WMDE scripts have requests going to the following places not via the webproxy: >... [16:23:24] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10Patch-For-Review: Review analytics-in4/6 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) And more redundant, as query.wikidata.org and wikidata.org are load balanced. [16:27:26] * elukey off!! [16:29:10] (03PS1) 10Addshore: Always request external urls via the webproxy [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/444907 (https://phabricator.wikimedia.org/T198623) [16:30:40] (03CR) 10Addshore: [C: 032] Always request external urls via the webproxy [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/444907 (https://phabricator.wikimedia.org/T198623) (owner: 10Addshore) [16:30:45] (03PS1) 10Addshore: Always request external urls via the webproxy [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/444911 (https://phabricator.wikimedia.org/T198623) [16:30:50] (03CR) 10Addshore: [C: 032] Always request external urls via the webproxy [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/444911 (https://phabricator.wikimedia.org/T198623) (owner: 10Addshore) [16:31:02] (03Merged) 10jenkins-bot: Always request external urls via the webproxy [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/444907 (https://phabricator.wikimedia.org/T198623) (owner: 10Addshore) [16:31:06] (03Merged) 10jenkins-bot: Always request external urls via the webproxy [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/444911 (https://phabricator.wikimedia.org/T198623) (owner: 10Addshore) [16:44:27] * addshore guesses clickstream just includes articles [17:14:15] Hi addshore - Can I try to help? [17:22:37] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10EBernhardson) a:03EBernhardson [17:23:04] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10EBernhardson) [17:25:38] 10Analytics-Tech-community-metrics, 10Code-Health, 10Release-Engineering-Team (Kanban): Develop canonical/single record of origin, machine readable list of all repos deployed to WMF sites. - https://phabricator.wikimedia.org/T190891 (10Jrbranaa) [17:27:18] 10Analytics, 10Discovery-Search (Current work): Use kafka for communication from analytics cluster to elasticsearch - https://phabricator.wikimedia.org/T198490 (10EBernhardson) The new parent task is tangentially related. To cleanly update our puppet we need to switch the TLS certs from the generic puppet cert... [17:48:40] joal: hi hi! I think I ended up finding the problem, the clickstream datasets only have the main namespace :P [18:18:28] 10Analytics, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Alarms on throughput on refined data - https://phabricator.wikimedia.org/T198908 (10Ottomata) [18:19:19] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: EventLogging in Hive data loss due to Camus and Kafka timestamp.type=CreateTime change - https://phabricator.wikimedia.org/T198906 (10Ottomata) I just published an incident report for this: https://wikitech.wikimedia.org/wiki/Incident_documentation/2018070... [18:38:19] https://softwareengineering.stackexchange.com/questions/49550/which-hashing-algorithm-is-best-for-uniqueness-and-speed [18:38:19] wow [18:38:28] collisions actually happen with just 200+K english words [18:38:38] would have guessed they'd be more rare than that [18:42:34] ottomata: thats a cool answer! [18:50:39] ottomata: nice post :) [18:50:57] addshore: Indeed, clickstream dataset only includes namespace 0 [20:25:15] joal, yt? [20:27:51] (03PS1) 10Ottomata: Use ipython for PySpark instead of Toree [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/445002 (https://phabricator.wikimedia.org/T190443) [20:30:01] (03CR) 10Ottomata: [V: 032 C: 032] Use ipython for PySpark instead of Toree [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/445002 (https://phabricator.wikimedia.org/T190443) (owner: 10Ottomata) [20:30:21] (03CR) 10Ottomata: [V: 032 C: 032] Add note about PYSPARK_PYTHON [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/444667 (https://phabricator.wikimedia.org/T190443) (owner: 10Ottomata) [20:45:25] (03PS1) 10Ottomata: Fix path to brunel jar for spark scala jupyter kernels [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/445015 (https://phabricator.wikimedia.org/T190443) [20:45:44] (03CR) 10Ottomata: [V: 032 C: 032] Fix path to brunel jar for spark scala jupyter kernels [analytics/jupyterhub/deploy] - 10https://gerrit.wikimedia.org/r/445015 (https://phabricator.wikimedia.org/T190443) (owner: 10Ottomata) [20:48:21] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Spark Jupyter Notebook integration - https://phabricator.wikimedia.org/T190443 (10Ottomata) @diego, joal and I talked today, and we indeed decided to ditch Toree for PySpark, and just go with the ipython kernel + spark integration. I just installed this on...