SlideShare a Scribd company logo

Hadoop World Spark Meetup: Interactive Spark in your Browser

gethue
gethue

Hue now offers a Notebook application for interactively processing, visualizing and sharing data. Through a new Spark REST Job Server, Spark Python and Scala shells are available as well as Streaming. Those are ideal for doing some quick big data crunching from anywhere or a Web browser! This talk details the architecture of the REST API and Notebook UI as well as their integration with the Hadoop ecosystem. It also describes the alternatives we tried and the challenges that were faced. The capabilities will then be lived demo in Hue’s Notebook Application through a real life scenario combining Spark and Hadoop. Attendees will learn how to ramp-up on Spark and see how they could open it up to all their users and analyse even more data.

1 of 22
Download to read offline
INTERACTIVE
SPARK IN YOUR
BROWSER
Spark	
  Lightning	
  Night	
  -­‐	
  Hadoop	
  World
September	
  2015
Hadoop World Spark Meetup: Interactive Spark in your Browser
Hadoop World Spark Meetup: Interactive Spark in your Browser
GOAL

OF HUE
WEB INTERFACE FOR ANALYZING DATA
WITH APACHE HADOOP	
  
SIMPLIFY AND INTEGRATE



FREE AND OPEN SOURCE
—> “BIG NOTEBOOK” FOR HADOOP
SPARK

ARCHITECTURE
Livy
Spark
Spark
Spark
YARN
Backend	
  partWeb	
  part
SPARK

ARCHITECTURE
Livy
Spark
Spark
Spark
YARN
Spark	
  ServerUI

Recommended

20150627 bigdatala
20150627 bigdatala20150627 bigdatala
20150627 bigdatalagethue
 
Apache Tez – Present and Future
Apache Tez – Present and FutureApache Tez – Present and Future
Apache Tez – Present and FutureDataWorks Summit
 
HBase + Hue - LA HBase User Group
HBase + Hue - LA HBase User GroupHBase + Hue - LA HBase User Group
HBase + Hue - LA HBase User Groupgethue
 
Big Data Scala by the Bay: Interactive Spark in your Browser
Big Data Scala by the Bay: Interactive Spark in your BrowserBig Data Scala by the Bay: Interactive Spark in your Browser
Big Data Scala by the Bay: Interactive Spark in your Browsergethue
 
Integrate Hue with your Hadoop cluster - Yahoo! Hadoop Meetup
Integrate Hue with your Hadoop cluster - Yahoo! Hadoop MeetupIntegrate Hue with your Hadoop cluster - Yahoo! Hadoop Meetup
Integrate Hue with your Hadoop cluster - Yahoo! Hadoop Meetupgethue
 
Hue: Big Data Web applications for Interactive Hadoop at Big Data Spain 2014
Hue: Big Data Web applications for Interactive Hadoop at Big Data Spain 2014Hue: Big Data Web applications for Interactive Hadoop at Big Data Spain 2014
Hue: Big Data Web applications for Interactive Hadoop at Big Data Spain 2014gethue
 
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with SparkSparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss
 

More Related Content

More from gethue

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...
Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...
Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...gethue
 
SF Solr Meetup - Interactively Search and Visualize Your Big Data
SF Solr Meetup - Interactively Search and Visualize Your Big DataSF Solr Meetup - Interactively Search and Visualize Your Big Data
SF Solr Meetup - Interactively Search and Visualize Your Big Datagethue
 
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and HueHadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Huegethue
 
Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0
Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0
Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0gethue
 
Interactively Search and Visualize Your Big Data
Interactively Search and Visualize Your Big DataInteractively Search and Visualize Your Big Data
Interactively Search and Visualize Your Big Datagethue
 
Sqoop2 refactoring for generic data transfer - NYC Sqoop Meetup
Sqoop2 refactoring for generic data transfer - NYC Sqoop MeetupSqoop2 refactoring for generic data transfer - NYC Sqoop Meetup
Sqoop2 refactoring for generic data transfer - NYC Sqoop Meetupgethue
 
LDAP, SAML and Hue
LDAP, SAML and HueLDAP, SAML and Hue
LDAP, SAML and Huegethue
 
Hadoop Israel - HBase Browser in Hue
Hadoop Israel - HBase Browser in HueHadoop Israel - HBase Browser in Hue
Hadoop Israel - HBase Browser in Huegethue
 
Hue: The Hadoop UI - Hadoop Singapore
Hue: The Hadoop UI - Hadoop SingaporeHue: The Hadoop UI - Hadoop Singapore
Hue: The Hadoop UI - Hadoop Singaporegethue
 
SF Dev Meetup - Hue SDK
SF Dev Meetup - Hue SDKSF Dev Meetup - Hue SDK
SF Dev Meetup - Hue SDKgethue
 
Hue: The Hadoop UI - Where we stand, Hue Meetup SF
Hue: The Hadoop UI - Where we stand, Hue Meetup SF Hue: The Hadoop UI - Where we stand, Hue Meetup SF
Hue: The Hadoop UI - Where we stand, Hue Meetup SF gethue
 
Hue: The Hadoop UI - HUG France
Hue: The Hadoop UI - HUG FranceHue: The Hadoop UI - HUG France
Hue: The Hadoop UI - HUG Francegethue
 
Hue: The Hadoop UI - Stockholm HUG
Hue: The Hadoop UI - Stockholm HUGHue: The Hadoop UI - Stockholm HUG
Hue: The Hadoop UI - Stockholm HUGgethue
 

More from gethue (13)

Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...
Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...
Spark Summit Europe: Building a REST Job Server for interactive Spark as a se...
 
SF Solr Meetup - Interactively Search and Visualize Your Big Data
SF Solr Meetup - Interactively Search and Visualize Your Big DataSF Solr Meetup - Interactively Search and Visualize Your Big Data
SF Solr Meetup - Interactively Search and Visualize Your Big Data
 
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and HueHadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
Hadoop Summit - Interactive Big Data Analysis with Solr, Spark and Hue
 
Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0
Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0
Harness the power of Spark and Solr in Hue: Big Data Amsterdam v.2.0
 
Interactively Search and Visualize Your Big Data
Interactively Search and Visualize Your Big DataInteractively Search and Visualize Your Big Data
Interactively Search and Visualize Your Big Data
 
Sqoop2 refactoring for generic data transfer - NYC Sqoop Meetup
Sqoop2 refactoring for generic data transfer - NYC Sqoop MeetupSqoop2 refactoring for generic data transfer - NYC Sqoop Meetup
Sqoop2 refactoring for generic data transfer - NYC Sqoop Meetup
 
LDAP, SAML and Hue
LDAP, SAML and HueLDAP, SAML and Hue
LDAP, SAML and Hue
 
Hadoop Israel - HBase Browser in Hue
Hadoop Israel - HBase Browser in HueHadoop Israel - HBase Browser in Hue
Hadoop Israel - HBase Browser in Hue
 
Hue: The Hadoop UI - Hadoop Singapore
Hue: The Hadoop UI - Hadoop SingaporeHue: The Hadoop UI - Hadoop Singapore
Hue: The Hadoop UI - Hadoop Singapore
 
SF Dev Meetup - Hue SDK
SF Dev Meetup - Hue SDKSF Dev Meetup - Hue SDK
SF Dev Meetup - Hue SDK
 
Hue: The Hadoop UI - Where we stand, Hue Meetup SF
Hue: The Hadoop UI - Where we stand, Hue Meetup SF Hue: The Hadoop UI - Where we stand, Hue Meetup SF
Hue: The Hadoop UI - Where we stand, Hue Meetup SF
 
Hue: The Hadoop UI - HUG France
Hue: The Hadoop UI - HUG FranceHue: The Hadoop UI - HUG France
Hue: The Hadoop UI - HUG France
 
Hue: The Hadoop UI - Stockholm HUG
Hue: The Hadoop UI - Stockholm HUGHue: The Hadoop UI - Stockholm HUG
Hue: The Hadoop UI - Stockholm HUG
 

Recently uploaded

Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdfAlexia Trejo
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache sparkVarun Garg
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyVarun Garg
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............mahetamanav24
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfSalamaAdel
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelTope Osanyintuyi
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby
 
Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023maker Money
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...ThinkInnovation
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxyosra Saidani
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEarley Information Science
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...DrSumathyV
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdfruwanp2000
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFProject Cubicle
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughsNikolas Markou
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Samuel Chukwuma
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxSugumarVenkai
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Sarma Pydipally
 

Recently uploaded (18)

Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdf
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache spark
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarify
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdf
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft Excel
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docx
 
Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptx
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5
 

Hadoop World Spark Meetup: Interactive Spark in your Browser

  • 1. INTERACTIVE SPARK IN YOUR BROWSER Spark  Lightning  Night  -­‐  Hadoop  World September  2015
  • 4. GOAL
 OF HUE WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP   SIMPLIFY AND INTEGRATE
 
 FREE AND OPEN SOURCE —> “BIG NOTEBOOK” FOR HADOOP
  • 8. • Married  with  full  ecosystem   • File,  Job  browsers   • Create  table  wizards   • Any  language  (Hive,  Spark...)   • Graphing   • Export/Import/Sharing   • MulT  users   • ImpersonaTon   HADOOP WITH SPARK
 NOTEBOOK
  • 10. • REST  Web  server  in  Scala   • InteracTve  Spark  Sessions  and  Batch  Jobs   • Type  IntrospecTon  for  VisualizaTon   • Running  sessions  in  YARN    local   • Backends:  Scala,  Python,  R   • Open  Source:  h[ps://github.com/cloudera/hue/ tree/master/apps/spark/java   • Play  with  Curl  h[p://gethue.com/how-­‐to-­‐use-­‐the-­‐ LIVY
 SPARK SERVER
  • 11. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker Livy  Server Scalatra Session  Manager Session
  • 12. LIVY WEB SERVER
 ARCHITECTURE Livy  Server YARN   Master Scalatra Spark   Client Session  Manager Session YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1
  • 13. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 Livy  Server Scalatra Session  Manager Session
  • 14. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 Livy  Server Scalatra Session  Manager Session
  • 15. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 4 Livy  Server Scalatra Session  Manager Session
  • 16. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 4 5 Livy  Server Scalatra Session  Manager Session
  • 17. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 4 5 6 Livy  Server Scalatra Session  Manager Session
  • 18. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 7 2 3 4 5 6 Livy  Server Scalatra Session  Manager Session
  • 19. SESSION CREATION
 AND EXECUTION %  curl  -­‐XPOST  localhost:8998/sessions        -­‐d  '{"kind":  "spark"}'   {      "id":  0,      "kind":  "spark",      "log":  [...],      "state":  "idle"   }   %  curl  -­‐XPOST  localhost:8998/sessions/0/statements  -­‐d  '{"code":  " {      "id":  0,      "output":  {          "data":  {  "text/plain":  "res0:  Int  =  2"  },          "execution_count":  0,          "status":  "ok"      },      "state":  "available"   }  
  • 21. • Stability  and  Scaling   • Security   • iPython/Jupyter  interpreter   and  file  format   COMING SOON