Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Transitioning Compute Models:
Hadoop MapReduce To Spark
Chicago Hadoop Users Group (CHUG)
February 12, 2015
Slim Baltagi
S...
Your Presenter – Slim Baltagi
2
•  Big Data Solutions Architect
living in Chicago.
•  Over 17 years of IT/Business
experie...
Typical Big Data Stack
3
Agenda
1. Evolution
2. Transition
3. Integration
4. Alternatives
5. Complementarity
6. Key Takeaways + Q&A
4
1. Evolution of Compute Models
•  When the Apache Hadoop project started in 2007,
MapReduce v1 was the only choice as a co...
1. Evolution:
•  This is how Hadoop MapReduce is branding itself: “A
YARN-based system for parallel processing of large da...
1. Evolution:
• Tez: Hindi for “speed”
• This is how Apache Tez is branding itself: “The
Apache Tez project is aimed at bu...
1. Evolution:
• ‘Spark’ for lightning fast speed.
• This is how Apache Spark is branding itself:
“Apache Spark™ is a fast ...
1. Evolution: Apache Flink
• Flink: German for “nimble, swift, speedy”
• This is how Apache Flink is branding itself: “Fas...
Hadoop MapReduce vs. Tez vs. Spark
Criteria
License Open Source
Apache 2.0, version
2.x
Open Source,
Apache 2.0,
version 0...
Hadoop MapReduce vs. Tez vs. Spark
Criteria
Installation Bound to Hadoop Bound to Hadoop Isn’t bound to
Hadoop
Ease of Use...
Hadoop MapReduce vs. Tez vs. Spark
Criteria
Deployment YARN YARN [Standalone, YARN*,
SIMR, Mesos]
Performance -  Good perf...
Agenda
1. Evolution
2. Transition
3. Integration
4. Alternatives
5. Complementarity
6. Key Takeaways + Q&A
13
2. Transition
• Existing Hadoop MapReduce projects can
migrate to Spark and leverage Spark Core as
execution engine:
1.  Y...
2. Transition
3. The following tools originally based on Hadoop
MapReduce are being ported to Apache Spark:
•  Pig, Hive, ...
è Pig on Spark (Spork)
•  Run Pig with “–x spark” option for an easy migration
without development effort.
•  Speed up yo...
èHive on Spark (Expected in Hive 1.1.0)
•  New alternative to using MapReduce or Tez:
hive> set hive.execution.engine=spa...
èHive on Spark (Expected in Hive 1.1.0)
• Design
http://blog.cloudera.com/blog/2014/07/apache-hive-on-apache-spark-
motiv...
è Sqoop on Spark
(Expected in Sqoop 2)
•  Sqoop ( a.k.a from SQL to Hadoop) was initially
developed as a tool to transfer...
(Expected in 3.1 release)
•  Cascading http://www.cascading.org is an application
development platform for building data a...
Apache Crunch
• The Apache Crunch Java library provides a
framework for writing, testing, and running
MapReduce pipelines....
(Expec (Expected in Mahout 1.0 )
•  Mahout News: 25 April 2014 - Goodbye MapReduce:
Apache Mahout, the original Machine Le...
Agenda
1. Evolution
2. Transition
3. Integration
4. Alternatives
5. Complementarity
6. Key Takeaways + Q&A
23
3. Integration
Service Open Source Tool
Storage/
Serving Layer
Data Formats
Data
Ingestion
Services
Resource
Management
Se...
3. Integration:
•  Spark was designed to read and write data from and to
HDFS, as well as other storage systems supported ...
3. Integration:
•  Out of the box, Spark can interface with HBase as it has
full support for Hadoop InputFormats via
newAP...
3. Integration:
• Spark Cassandra Connector This library lets
you expose Cassandra tables as Spark RDDs,
write Spark RDDs ...
3. Integration:
• Benchmark of Spark & Cassandra Integration
using different approaches.
http://www.stratio.com/deep-vs-da...
3. Integration:
• MongoDB is not directly served by Spark, although
it can be used from Spark via an official Mongo-
Hadoo...
3. Integration:
• There is also NSMC: Native Spark MongoDB
Connector for reading and writing MongoDB
collections directly ...
3. Integration: YARN
• Integration still improving.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK
%20AN...
3. Integration:
• Spark SQL provides built in support for Hive
tables:
• Import relational data from Hive tables
• Run SQL...
3. Integration:
•  Drill is intended to achieve the sub-second latency
needed for interactive data analysis and exploratio...
3. Integration:
• Apache Kafka is a high throughput distributed
messaging system. http://kafka.apache.org/
• Spark Streami...
3. Integration:
• Apache Flume is a streaming event data
ingestion system that is designed for Big Data
ecosystem. http://...
3. Integration:
• Spark SQL provides built in support for JSON that
is vastly simplifying the end-to-end-experience of
wor...
3. Integration:
•  Apache Parquet is a columnar storage format
available to any project in the Hadoop ecosystem,
regardles...
3. Integration:
•  Spark SQL Avro Library for querying Avro data
with Spark SQL. This library requires Spark 1.2+.
https:/...
3. Integration: Kite SDK
•  The Kite SDK provides high level abstractions to
work with datasets on Hadoop, hiding many of ...
3. Integration:
•  Elasticsearch is a real-time distributed search and analytics
engine. http://www.elasticsearch.org
•  A...
3. Integration:
•  Apache Solr, added a Spark-based indexing tool for
fast and easy indexing, ingestion, and serving
searc...
3. Integration:
• HUE is the open source Apache Hadoop Web UI
that lets users use Hadoop directly from their
browser and b...
Agenda
1. Evolution
2. Transition
3. Integration
4. Alternatives
5. Complementarity
6. Key Takeaways + Q&A
43
4. Alternatives
Hadoop ecosystem Spark Ecosystem
Component
HDFS Tachyon
YARN Mesos
Tools
Pig Spark native API
Hive Spark S...
è
• Tachyon is a memory-centric distributed file
system enabling reliable file sharing at memory-
speed across cluster fr...
è
• Mesos enables fine grained sharing which allows
a Spark job to dynamically take advantage of the
idle resources in th...
YARN vs. Mesos
Criteria
Resource
sharing
Yes Yes
Written in Java C++
Scheduling Memory only CPU and Memory
Running tasks U...
è Spark Native API
• Spark Native API in Scala, Java and Python.
• Interactive shell in Scala and Python.
• Spark support...
è Spark SQL
•  Spark SQL is a new SQL engine designed from ground-
up for Spark
•  Spark SQL provides SQL performance and...
è Spark MLlib
50
è Spark Streaming
51
Storm vs. Spark Streaming
Criteria
Processing Model Record at a time Mini batches
Latency Sub second Few seconds
Fault tol...
è GraphX
53
è Notebook
54
•  Zeppelin http://zeppelin-project.org, is a web-based
notebook that enables interactive data analytics.
H...
Agenda
1. Evolution
2. Transition
3. Integration
4. Alternatives
5. Complementarity
6. Key Takeaways + Q&A
55
5. Complementarity
‘Pillars’ of Hadoop ecosystem and Spark ecosystem can
work together: each for what it is especially goo...
5. Complementarity: + +
•  Tachyon is an memory distributed file system. By storing
the file-system contents in the main m...
5. Complementarity: +
• Mesos and YARN can work together: each for
what it is especially good at, rather than choosing
one...
5. Complementarity: +
References:
•  Apache Mesos vs. Apache Hadoop YARN
https://www.youtube.com/watch?v=YFC4-gtC19E Jim S...
5. Complementarity: +
• Spark on Tez for efficient ETL:
https://github.com/hortonworks/spark-native-yarn
•  Tez could take...
5. Complementarity: +
•  Data >> RAM: Processing huge data volumes,
much bigger than cluster RAM: Tez might be better,
sin...
Agenda
1. Evolution
2. Replacement
3. Integration
4. Alternatives
5. Complementarity
6. Key Takeaways + Q&A
62
6. Key Takeaways + Q&A
1.  Evolution: of compute models is still ongoing. Watch
out Apache Flink project for true low-late...
Upcoming SlideShare
Loading in …5
×

of

YouTube videos are no longer supported on SlideShare

View original on YouTube

Transitioning Compute Models: Hadoop MapReduce to Spark Slide 2 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 3 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 4 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 5 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 6 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 7 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 8 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 9 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 10 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 11 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 12 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 13 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 14 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 15 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 16 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 17 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 18 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 19 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 20 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 21 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 22 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 23 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 24 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 25 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 26 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 27 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 28 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 29 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 30 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 31 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 32 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 33 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 34 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 35 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 36 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 37 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 38 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 39 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 40 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 41 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 42 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 43 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 44 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 45 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 46 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 47 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 48 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 49 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 50 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 51 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 52 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 53 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 54 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 55 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 56 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 57 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 58 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 59 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 60 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 61 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 62 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 63 Transitioning Compute Models: Hadoop MapReduce to Spark Slide 64

72

Share

Download to read offline

Transitioning Compute Models: Hadoop MapReduce to Spark

Download to read offline

This presentation is an analysis of the observed trends in the transition from the Hadoop ecosystem to the Spark ecosystem. The related talk took place at the Chicago Hadoop User Group (CHUG) meetup held on February 12, 2015.

Transitioning Compute Models: Hadoop MapReduce to Spark

  1. 1. Transitioning Compute Models: Hadoop MapReduce To Spark Chicago Hadoop Users Group (CHUG) February 12, 2015 Slim Baltagi SparkBigData.com
  2. 2. Your Presenter – Slim Baltagi 2 •  Big Data Solutions Architect living in Chicago. •  Over 17 years of IT/Business experience. •  Over 4 years of Big Data experience working on over a dozen Hadoop projects. •  Speaker at a few Big Data conferences •  Creator and maintainer of the Apache Spark Knowledge Base: www.SparkBigData.com •  @SlimBaltagi •  sbaltagi@gmail.com Disclaimer: This is a vendor-independent talk that expresses my own opinions and not necessarily those of my current employer: Hortonworks Inc.
  3. 3. Typical Big Data Stack 3
  4. 4. Agenda 1. Evolution 2. Transition 3. Integration 4. Alternatives 5. Complementarity 6. Key Takeaways + Q&A 4
  5. 5. 1. Evolution of Compute Models •  When the Apache Hadoop project started in 2007, MapReduce v1 was the only choice as a compute model (Execution Engine) on Hadoop. Now we have: 5 •  Batch •  Batch •  Interactive •  Batch •  Interactive •  Near-Real time •  Batch •  Interactive •  Real-Time •  Iterative
  6. 6. 1. Evolution: •  This is how Hadoop MapReduce is branding itself: “A YARN-based system for parallel processing of large data sets. http://hadoop.apache.org •  Batch •  Scalability •  User Defined Functions (UDFs) •  Hadoop MapReduce (MR) works pretty well if you can express your problem as a single MR job. In practice, most problems don't fit neatly into a single MR job. •  Need to integrate many disparate tools for advanced Big Data Analytics for Queries, Streaming Analytics, Machine Learning and Graph Analytics. 6
  7. 7. 1. Evolution: • Tez: Hindi for “speed” • This is how Apache Tez is branding itself: “The Apache Tez project is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data. It is currently built atop YARN.” Source: http://tez.apache.org/ • Apache™ Tez is an extensible framework for building high performance batch and interactive data processing applications, coordinated by YARN in Apache Hadoop. 7
  8. 8. 1. Evolution: • ‘Spark’ for lightning fast speed. • This is how Apache Spark is branding itself: “Apache Spark™ is a fast and general engine for large-scale data processing.” https://spark.apache.org • Apache Spark is a general purpose cluster computing framework, its execution model supports wide variety of use cases: batch, interactive, near-real time. •  The rapid in-memory processing of resilient distributed datasets (RDDs) is the “core capability” of Apache Spark. 8
  9. 9. 1. Evolution: Apache Flink • Flink: German for “nimble, swift, speedy” • This is how Apache Flink is branding itself: “Fast and reliable large-scale data processing engine” • Apache Flink http://flink.apache.org/ offers: • Batch and Streaming in the same system • Beyond DAGs (Cyclic operator graphs) • Powerful, expressive APIs • Inside-the-system iterations • Full Hadoop compatibility • Automatic, language independent optimizer 9
  10. 10. Hadoop MapReduce vs. Tez vs. Spark Criteria License Open Source Apache 2.0, version 2.x Open Source, Apache 2.0, version 0.x Open Source, Apache 2.0, version 1.x Processing Model On-Disk (Disk- based parallelization), Batch On-Disk, Batch, Interactive In-Memory, On-Disk, Batch, Interactive, Streaming (Near Real- Time) Language written in Java Java Scala API [Java, Python, Scala], User-Facing Java,[ ISV/ Engine/Tool builder] [Scala, Java, Python], User-Facing Libraries None, separate tools None [Spark Core, Spark Streaming, Spark SQL, MLlib, GraphX] 10
  11. 11. Hadoop MapReduce vs. Tez vs. Spark Criteria Installation Bound to Hadoop Bound to Hadoop Isn’t bound to Hadoop Ease of Use Difficult to program, needs abstractions No Interactive mode except Hive Difficult to program No Interactive mode except Hive Easy to program, no need of abstractions Interactive mode Compatibility to data types and data sources is same to data types and data sources is same to data types and data sources is same YARN integration YARN application Ground up YARN application Spark is moving towards YARN 11
  12. 12. Hadoop MapReduce vs. Tez vs. Spark Criteria Deployment YARN YARN [Standalone, YARN*, SIMR, Mesos] Performance -  Good performance when data fits into memory -  performance degradation otherwise Security More features and projects More features and projects Still in its infancy 12 * Partial support
  13. 13. Agenda 1. Evolution 2. Transition 3. Integration 4. Alternatives 5. Complementarity 6. Key Takeaways + Q&A 13
  14. 14. 2. Transition • Existing Hadoop MapReduce projects can migrate to Spark and leverage Spark Core as execution engine: 1.  You can often reuse your mapper and reducer functions and just call them in Spark, from Java or Scala. 2.  You can translate your code from MapReduce to Apache Spark. How-to: Translate from MapReduce to Apache Spark http://blog.cloudera.com/blog/2014/09/how-to-translate-from- mapreduce-to-apache-spark/ 14
  15. 15. 2. Transition 3. The following tools originally based on Hadoop MapReduce are being ported to Apache Spark: •  Pig, Hive, Sqoop, Cascading, Crunch, Mahout, … 15
  16. 16. è Pig on Spark (Spork) •  Run Pig with “–x spark” option for an easy migration without development effort. •  Speed up your existing pig scripts on Spark ( Query, Logical Plan, Physical Pan) •  Leverage new Spark specific operators in Pig such as Cache •  Still leverage many existing Pig UDF libraries •  Pig on Spark Umbrella Jira (Status: Passed end-to-end test cases on Pig, still Open) https://issues.apache.org/jira/browse/PIG-4059 •  Fix outstanding issues and address additional Spark functionality through the community 16
  17. 17. èHive on Spark (Expected in Hive 1.1.0) •  New alternative to using MapReduce or Tez: hive> set hive.execution.engine=spark; •  Help existing Hive applications running on MapReduce or Tez easily migrate to Spark without development effort. •  Exposes Spark users to a viable, feature-rich de facto standard SQL tool on Hadoop. •  Performance benefits especially for Hive queries, involving multiple reducer stages •  Hive on Spark Umbrella Jira (Status: Open).Q1 2015 https://issues.apache.org/jira/browse/HIVE-7292 17
  18. 18. èHive on Spark (Expected in Hive 1.1.0) • Design http://blog.cloudera.com/blog/2014/07/apache-hive-on-apache-spark- motivations-and-design-principles/ • Demo http://blog.cloudera.com/blog/2014/11/apache-hive-on-apache-spark-the-first- demo/ • Hands-on sandbox http://blog.cloudera.com/blog/2014/12/hands-on-hive-on-spark-in-the-aws- cloud/ • Getting Started https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark:+Getting +Started 18
  19. 19. è Sqoop on Spark (Expected in Sqoop 2) •  Sqoop ( a.k.a from SQL to Hadoop) was initially developed as a tool to transfer data from RDBMS to Hadoop. •  The next version of Sqoop, referred to as Sqoop2 supports data transfer across any two data sources. •  Sqoop 2 Proposal is still under discussion. https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Proposal •  Sqoop2: Support Sqoop on Spark Execution Engine (Jira Status: Work In Progress). The goal of this ticket is to support a pluggable way to select the execution engine on which we can run the Sqoop jobs. https://issues.apache.org/jira/browse/SQOOP-1532 19
  20. 20. (Expected in 3.1 release) •  Cascading http://www.cascading.org is an application development platform for building data applications on Hadoop. •  Support for Apache Spark is on the roadmap and will be available in Cascading 3.1 release. Reference : http://www.cascading.org/new-fabric-support/ •  Spark-scalding is a library that aims to make the transition from Cascading/Scalding to Spark a little easier by adding support for Cascading Taps, Scalding Sources and the Scalding Fields API in Spark. Reference : http://scalding.io/2014/10/running-scalding-on-apache-spark/ 20
  21. 21. Apache Crunch • The Apache Crunch Java library provides a framework for writing, testing, and running MapReduce pipelines. https://crunch.apache.org • Apache Crunch 0.11 releases with a SparkPipeline class, making it easy to migrate data processing applications from MapReduce to Spark. https://crunch.apache.org/apidocs/0.11.0/org/apache/crunch/impl/ spark/SparkPipeline.html • Running Crunch with Spark http://www.cloudera.com/content/cloudera/en/documentation/core/ v5-2-x/topics/cdh_ig_running_crunch_with_spark.html 21
  22. 22. (Expec (Expected in Mahout 1.0 ) •  Mahout News: 25 April 2014 - Goodbye MapReduce: Apache Mahout, the original Machine Learning (ML) library for Hadoop since 2009, is rejecting new MapReduce algorithm implementations. http://mahout.apache.org •  Integration of Mahout and Spark: •  Reboot with new Mahout Scala DSL for Distributed Machine Learning on Spark: Programs written in this DSL are automatically optimized and executed in parallel on Apache Spark. •  Mahout Interactive Shell: Interactive REPL shell for Spark optimized Mahout DSL. •  Example: http://mahout.apache.org/users/recommender/intro-cooccurrence-spark.html 22
  23. 23. Agenda 1. Evolution 2. Transition 3. Integration 4. Alternatives 5. Complementarity 6. Key Takeaways + Q&A 23
  24. 24. 3. Integration Service Open Source Tool Storage/ Serving Layer Data Formats Data Ingestion Services Resource Management Search SQL 24
  25. 25. 3. Integration: •  Spark was designed to read and write data from and to HDFS, as well as other storage systems supported by Hadoop API, such as your local file system, Hive, HBase, Cassandra and Amazon’s S3. •  Stronger integration between Spark and HDFS caching (SPARK-1767) to allow multiple tenants and processing frameworks to share the same in-memory https://issues.apache.org/jira/browse/SPARK-1767 •  Use DDM: Discardable Distributed Memory http://hortonworks.com/blog/ddm/ to store RDDs in memory.This allows many Spark applications to share RDDs since they are now resident outside the address space of the application. Related HDFS-5851 is planned for Hadoop 3.0 https://issues.apache.org/jira/browse/HDFS-5851 25
  26. 26. 3. Integration: •  Out of the box, Spark can interface with HBase as it has full support for Hadoop InputFormats via newAPIHadoopRDD. Example: HBaseTest.scala from Spark code. https://github.com/apache/spark/blob/master/examples/src/ main/scala/org/apache/spark/examples/HBaseTest.scala •  There are also Spark RDD implementations available for reading from and writing to HBase without the need of using Hadoop API anymore: Spark-HBase Connector https://github.com/nerdammer/spark-hbase-connector •  SparkOnHBase is a project for HBase integration with Spark. Status: Still in experimentation and no timetable for possible support. http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs- sparkonhbase/ 26
  27. 27. 3. Integration: • Spark Cassandra Connector This library lets you expose Cassandra tables as Spark RDDs, write Spark RDDs to Cassandra tables, and execute arbitrary CQL queries in your Spark applications. Supports also integration of Spark Streaming with Cassandra https://github.com/datastax/spark-cassandra-connector • Spark + Cassandra using Deep: The integration is not based on the Cassandra's Hadoop interface. http://stratio.github.io/deep-spark/ 27
  28. 28. 3. Integration: • Benchmark of Spark & Cassandra Integration using different approaches. http://www.stratio.com/deep-vs-datastax/ •  Calliope is a library providing an interface to consume data from Cassandra to spark and store Resilient Distributed Datasets (RDD) from Spark to Cassandra. http://tuplejump.github.io/calliope/ •  Cassandra storage backend with Spark is opening many new avenues. •  Kindling: An Introduction to Spark with Cassandra (Part 1) http://planetcassandra.org/blog/kindling-an-introduction-to-spark-with- cassandra/ 28
  29. 29. 3. Integration: • MongoDB is not directly served by Spark, although it can be used from Spark via an official Mongo- Hadoop connector. •  MongoDB-Spark Demo https://github.com/crcsmnky/mongodb-spark-demo •  MongoDB and Hadoop: Driving Business Insights •  http://www.slideshare.net/mongodb/mongodb-and-hadoop-driving- business-insights • Spark SQL also provides indirect support via its support for reading and writing JSON text files. https://github.com/mongodb/mongo-hadoop 29
  30. 30. 3. Integration: • There is also NSMC: Native Spark MongoDB Connector for reading and writing MongoDB collections directly from Apache Spark (still experimental) •  GitHub https://github.com/spirom/spark-mongodb-connector •  Examples https://github.com/spirom/spark-mongodb-examples/tree/ depends-v0.3.0 •  Blog http://www.river-of-bytes.com/2015/01/nsmc-native-mongodb-connector- for.html 30
  31. 31. 3. Integration: YARN • Integration still improving. https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK %20AND%20summary%20~%20yarn%20AND%20status%20%3D %20OPEN%20ORDER%20BY%20priority%20DESC%0A • Some issues are critical ones. http://spark.apache.org/docs/latest/running-on-yarn.html Running Spark on YARN http://spark.apache.org/docs/latest/running-on-yarn.html • Get the most out of Spark on YARN https://www.youtube.com/watch?v=Vkx-TiQ_KDU 31
  32. 32. 3. Integration: • Spark SQL provides built in support for Hive tables: • Import relational data from Hive tables • Run SQL queries over imported data • Easily write RDDs out to Hive tables • Hive 0.13 is supported in Spark 1.2.0. • Support of ORCFile (Optimized Row Columnar file) format is targeted in Spark 1.3.0 Spark-2883 https://issues.apache.org/jira/browse/SPARK-2883 • Hive can be used both for analytical queries and for fetching dataset machine learning algorithms in MLlib. 32
  33. 33. 3. Integration: •  Drill is intended to achieve the sub-second latency needed for interactive data analysis and exploration. http://drill.apache.org •  Drill and Spark Integration is work in progress in 2015 to address new use cases: •  Use a Drill query (or view) as the input to Spark. Drill extracts and pre-processes data from various data sources and turns it into input to Spark. •  Use Drill to query Spark RDDs. Use BI tools to query in-memory data in Spark. Embed Drill execution in a Spark data pipeline. •  Reference: What's Coming in 2015 for Drill? http://drill.apache.org/blog/2014/12/16/whats-coming-in-2015/ 33
  34. 34. 3. Integration: • Apache Kafka is a high throughput distributed messaging system. http://kafka.apache.org/ • Spark Streaming integrates natively with Kafka: Spark Streaming + Kafka Integration Guide http://spark.apache.org/docs/latest/streaming-kafka-integration.html • Tutorial: Integrating Kafka and Spark Streaming: Code Examples and State of the Game http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming- integration-example-tutorial/ 34
  35. 35. 3. Integration: • Apache Flume is a streaming event data ingestion system that is designed for Big Data ecosystem. http://flume.apache.org/ • Spark Streaming integrates natively with Flume. There are two approaches to this: • Approach 1: Flume-style Push-based Approach • Approach 2 (Experimental): Pull-based Approach using a Custom Sink. • Spark Streaming + Flume Integration Guide https://spark.apache.org/docs/latest/streaming-flume-integration.html 35
  36. 36. 3. Integration: • Spark SQL provides built in support for JSON that is vastly simplifying the end-to-end-experience of working with JSON data. • Spark SQL can automatically infer the schema of a JSON dataset and load it as a SchemaRDD. No more DDL. Just point Spark SQL to JSON files and query. Starting Spark 1.3, SchemaRDD will be renamed to DataFrame. • An introduction to JSON support in Spark SQL http://databricks.com/blog/2015/02/02/an-introduction-to-json-support-in- spark-sql.html 36
  37. 37. 3. Integration: •  Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language. http://parquet.incubator.apache.org/ •  Built in support in Spark SQL allows to: •  Import relational data from Parquet files •  Run SQL queries over imported data •  Easily write RDDs out to Parquet files http://spark.apache.org/docs/latest/sql-programming- guide.html#parquet-files •  This is an illustrating example of integration of Parquet and Spark SQL http://www.infoobjects.com/spark-sql-parquet/ 37
  38. 38. 3. Integration: •  Spark SQL Avro Library for querying Avro data with Spark SQL. This library requires Spark 1.2+. https://github.com/databricks/spark-avro •  This is an example of using Avro and Parquet in Spark SQL. http://www.infoobjects.com/spark-with-avro/ •  Avro/Spark Use case: http://www.slideshare.net/DavidSmelker/bdbdug-data-types-jan-2015 •  Problem •  Various inbound data sets •  Data Layout can change without notice •  New data sets can be added without notice Result •  Leverage Spark to dynamically split the data •  Leverage Avro to store the data in a compact binary format 38
  39. 39. 3. Integration: Kite SDK •  The Kite SDK provides high level abstractions to work with datasets on Hadoop, hiding many of the details of compression codecs, file formats, partitioning strategies, etc. http://kitesdk.org/docs/current/ • Spark support has been added to Kite 0.16 release, so Spark jobs can read and write to Kite datasets. • Kite Java Spark Demo https://github.com/kite-sdk/kite-examples/tree/master/spark 39
  40. 40. 3. Integration: •  Elasticsearch is a real-time distributed search and analytics engine. http://www.elasticsearch.org •  Apache Spark Support in Elasticsearch added in 2.1 http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/ spark.html •  Deep-Spark provides also an integration with Spark. https://github.com/Stratio/deep-spark •  elasticsearch-hadoop provides native integration between Elasticsearch and Apache Spark, in the form of RDD that can read data from Elasticsearch. Also, any RDD can be saved to Elasticsearch as long as its content can be translated into documents. •  Great use case by NTT Data integrating Apache Spark Streaming and Elasticsearch. http://www.intellilink.co.jp/article/column/bigdata-kk02.html 40
  41. 41. 3. Integration: •  Apache Solr, added a Spark-based indexing tool for fast and easy indexing, ingestion, and serving searchable complex data. “CrunchIndexerTool on Spark” •  Solr-on-Spark solution using Apache Solr, Spark, Crunch, and Morphlines: •  Migrate ingestion of HDFS data into Solr from MapReduce to Spark •  Update and delete existing documents in Solr at scale •  Ingesting HDFS data into Solr using Spark http://www.slideshare.net/whoschek/ingesting-hdfs- intosolrusingsparktrimmed 41
  42. 42. 3. Integration: • HUE is the open source Apache Hadoop Web UI that lets users use Hadoop directly from their browser and be productive. http://www.gethue.com • A Hue application for Apache Spark called Spark Igniter lets users execute and monitor Spark jobs directly from their browser and be more productive. • Demo of Spark Igniter http://vimeo.com/83192197 • Big Data Web applications for Interactive Hadoop https://speakerdeck.com/bigdataspain/big-data-web-applications-for- interactive-hadoop-by-enrico-berti-at-big-data-spain-2014 42
  43. 43. Agenda 1. Evolution 2. Transition 3. Integration 4. Alternatives 5. Complementarity 6. Key Takeaways + Q&A 43
  44. 44. 4. Alternatives Hadoop ecosystem Spark Ecosystem Component HDFS Tachyon YARN Mesos Tools Pig Spark native API Hive Spark SQL Mahout MLlib Storm Spark Streaming Giraph GraphX HUE Spark Notebook/ISpark 44
  45. 45. è • Tachyon is a memory-centric distributed file system enabling reliable file sharing at memory- speed across cluster frameworks, such as Spark and MapReduce. https://http://tachyon-project.org • Tachyon is Hadoop compatible. Existing Spark and MapReduce programs can run on top of it without any code change. • Tachyon is the storage layer of the Berkeley Data Analytics Stack (BDAS) https://amplab.cs.berkeley.edu/software/ 45
  46. 46. è • Mesos enables fine grained sharing which allows a Spark job to dynamically take advantage of the idle resources in the cluster during its execution. •  This leads to considerable performance improvements, especially for long running Spark jobs. •  Mesos as Datacenter “OS”: •  Share datacenter between multiple cluster computing apps. •  Provide new abstractions and services •  Mesosphere DCOS: Datacenter services, including Apache Spark, Apache Cassandra, Apache YARN, Apache HDFS… 46
  47. 47. YARN vs. Mesos Criteria Resource sharing Yes Yes Written in Java C++ Scheduling Memory only CPU and Memory Running tasks Unix processes Linux Container groups Requests Specific requests and locality preference More generic but more coding for writing frameworks Maturity Less mature Relatively more mature 47
  48. 48. è Spark Native API • Spark Native API in Scala, Java and Python. • Interactive shell in Scala and Python. • Spark supports Java 8 for a much more concise Lambda expressions to get code nearly as simple as the Scala API. 48
  49. 49. è Spark SQL •  Spark SQL is a new SQL engine designed from ground- up for Spark •  Spark SQL provides SQL performance and maintains compatibility with Hive. It supports all existing Hive data formats, user-defined functions (UDF), and the Hive metastore. •  Spark SQL also allows manipulating (semi-) structured data as well as ingesting data from sources that provide schema, such as JSON, Parquet, Hive, or EDWs. It unifies SQL and sophisticated analysis, allowing users to mix and match SQL and more imperative programming APIs for advanced analytics. 49
  50. 50. è Spark MLlib 50
  51. 51. è Spark Streaming 51
  52. 52. Storm vs. Spark Streaming Criteria Processing Model Record at a time Mini batches Latency Sub second Few seconds Fault tolerance– every record processed At least one ( may be duplicates) Exactly one Batch Framework integration Not available Core Spark API Supported languages Any programming language Scala, Java, Python 52
  53. 53. è GraphX 53
  54. 54. è Notebook 54 •  Zeppelin http://zeppelin-project.org, is a web-based notebook that enables interactive data analytics. Has built-in Apache Spark support. • Spark Notebook is an interactive web-based editor that can combine Scala code, SQL queries, Markup or even JavaScript in a collaborative manner. https://github.com/andypetrella/spark-notebook • ISpark is an Apache Spark-shell backend for IPython https://github.com/tribbloid/ISpark
  55. 55. Agenda 1. Evolution 2. Transition 3. Integration 4. Alternatives 5. Complementarity 6. Key Takeaways + Q&A 55
  56. 56. 5. Complementarity ‘Pillars’ of Hadoop ecosystem and Spark ecosystem can work together: each for what it is especially good at, rather than choosing one of them. 56 Hadoop ecosystem Spark ecosystem
  57. 57. 5. Complementarity: + + •  Tachyon is an memory distributed file system. By storing the file-system contents in the main memory of all cluster nodes, the system achieves higher throughput than traditional disk-based storage systems like HDFS. •  The Future Architecture of a Data Lake: In-memory Data Exchange Platform Using Tachyon and Apache Spark http://blog.pivotal.io/big-data-pivotal/news-2/the-future-architecture-of-a- data-lake-in-memory-data-exchange-platform-using-tachyon-and-apache- spark •  Spark and in-memory databases:Tachyon leading the pack http://dynresmanagement.com/1/post/2015/01/spark-and-in-memory- databases-tachyon-leading-the-pack.html 57
  58. 58. 5. Complementarity: + • Mesos and YARN can work together: each for what it is especially good at, rather than choosing one of the two for Hadoop deployment. • Big data developers get the best of YARN’s power for Hadoop-driven workloads, and Mesos’ ability to run any other kind of workload, including non-Hadoop applications like Web applications and other long-running services.” 58
  59. 59. 5. Complementarity: + References: •  Apache Mesos vs. Apache Hadoop YARN https://www.youtube.com/watch?v=YFC4-gtC19E Jim Scott, MapR •  Myriad: A Mesos framework for scaling a YARN cluster https://github.com/mesos/myriad •  Myriad Project Marries YARN and Apache Mesos Resource Management http://ostatic.com/blog/myriad-project-marries-yarn-and- apache-mesos-resource-management •  YARN vs. MESOS: Can’t We All Just Get Along? http://strataconf.com/big-data-conference-ca-2015/public/schedule/ detail/40620 59
  60. 60. 5. Complementarity: + • Spark on Tez for efficient ETL: https://github.com/hortonworks/spark-native-yarn •  Tez could takes care of the pure Hadoop optimization strategies (building the DAG with knowledge of data distribution, statistics or… HDFS caching). •  Spark execution layer could be leveraged without the need of a nasty Spark/Hadoop coupling. •  Tez is good on fine-grained resource isolation with YARN (resource chaining in clusters) •  Tez supports enterprise security 60
  61. 61. 5. Complementarity: + •  Data >> RAM: Processing huge data volumes, much bigger than cluster RAM: Tez might be better, since it is more “stream oriented” , has more mature shuffling implementation, closer YARN integration. •  Data << RAM: Since Spark can cache in memory parsed data, it can be much better when we process data smaller than cluster’s memory. •  Improving Spark for Data Pipelines with Native YARN Integration http://hortonworks.com/blog/improving-spark-data-pipelines-native- yarn-integration/ • Get the most out of Spark on YARN https://www.youtube.com/watch?v=Vkx-TiQ_KDU 61
  62. 62. Agenda 1. Evolution 2. Replacement 3. Integration 4. Alternatives 5. Complementarity 6. Key Takeaways + Q&A 62
  63. 63. 6. Key Takeaways + Q&A 1.  Evolution: of compute models is still ongoing. Watch out Apache Flink project for true low-latency and iterative use cases and better performance! 2.  Transition: Tools from the Hadoop ecosystem are still being ported to Spark. Keep watching general availability and balance risk and opportunity. 3.  Integration: Healthy dose of Hadoop ecosystem integration with Spark. More integration is on the way. 4.  Alternatives: Do your due diligence based on your own use case and research pros and cons before picking a specific tool or switching from one tool to another. 5.  Complementarity: Components and tools from Hadoop ecosystem and Spark ecosystem can work together: each for what it is especially good at. One size doesn’t fit all! 63
  • thaweesakb

    Aug. 14, 2017
  • flei_98

    Jan. 11, 2017
  • ohsgx

    Nov. 23, 2016
  • vaquarkhan1

    Oct. 3, 2016
  • SanjaykumarChavan

    Aug. 16, 2016
  • PetrosAnagnostopoulo

    Jul. 26, 2016
  • kalo.liang

    Apr. 26, 2016
  • Bii03

    Mar. 2, 2016
  • mring33

    Jan. 13, 2016
  • jxauwxj

    Jan. 6, 2016
  • otletian

    Dec. 29, 2015
  • kelsonzhaoss

    Nov. 24, 2015
  • romanchukh

    Nov. 1, 2015
  • Kynnar

    Sep. 22, 2015
  • davidginzburg

    Sep. 12, 2015
  • guard0g

    Aug. 29, 2015
  • joobn

    Aug. 18, 2015
  • SarangAnajwala

    Aug. 14, 2015
  • giabar8

    May. 29, 2015
  • SharanTheja

    Apr. 22, 2015

This presentation is an analysis of the observed trends in the transition from the Hadoop ecosystem to the Spark ecosystem. The related talk took place at the Chicago Hadoop User Group (CHUG) meetup held on February 12, 2015.

Views

Total views

12,388

On Slideshare

0

From embeds

0

Number of embeds

68

Actions

Downloads

648

Shares

0

Comments

0

Likes

72

×