Abstract:- Of all the developers delight, none is more attractive than a set of APIs that make developers productive, that are easy to use, and that are intuitive and expressive. Apache Spark offers these APIs across components such as Spark SQL, Streaming, Machine Learning, and Graph Processing to operate on large data sets in languages such as Scala, Java, Python, and R for doing distributed big data processing at scale. In this talk, I will explore the evolution of three sets of APIs - RDDs, DataFrames, and Datasets available in Apache Spark 2.x. In particular, I will emphasize why and when you should use each set as best practices, outline its performance and optimization benefits, and underscore scenarios when to use DataFrames and Datasets instead of RDDs for your big data distributed processing. Through simple notebook demonstrations with API code examples, you'll learn how to process big data using RDDs, DataFrames, and Datasets and interoperate among them.

1. A Tale of Three Apache
Spark APIs: RDDs,
DataFrames & Datasets
Jules S. Damji
August 5, 2017, Big Data Day Los Angeles
@2twitme

2. I have used Apache Spark Before…

3. I know the difference between
DataFrame and RDDs…

4. Spark Community Evangelist &Developer
Advocate @ Databricks
Developer Advocate @ Hortonworks
Software engineering @: Sun Microsystems,
Netscape, @Home, VeriSign, Scalix, Centrify,
LoudCloud/Opsware, ProQuest

5. Agenda…
Why are we here today, what’s the problem?
• Resilient Distributed Datasets (RDDs)
• Structure in Spark
• DataFrames and Datasets
• Demo
• Q & A

6. Not the Tale of Three Kings..

7. Resilient Distributed Dataset
(RDD)

8. What are RDDs?
• … Distributed data abstraction
• … Resilient & Immutable
• … Lazy
• … Compile Type-safe
• … Semi-structuredor unstructured

9. Unstructured Data: Text (logs, tweets, articles, social)

10. Structured Tabular data..

11. Why Use RDDs?
• … Offer Control & flexibility
• ... Low-level API
• ... Type-safe
• ... Encourage how-to

12. Some code to read Wikipedia
val rdd = sc.textFile("/mnt/wikipediapagecounts.gz")
val parsedRDD = rdd.flatMap {
line => line.split("""s+""") match {
case Array(project, page, numRequests, _) => Some((project, page, numRequests))
case _ => None
}
}
// filter only English pages
parsedRDD.filter { case (project, page, numRequests) => project == "en" }.
map { case (_, page, numRequests) => (page, numRequests) }.
reduceByKey(_ + _).
take(100). foreach { case (page, requests) => println(s"$page: $requests") }

13. When to Use RDDs?
• ... Low-level API & control of dataset
• ... Dealing with unstrucrured data (media streams or texts)
• ... Manipulate data with lambda functions than DSL
• ... Don’t care schema or structure of data
• ... Sacrifice optimization, performance & inefficiecies

14. Why When
Use RDDs
• Unstructured Data & No schema
• No code optimization &
performance
• Low-level APIs, not DSL or high-level
• Control and flexibility
• Low-level APIs and Lambda
functions
• Type-safety
• How-do

15. What’s the problem?
• ... Express how-to solution, not what-to
• ... Not optimized by Spark
• ... Slow for non-JVM languages like Python
• ... Inadverdent inefficiecies

16. Inadvertent inefficiencies in RDDs
parsedRDD.filter { case (project, page, numRequests) => project == "en" }.
map { case (_, page, numRequests) => (page, numRequests) }.
reduceByKey(_ + _).
filter { case (page, _) => ! isSpecialPage(page) }.
take(100). foreach { case (project, requests) => println (s"project: $requests") }

17. Structured in Spark
DataFrames & Datasets APIs

18. Background: What is in an RDD?
•Dependencies
• Partitions (with optional localityinfo)
• Compute function: Partition =>Iterator[T]
Opaque Computation
& Opaque Data

19. Structured APIs In Spark
19
SQL DataFrames Datasets
Syntax
Errors
Analysis
Errors
Runtime Compile
Time
Runtime
Compile
Time
Compile
Time
Runtime
Analysis errors are reported before a distributed job starts

20. Unification of APIs in Spark 2.0

21. DataFrame API code.
// convert RDD -> Df with column names
val df = parsedRDD.toDF("project", "page", "numRequests")
//filter, groupBy, sum, and then agg()
df.filter($"project" === "en").
groupBy($"page").
agg(sum($"numRequests").as("count")).
limit(100).
show(100)

22. Take DataFrame à SQL Table à Query
df. createOrReplaceTempView(("edits")
val results = spark.sql("""SELECT page, sum(numRequests)
AS count FROM edits WHERE project = 'en' GROUP BY page
LIMIT 100""")
results.show(100)

23. 23
Using Catalyst in Spark SQL
Unresolved
Logical Plan
Logical Plan
Optimized
Logical Plan
RDDs
Selected
Physical Plan
Analysis
Logical
Optimization
Physical
Planning
CostModel
Physical
Plans
Code
Generation
Catalog
Analysis: analyzinga logicalplan to resolve references
Logical Optimization: logicalplan optimization
Physical Planning: Physical planning
Code Generation:Compileparts of the query to Java bytecode
SQL AST
DataFrame
Datasets

24. PhysicalPlan
with Predicate Pushdown
and Column Pruning
join
optimized
scan
(events)
optimized
scan
(users)
LogicalPlan
filter
join
PhysicalPlan
join
scan
(users)events file userstable
24
scan
(events)
filter
users.join(events, users("id") === events("uid")) .
filter(events("date") > "2015-01-01")
DataFrame Optimization

25. Type-safe:operate
on domain objects
with compiled
lambda functions
8
Dataset API in Spark 2.x
val df = spark.read.j son("people.json")
/ / Convert data to domain obj ects.
case cl ass Person(name: Stri ng, age: I n t )
val ds: Dataset[Person] = df.as[Person]
val = fi l terD S = d s . f i l t e r (p = > p. ag e > 30

26. DataFrames are Faster than RDDs

27. Datasets < Memory RDDs

28. Datasets Faster…

29. Why When
DataFrames & Datasets
• StructuredData schema
• Code optimization & performance
• Space efficiency with Tungsten
• High-level APIs and DSL
• StrongType-safety
• Ease-of-use & Readability
• What-to-do

30. 43
Spark Core (RDD)
Catalyst
DataFrame/DatasetSQL
MLPipelines
Structured
Streaming
{ JSON }
JDBC
andmore…
FoundationalSpark2.x Components
Spark SQL
GraphFrames

31. Source: michaelmalak
Putting all Together: Conclusion

32. Demo

33. https://spark-summit.org/eu-2017/

34. http://dbricks.co/2sK35XT

35. Resources
• Getting Started Guide with Apache Spark on Databricks
• docs.databricks.com
• Spark Programming Guide
• https://databricks.com/blog/2016/01/04/introducing-apache-spark-
datasets.html
• https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-
apis-rdds-dataframes-and-datasets.html
• https://github.com/bmc/rdds-dataframes-datasets-presentation-2016
• Databricks Engineering Blogs

36. Do you have any questions for my preparedanswers?

37. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Q1
Q2
Q3
Q4
Title
Blue
Orange
Green
Use this chart to start

38. Here are some icons to use - scalable
DB Benefits
DB Features
General /Data Science
Icons can be recoloredwithinPowerpoint — see: format picture/ picture color / recolor
Orange, Green, and Black versions (no recolorationnecessary) can be found in go/icons

39. More icons
Industries
Security
Spark Benefits Spark Features

40. Misc
Even more
Misc

41. Slide for Large Question
or Section Headers

42. Thank You
Parting words or contact information go here.