Scala and Spark: Coevolving Ecosystems for Big Data

872 views

Published on

Presented at Datapalloza Seattle, Feb 10-11, 2016

Published in: Data & Analytics

Scala and Spark: Coevolving Ecosystems for Big Data

  1. 1. Scala and Spark: Coevolving Ecosystems for Big Data John Nestor 47 Degrees Datapalooza Seattle February 10-11, 2016 www.47deg.com 147deg.com
  2. 2. 47deg.com Outline • Scala • Spark • Scala Impact on Spark • Spark Impact on Scala • Summary • Questions 2
  3. 3. Scala 3
  4. 4. 47deg.com Scala History • Scala created by Martin Odersky, EFPL Switzerland • wrote javac, the compiler for Java • designed Java generics • 2004 Scala announced • 2006 Scala 2.0 • 2011 Typesafe started, Scala 2.9 • 2012 Scala 2.10 • 2014 Scala 2.11 • 2015 Scala 2.12-RC1 • 2016 Scala now 12 years old and quite mature 4
  5. 5. 47deg.com Why Scala? • Open Source • Strong typing • Concise elegant syntax • Runs on JVM (Java Virtual Machine) • Supports both object-oriented and functional • Small simple programs (REPL) through very large multi-server systems • Easy to cleanly extend with new libraries and DSL’s • Ideal for parallel and distributed systems 5
  6. 6. 47deg.com Scala: Strong Typing and Concise Syntax • Strong typing like Java. • Compile time checks • Better modularity via strongly typed interfaces • Easier maintenance: types make code easier to understand • Concise syntax like Python. • Type inference. Compiler infers most types that had to be explicit in Java. • Powerful syntax that avoid much of the boilerplate of Java code (see next slide). • Best of both worlds: safety of strong typing with conciseness (like Python). 6
  7. 7. 47deg.com Scala Case Class • Java version
 
 class User {
 private String name;
 private Int age;
 public User(String name, Int age) {
 this.name = name; this.age = age;
 }
 public getAge() { return age; }
 public setAge(Int age) { this.age = age;}
 }
 User joe = new User(“Joe”, 30);
 • Scala version
 
 case class User(name:String, var age:Int)
 val joe = User(“Joe”, 30) 7
  8. 8. 47deg.com Key Scala Features • Immutable Collections • Seq, Set, Map • Functional Programming • (i:Int) => i + 1 • Functions can be parameters and results • For collections: map, filter, flatMap, groupBy, … • Case Classes • case class Person(name:String, age:Int) • Define Domain Specific Languages (DSLs) • Akka, SBT, Spray, ScalaTest and now Spark 8
  9. 9. 47deg.com View Sample Scala Code • Word Count in Scala 9
  10. 10. Spark 10
  11. 11. 47deg.com Spark History • 2009 Mesos developed at UC Berkeley AmpLab • 2009 Matei Zaharia starts Spark as a test case for Mesos • 2012 Spark 0.5 • 2014 Spark 1.0, Top Level Apache Project • 2014 Databricks started • 2016 Spark 1.6 11
  12. 12. 47deg.com Why Spark? • Support for not only batch but also (near) real-time • Fast - keeps data in memory as much as possible • Often 10X to 100X Hadoop speed • A clean easy-to-use API • A richer set of functional operations than just map and reduce • A foundation for a wide set of integrated data applications • Can recover from failures - recompute or (optional) replication • Scalable for very large data sets and reduced time 12
  13. 13. 47deg.com Spark Components • Spark Core • Scalable multi-node cluster • Failure detection and recovery • RDDs, Dataframes and functional operations • MLLib - for machine learning • linear regression, SVMs, clustering, collaborative filtering, dimension reduction • more on the way! • GraphX - for graph computation • Streaming - for near real-time 13
  14. 14. 47deg.com Spark RDDs • RDD[T] - resilient distributed data set • typed • immutable • ordered • can be processed in parallel • lazy evaluation - permits more global optimizations • Rich set of functional operations ( a small sample) • map, flatMap, reduce, … • filter, groupBy, sortBy, take, drop, … 14
  15. 15. 47deg.com Spark Data Frames • Similar to SQL tables • can be transformed using SQL • Focus of much of the work on Spark performance optimization • Unlike RDD’s, optimized knows about fields • Dynamically typed • Not a natural fit for Scala (more on this later) 15
  16. 16. 47deg.com View Sample Spark Code - RDDs • Word Count for Spark written Scala 16
  17. 17. 47deg.com View Sample Spark Code - Data Frames • Tweet language count using Data Frames written in Scala 17
  18. 18. Scala Impact on Spark 18
  19. 19. 47deg.com Spark Impact on Scala • Written in Scala • Scala is primary API • Must use Scala to extend • Source code as documentation: Scala code • Design from Scala • functional programming • immutable collections • Spark is a Scala DSL 19
  20. 20. 47deg.com Spark Application Language Choices - 1 • 71% Scala, 58% Python • Code length similar • Scala faster for RDDs (no need to move serialized data between JVM and Python) • Scala is generally faster • Scala provides strong typing • Compile time checks • Code is easier to understand • Types help when writing and maintaining code 20
  21. 21. 47deg.com Spark Application Language Choices- 2 • In addition to Scala and Python also supports Java and R • If you want to build scalable high performance production code based on Spark • R by itself is too specialized • Python is too slow • Java is tedious to write and maintain • Scala is “just right” 21
  22. 22. 47deg.com Transformations: Parallel versus Serial • Scala has operations that traverse sequence elements in serial order: foldLeft, scanLeft • Spark RDDs can only process sequence elements in parallel • Enables full use of multiple works with multiple cores • Serial operations like foldLeft would be slow compared to current parallel operations • But there are cases where foldLeft or scanLeft is really needed • Example: time series where early event can affect later event • Example: Sherlock, word count by story • So should Spark add foldLeft or similar operators? 22
  23. 23. 47deg.com Sample Spark Internal Code • Lets examine the internal code in Spark • Spark Context • RDD 23
  24. 24. Spark Impact on Scala 24
  25. 25. 47deg.com Pair RDD’s for Scala 25 Scala Sequence Key Value Unique Set Map Duplicate Seq ??? Odersky Spark Sequence Key Value Unique Duplicate RDD Pair RDD .distinct
  26. 26. 47deg.com Lazy Evaluation • Strict evaluation (Scala) • Each transformation is evaluated as it is seen • Easy to understand and debug • Lazy evaluation (Spark) • Transformations are collected into a linage graph • Evaluated only when a final action is applied • Allows cross transformation optimization 26
  27. 27. 47deg.com Lazy Evaluation in Scala • Scala has Stream (a kind of sequence) • Elements are evaluated in order as needed • Allows infinite collections • But no cross transform optimization • Lazy Spark evaluation does all elements at once • Enables parallelism • Scala may add lazy versions of its collections that are more like Spark (Odersky) 27
  28. 28. 47deg.com © Copyright 2015 47 Degrees Spark Clusters - Serialization 28 Your Spark App Spark Context Cluster Manager Worker Node Executer Task Task Spark Driver Worker Node Executer Task Task ... Static Code Jar File Dynamic Code Closures
  29. 29. 47deg.com Dynamic Code Serialization • Depends on values known at run-time • Often a function passed to Spark transformations such as map, filter, and flatMap • Sent from driver to workers • Serialized on driver to byte array • Deserialized on each work 29
  30. 30. 47deg.com Serialization of Closures • Often include more than needed (or expected) • Demo • Scala may become smarter about including less in closure serialization (Odersky) 30
  31. 31. 47deg.com Datasets • DataFrames are becoming the focus of Spark optimization • DataFrames are dynamically typed • Scala API would be nicer if there was static typing • Datasets (new experimental in Spark 1.6) attempt to solve this • Sample Code • It would be nice if DataBricks and Typesafe could work together to produce a better solution 31
  32. 32. Scala and Spark in Seattle 32
  33. 33. 47deg.com Scala and Spark in Seattle • Seattle Meetups • Scala at the Sea Meetup (over 1000 members)
 http://www.meetup.com/Seattle-Scala-User-Group/ • Seattle Spark Meetup (over 1400 members)
 http://www.meetup.com/Seattle-Spark-Meetup/ • 47 DegreesTraining (Seattle and Worldwide) 
 http://www.47deg.com/events#training • Typesafe Scala Training: Scala, Akka • Spark Training: Programming Spark with Scala • UW Scala Professional Certificate Program 
 http://www.pce.uw.edu/certificates/scala-functional-reactive-programming.html 33
  34. 34. Summary 34
  35. 35. 47deg.com Summary • Scala and Spark are great technologies for big data applications • Both are functional • Both have immutable Data • Both can be used to build very large systems • Typesafe and Databricks evolving relationship • Databricks is a consumer of Typesafe components • Typesafe now supports Spark in addition to the Scala compiler and other Scala components • They are working together to evolve toward ever more coordinated and integrated ecosystem for big data • Martin Odersky - Spark — The Ultimate Scala Collections 35
  36. 36. Questions 36

×