Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

2,170 views

Published on

This introductory workshop is aimed at data analysts & data engineers new to Apache Spark and exposes them how to analyze big data with Spark SQL and DataFrames.

In this partly instructor-led and self-paced labs, we will cover Spark concepts and you’ll do labs for Spark SQL and DataFrames
in Databricks Community Edition.

Toward the end, you’ll get a glimpse into newly minted Databricks Developer Certification for Apache Spark: what to expect & how to prepare for it.

* Apache Spark Basics & Architecture
* Spark SQL
* DataFrames
* Brief Overview of Databricks Certified Developer for Apache Spark

Published in: Technology
  • Be the first to comment

Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3

  1. 1. Spark Saturday: Spark SQL & DataFrames Workshop w/ Apache Spark 2.3 Jules S. Damji Apache Spark Developer & Community Advocate Spark Saturday , Santa ClaraAugust 4th,2018
  2. 2. SSID: Password:
  3. 3. I have used Apache Spark Before…
  4. 4. I have used SQL or Spark SQL Before…
  5. 5. I know the difference between DataFrame and RDDs…
  6. 6. Spark Community &Developer Advocate @ Databricks Developer Advocate @ Hortonworks Software engineering @: Sun Microsystems, Netscape, @Home, VeriSign, Scalix, Centrify, LoudCloud/Opsware, ProQuest https://www.linkedin.com/in/dmatrix @2twitme, jules@databricks.com
  7. 7. Morning Afternoon Agenda for the day • Introduction to DataFrames & Datasets • DataFrames Labs • Break • DeveloperCertification • Get to know Databricks • Overview of Spark Fundamentals & Architecture • Unified APIs:SparkSessions, SQL, DataFrames, Datasets… • Break • Spark SQL Labs • Lunch
  8. 8. Know Thy Neighbor! J
  9. 9. Get to know Databricks • Keepthis URL Open in a separate tab https://dbricks.co/spark-saturday-bayarea • Labs @Copyrightedby Databricks. Cannotbe repurposedfor Commercialuse! Use This
  10. 10. Why Apache Spark?
  11. 11. Big Data Systems of Yesterday… MapReduce/Hadoop Generalbatch processing Drill Storm Pregel Giraph Dremel Mahout Storm Impala Drill . . . Specialized systems for newworkloads Hard to combine in pipelines
  12. 12. MapReduce Generalbatch processing Unified engine Big Data Systems Today ? Pregel Dremel Millwheel Drill Giraph ImpalaStorm S4 . . . Specialized systems for newworkloads
  13. 13. Faster, Easier to Use, Unified 13 First Distributed Processing Engine Specialized Data Processing Engines Unified Data Processing Engine
  14. 14. Apache Spark Philosophy Unified engine for complete data applications High-level user-friendly APIs SQLStreaming ML Graph … DL Applications
  15. 15. An Analogy …. New applications
  16. 16. Unified engineacross diverse workloads & environments
  17. 17. Apache Spark: The First Unified Analytics Engine Runtime Delta Spark Core Engine Big Data Processing ETL + SQL + Streaming Machine Learning MLlib + SparkR Uniquelycombines Data & AI technologies
  18. 18. DATABRICKS WORKSPACE Databricks Delta ML Frameworks DATABRICKS CLOUD SERVICE DATABRICKS RUNTIME Reliable & Scalable Simple & Integrated Databricks Unified Analytics Platform APIs Jobs Models Notebooks Dashboards End to end ML lifecycle
  19. 19. Where Apache Spark is Used?
  20. 20. Common Spark Use Cases 2 ETL MACHINE LEARNING SQL ANALYTICS STREAMING
  21. 21. The Benefits of Apache Spark? SPEED 100x faster than Hadoop for large scale data processing EASE OF USE Simple APIs for operating on large data sets UNIFIED ENGINE Packaged withhigher- level libraries (SQL, Streaming, ML, Graph)
  22. 22. Spark in the Enterprise 22
  23. 23. Apache Spark at Massive Scale 23 60TB+ Compressed data 250,000+ # of tasks in a single job 4.5-6x CPU performance improvement over Hive https://databricks.com/blog/2016/08/31/apache- spark-scale-a-60-tb-production-use-case.html
  24. 24. Apache Spark Architecture
  25. 25. Apache Spark Architecture Deployments Modes • Local • Standalone • YARN • Mesos
  26. 26. Driver + Executor Driver + Executor Container EC2 Machine Student-1 Notebook Student-2 Notebook Container JVM JVM Local Mode in Databricks
  27. 27. 30 GB Container 30 GB Container 22 GB JVM 22 GB JVM S S S S S S S S Ex. Ex. 30 GB Container 30 GB Container 22 GB JVM 22 GB JVM S S S S Dr Ex. ... ... Standalone Mode
  28. 28. Spark Deployment Modes As spark 2.3 Kubernetes
  29. 29. Native Spark App in K8S • New Spark scheduler backend • Driver runs in a Kubernetes pod created by the submission client and creates pods that run the executors in response to requests from the Spark scheduler. [K8S-34377] [SPARK-18278] • Make direct use of Kubernetes clusters for multi-tenancy and sharing through Namespaces and Quotas, as well as administrative features such as Pluggable Authorization, and Logging. 29
  30. 30. Spark on Kubernetes Supported: • Supports Kubernetes1.6 and up • Supports cluster mode only • Staticresource allocation only • Supports Java and Scala applications • Can use container-local and remote dependencies that are downloadable 30 In roadmap (2.4): • Client mode • Dynamic resource allocation + external shuffle service • Python and R support • Submission client local dependencies + Resource staging server (RSS) • Non-secured and KerberizedHDFS access (injection of Hadoop configuration)
  31. 31. Apache Spark Application Anatomy
  32. 32. Apache Spark Architecture An Anatomy ofan Application Spark Application • Jobs • Stages • Tasks
  33. 33. S S Container S* * * * * * * * JVM T * * DF/RDD A Spark Executor
  34. 34. Resilient Distributed Dataset (RDD)
  35. 35. What are RDDs?
  36. 36. A Resilient Distributed Dataset (RDD) 1. Distributed Data Abstraction Logical Model Across Distributed Storage S3, Blob or HDFS
  37. 37. 2. Resilient & Immutable RDD RDD RDDT RDD à Tà RDD -> RDD T T = Transformation
  38. 38. 3. Compile-time Type-safe Integer RDD String or Text RDD Double or Binary RDD
  39. 39. 4. Unstructured/Structured Data: Text (logs, tweets, articles, social)
  40. 40. 5. Lazy RDD RDD RDDT RDD à Tà RDD à Tà RDD T T = Transformation A = Action RDD RDD RDDT A
  41. 41. 2 kinds of Actions collect, count, reduce, take, show..saveAsTextFile, (HDFS, S3, SQL, NoSQL, etc.)
  42. 42. Unified API Foundation for the Future: SparkSessions, DataFrame, Dataset, MLlib, Structured Streaming
  43. 43. Major Themes in Apache Spark 2.x TungstenPhase 2 speedupsof 5-10x & Catalyst Optimizer Faster StructuredStreaming real-time engine on SQL / DataFrames Smarter Unifying Datasets and DataFrames & SparkSessions Easier
  44. 44. SparkSession – A Unified entry point to Spark • Conduit to Spark – Creates Datasets/DataFrames – Reads/writes data – Works with metadata – Sets/gets Spark Configuration – Driver uses for Cluster resource management
  45. 45. SparkSession vs SparkContext SparkSessions Subsumes • SparkContext • SQLContext • HiveContext • StreamingContext • SparkConf
  46. 46. DataFrame & Dataset Structure
  47. 47. Long Term • RDD as the low-level API in Spark • For control and certain type-safety in Java/Scala • Datasets & DataFrames give richer semantics & optimizations • For semi-structured data and DSL like operations • New libraries will increasingly use these as interchange format • Examples: Structured Streaming, MLlib, GraphFrames, and Deep Learning Pipelines
  48. 48. Spark 1.6 vs Spark 2.x
  49. 49. Spark 1.6 vs Spark 2.x
  50. 50. DataFrames/Dataset, Spark SQL & Catalyst Optimizer
  51. 51. The not so secret truth… SQL is not about SQL is about more thanSQL
  52. 52. 10 Is About Creating and Running Spark Programs Faster: •  Write less code •  Read less data •  Do less work • optimizerdoes the hard work Spark SQL: The wholestory
  53. 53. Spark SQL Architecture Logical Plan Physical Plan Catalog Optimizer RDDs … Data Source API SQL DataFrames Code Generator Datasets
  54. 54. 59 Using Catalyst in Spark SQL Unresolved Logical Plan Logical Plan Optimized Logical Plan RDDs Selected Physical Plan Analysis Logical Optimization Physical Planning CostModel Physical Plans Code Generation Catalog Analysis: analyzinga logicalplan to resolve references Logical Optimization: logicalplan optimization Physical Planning: Physical planning Code Generation:Compileparts of the query to Java bytecode SQL AST DataFrame Datasets
  55. 55. LOGICAL OPTIMIZATIONS PHYSICAL OPTIMIZATIONS Catalyst Optimizations • Catalyst compiles operations into physical plan for execution and generates JVM byte code • Intelligently choose between broadcast joins and shuffle joins to reduce network traffic • Lower level optimizations: eliminate expensive object allocations and reduce virtual functions calls • Push filter predicate down to data source, so irrelevant data can be skipped • Parquet: skip entire blocks, turn comparisons into cheaper integer comparisons via dictionary coding • RDMS: reduce amount of data traffic by pushing down predicates
  56. 56. PhysicalPlan with Predicate Pushdown and Column Pruning join optimized scan (events) optimized scan (users) LogicalPlan filter join PhysicalPlan join scan (users)events file userstable 61 scan (events) filter users.join(events, users("id") === events("uid")) . filter(events("date") > "2015-01-01") DataFrame Optimization
  57. 57. Columns: Predicate pushdown spark.read .format("jdbc") .option("url", "jdbc:postgresql:dbserver") .option("dbtable", "people") .load() .where($"name" === "michael") 62 You Write Spark Translates For Postgres SELECT * FROM people WHERE name = 'michael'
  58. 58. Columns: Predicate pushdown SELECT firstName, LastName, SSN, COO, title FROM people where firstName = ‘jules’ and COO = ‘tz’; 63 You Write SparkWill Push it down To Postgres or Parquet SELECT <items, item, …items > FROM people WHERE <condition>
  59. 59. 43 Spark Core (RDD) Catalyst & Tungsten DataFrame/DatasetSQL MLPipelines Structured Streaming { JSON } JDBC andmore… FoundationalSpark2.x Components Spark SQL GraphFrames DL Pipelines TensorFrames
  60. 60. Spark SQL Lab (Pair Up J)
  61. 61. DataFrames & Datasets Spark 2.x APIs
  62. 62. Background: What is in an RDD? •Dependencies • Partitions (with optional localityinfo) • Compute function: Partition =>Iterator[T] Opaque Computation & Opaque Data
  63. 63. Structured APIs In Spark 68 SQL DataFrames Datasets Syntax Errors Analysis Errors Runtime Compile Time Runtime Compile Time Compile Time Runtime Analysis errors are reported before a distributed job starts
  64. 64. Unification of APIs in Spark 2.0
  65. 65. DataFrame API code. // convert RDD -> DF with column names val df = parsedRDD.toDF("project", "page", "numRequests") //filter, groupBy, sum, and then agg() df.filter($"project" === "en"). groupBy($"page"). agg(sum($"numRequests").as("count")). limit(100). show(100) project page numRequests en 23 45 en 24 200
  66. 66. Take DataFrame à SQL Table à Query df. createOrReplaceTempView(("edits") val results = spark.sql("""SELECT page, sum(numRequests) AS count FROM edits WHERE project = 'en' GROUP BY page LIMIT 100""") results.show(100) project page numRequests en 23 45 en 24 200
  67. 67. Easy to write code... Believe it! from pyspark.sql.functions import avg dataRDD = sc.parallelize([("Jim", 20), ("Anne", 31), ("Jim", 30)]) dataDF = dataRDD.toDF(["name", "age"]) # Using RDD code to compute aggregate average (dataRDD.map(lambda (x,y): (x, (y,1))) .reduceByKey(lambda x,y: (x[0] +y[0], x[1] +y[1])) .map(lambda (x, (y, z)): (x, y / z))) # Using DataFrame dataDF.groupBy("name").agg(avg("age")) name age Jim 20 Ann 31 Jim 30
  68. 68. Why structure APIs? data.map { case (dept, age) => dept -> (age, 1) } .reduceByKey { case ((a1, c1), (a2, c2)) => (a1 + a2, c1 + c2)} .map { case (dept, (age, c)) => dept -> age / c } select dept, avg(age) from data group by 1 SQL DataFrame RDD data.groupBy("dept").avg("age")
  69. 69. Type-safe:operate on domain objects with compiled lambda functions 8 Dataset API in Spark 2.x v a l d f = s p a r k .r e ad.j s on( "pe opl e.js on ") / / Convert data to domain o b j e c ts . case c l a s s Person(name: S tr i n g , age: I n t ) v a l d s : Dataset[Person] = d f.a s [P e r s on ] v a l fi l te r D S = d s . f i l t e r ( p = > p . a g e > 30)
  70. 70. Datasets: Lightning-fast Serialization with Encoders
  71. 71. DataFrames are Faster than RDDs
  72. 72. Datasets < Memory RDDs
  73. 73. Why When DataFrames & Datasets • StructuredData schema • Code optimization & performance • Space efficiency with Tungsten • High-level APIs and DSL • StrongType-safety • Ease-of-use & Readability • What-to-do
  74. 74. Source: michaelmalak
  75. 75. BLOG: http://dbricks.co/3-apis Spark Summit Talk: http://dbricks.co/summit-3aps
  76. 76. DataFrame Lab (Pair Up J)
  77. 77. Databricks Developer Certification for Apache Spark 2.x
  78. 78. 83 Why: Build Your Skills - Certification ● The industry standard for Apache Spark certification from original creators at Databricks ○ Validate your overall knowledge on Apache Spark ○ Assure clients that you are up-to-date with the fast moving Apache Spark project with features in new releases
  79. 79. 84 What: Build Your Skills - Certification ● Databricks Certification Exam ○ The test is approximately 3 hours and is proctored either online or at a test center ○ Series of randomly generated multiple choice questions ○ Test fee is $300 ○ Two editions: Scala & Python ○ Can take it twice
  80. 80. 85 How To Prepare for Certification • Knowledge of Apace Spark Basics • Structured Streaming, Spark Architecture, MLlib, Performance & Debugging, Spark SQL, GraphFrames, Programming Languages (offered Python or Scala) • Experience Developing Spark apps in production • Courses: • Databricks Apache Spark Programing 105 & 110 • Getting Started with Apache Spark SQL • 7 Steps for a Developer to Learn Apache Spark • Spark: The Definitive Guide
  81. 81. 86 Where To Sign for Certification REGISTER: Databricks Certified Developer: Apache Spark 2.X LOGISTICS: How to Take the Exam
  82. 82. https://dbricks.co/developer-cert
  83. 83. Resources • Getting Started Guide with Apache Spark on Databricks • docs.databricks.com • Spark Programming Guide • Structured Streaming Programming Guide • Databricks Engineering Blogs • spark-packages.org
  84. 84. http://dbricks.co/spark-guide
  85. 85. https://databricks.com/company/careers
  86. 86. Do you have any questions for my preparedanswers?

×