Recommended
More Related Content
Similar to Apache Spark Data intensive processing in practice
Similar to Apache Spark Data intensive processing in practice (20)
Recently uploaded
Recently uploaded (20)
Apache Spark Data intensive processing in practice
- 2. www.tantusdata.com About me • Data Engineer @TantusData • Have worked for: Spotify, Apple, telcos, startups • Cluster installations, application architecture and development, training, data team support marcin@tantusdata.com marcin.szymaniuk@gmail.com @mszymani
- 3. www.tantusdata.com Agenda • What is Spark? • Use case overview • Architecture • Big picture
- 4. www.tantusdata.com What is Spark? • Engine for distributed data processing • Java, Scala, R, Python, SQL API • Streaming, Machine Learning
- 10. www.tantusdata.com Spark API - DataFrame eventsDf .groupBy("userId") .agg( max("value").alias("maxVal"), avg("value").alias("avgValue") ) .join(usersDf, usersDf("id") === eventsDf("userId")) .select("userId", "maxVal", "avgValue","name")
- 11. www.tantusdata.com Spark API - DataFrame eventsDf .groupBy("userId") .agg( max("value").alias("maxVal"), avg("value").alias("avgValue") ) .join(usersDf, usersDf("id") === eventsDf("userId")) .select("userId", "maxVal", "avgValue","name")
- 13. www.tantusdata.com General use cases KPI ACQUISITION ACTIVATION RETENTION REFERRAL REVENUE
- 14. www.tantusdata.com General use cases • A/B tests • Anonymization • Fraud • Churn • ML • …
- 16. www.tantusdata.com Network improvement • Score historical customer network quality • Define a model predicting churn based on historical score • Simulate base station upgrade and calculate expected score after the upgrade • Use the simulated score with churn prediction model
- 17. www.tantusdata.com Bring analysis to data DATA R / PYTHON / SAS Sample • Sample only (region, latest month…) • Coarse aggregate eg. month vs hour (1:720)
- 18. www.tantusdata.com Bring analysis to data Photo credit: productcoalition.com
- 19. www.tantusdata.com Bring analysis to data DATA • Analyze all data • Faster analysis • No extra data copies (GDPR!) • Many solutions are already implemented (MLib, GraphX…)
- 20. www.tantusdata.com Geospatial data • General map service • Self driving cars
- 21. www.tantusdata.com Geospatial data Map V1Car + AI Map V2 Map OSMap OS Editors Map V2 Vendors
- 26. www.tantusdata.com Spark use cases - recap • Massive datasets processing - distribute it! • Computation intensive processing - distribute it! • SQL-like interface - analyst friendly • Functional programming for complex logic
- 27. www.tantusdata.com Deep dive • Spark execution model • Partitioning • Caching
- 28. www.tantusdata.com Deep dive Photo credit: amazon.ca
- 29. www.tantusdata.com RDD / DataFrame RDD Partition 1 Partition 2 Partition 3
- 30. www.tantusdata.com Narrow transformation f:x toUpperCase(x) rdd.map(f) foo Partition 1 Partition 2 Partition 3 RDD Bar baz blah blah LOREm ipsum dolor sit amet FOO Partition 1 Partition 2 Partition 3 New RDD BAR BAZ BLAH BLAH LOREM IPSUM DOLOR SIT AMET
- 31. www.tantusdata.com Narrow transformation RDD Partition 1 NEW RDD New Partition 1 NEW RDD2 Partition 1 Partition 2 New Partition 2 Partition 2 Partition 3 New Partition 3 Partition 3
- 32. www.tantusdata.com Partitions, tasks RDD Partition 1 NEW RDD Partition 1 NEW RDD2 Partition 1 Partition 3 Partition 3 Partition 3 Partition 2 Partition 2 Partition 2 TASK 1TASK 1
- 33. www.tantusdata.com Partitions, tasks RDD Partition 1 NEW RDD Partition 1 NEW RDD2 Partition 1 Partition 3 Partition 3 Partition 3 Partition 2 Partition 2 Partition 2TASK 2
- 34. www.tantusdata.com Wide transformations groupByKey 1 | CLICK Partition 1 Partition 2 Partition 3 RDD 2 | VIEW 1 | SEND 1 | CLICK 3 | SEND 2 | SEND 2 | VIEW 3 | CALL 3 | SEND 1 | CLICK New RDD 1 | CALL
- 35. www.tantusdata.com Wide transformations groupByKey 1 | CLICK Partition 1 Partition 2 Partition 3 RDD 2 | VIEW 1 | SEND 1 | CLICK 3 | SEND 2 | SEND 2 | VIEW 3 | CALL 3 | SEND 1 | CLICK Partition 1 Partition 2 Partition 3 New RDD 1 | CALL 1 | CLICK 1 | SEND 1 | CLICK 1 | CLICK1 | CALL
- 36. www.tantusdata.com Wide transformations groupByKey 1 | CLICK Partition 1 Partition 2 Partition 3 RDD 2 | VIEW 1 | SEND 1 | CLICK 3 | SEND 2 | SEND 2 | VIEW 3 | CALL 3 | SEND 1 | CLICK Partition 1 Partition 2 Partition 3 New RDD 1 | CALL 1 | CLICK 1 | SEND 1 | CLICK 1 | CLICK1 | CALL 2 | VIEW 2 | SEND2 | VIEW
- 37. www.tantusdata.com Wide transformations groupByKey 1 | CLICK Partition 1 Partition 2 Partition 3 RDD 2 | VIEW 1 | SEND 1 | CLICK 3 | SEND 2 | SEND 2 | VIEW 3 | CALL 3 | SEND 1 | CLICK Partition 1 Partition 2 Partition 3 New RDD 1 | CALL 1 | CLICK 1 | SEND 1 | CLICK 1 | CLICK1 | CALL 2 | VIEW 2 | SEND2 | VIEW 3 | SEND 3 | CALL 3 | SEND
- 41. www.tantusdata.com Spark application STAGE 2 STAGE 1 SHUFFLE STAGE N
- 42. www.tantusdata.com Simplest scenario ever val df = spark.read.parquet(“…”) HDFS TASK
- 43. www.tantusdata.com Simplest scenario ever HDFS TASK ADDCOL ADDCOL val df = spark.read.parquet(“…”) df .withColumn("year", year(col(“timestamp"))) .withColumn("month", month(col(“timestamp”))) .withColumn("day", dayofmonth(col(“timestamp”)))
- 44. www.tantusdata.com Simplest scenario ever HDFS TASK ADDCOL ADDCOL HDFS val df = spark.read.parquet(“…”) df .withColumn("year", year(col(“timestamp”))) .withColumn("month", month(col("timestamp"))) .withColumn("day", dayofmonth(col("timestamp"))) .write.save(output)
- 45. www.tantusdata.com Simplest scenario ever 1TB of data raw events
- 46. www.tantusdata.com Simplest scenario ever 1TB of data raw events 1000 x 1GB 1GB file = 8 blocks 128MB per block
- 47. www.tantusdata.com Simplest scenario ever 1TB of data raw events 1000 x 1GB 1GB file = 8 blocks 128MB per block 8000 blocks 8000 Tasks
- 48. HDFS TASK ADDCOL ADDCOL HDFS HDFS TASK ADDCOL ADDCOL HDFS HDFS TASK ADDCOL ADDCOL HDFS HDFS TASK ADDCOL ADDCOL HDFS … 8000
- 49. www.tantusdata.com EXECUTOR —executor-cores 3 —executor-memory 10g Executors —executor-cores 3 —executor-memory 10g EXECUTOR PENDING TASKS COMPLETE TASKS DRIVER EXECUTOR
- 53. HDFS TASK Bucket 1 Bucket 2 LOCAL HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL … … … HDFS HDFS
- 54. HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL … … … HDFS HDFS
- 55. HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL TASK … … … HDFS HDFS
- 56. .config("spark.sql.shuffle.partitions", "200") HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL TASK … … … HDFS HDFS
- 57. HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL TASK … … … HDFS HDFS 10TB/200 = 50GB/TASK.config("spark.sql.shuffle.partitions", "200")
- 58. 10TB/200 = 50GB/TASK HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL TASK … … … HDFS HDFS 50GB 50GB .config("spark.sql.shuffle.partitions", "200")
- 59. www.tantusdata.com Problems with join • Spill to disk • Timeouts • GC overhead limit exceeded • OOM • ExecutorLostFailure
- 60. www.tantusdata.com What to do? • Understand your data! • Control the level of parallelism .config("spark.sql.shuffle.partitions", “2000") rdd.join(anotherRDD, 2000) .repartition(2000)
- 61. www.tantusdata.com Skewed join 10TB of events One user with 1 TB of events others are uniformly distributed
- 62. HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL TASK … … … HDFS HDFS
- 63. HDFS TASK Bucket 1 Bucket 2 LOCAL TASK HDFS TASK 2 Bucket 1 Bucket 2 LOCAL HDFS TASK X Bucket 1 Bucket 2 LOCAL TASK … … … HDFS HDFS 1TB
- 64. www.tantusdata.com Skewed join • Bad data? • Wrong logic? • Just ok? Photo credit: hiveminer.com
- 65. www.tantusdata.com Skewed Join userId … 1 2 3 … … eventId userId … af8 1 bf9 1 ff1 1 881 1 91f 2 cc6 1 b22 1 ee4 1
- 66. www.tantusdata.com userId … 1 2 3 … … eventId userId … af8 1 bf9 1 ff1 1 881 1 91f 2 cc6 1 b22 1 ee4 1 Skewed Join
- 67. www.tantusdata.com userId … 1 2 3 … … eventId userId … salt af8 1 1 bf9 1 2 ff1 1 1 881 1 3 91f 2 2 cc6 1 3 b22 1 3 ee4 1 1 Skewed Join
- 68. www.tantusdata.com userId … salt 1 1 1 2 1 3 2 1 2 2 2 3 3 1 3 2 3 3 … … Skewed Join eventId userId … salt af8 1 1 bf9 1 2 ff1 1 1 881 1 3 91f 2 2 cc6 1 3 b22 1 3 ee4 1 1
- 69. www.tantusdata.com userId … salt 1 1 1 2 1 3 2 1 2 2 2 3 3 1 3 2 3 3 … … eventId userId … salt af8 1 1 bf9 1 2 ff1 1 1 881 1 3 91f 2 2 cc6 1 3 b22 1 3 ee4 1 1 Skewed Join
- 70. www.tantusdata.com eventId userId … salt af8 1 1 bf9 1 2 ff1 1 1 881 1 3 91f 2 2 cc6 1 3 b22 1 3 ee4 1 1 Skewed Join userId … salt 1 1 1 2 1 3 2 1 2 2 2 3 3 1 3 2 3 3 … …
- 71. www.tantusdata.com userId … salt 1 1 1 2 1 3 2 1 2 2 2 3 3 1 3 2 3 3 … … Skewed Join eventId userId … salt af8 1 1 bf9 1 2 ff1 1 1 881 1 3 91f 2 2 cc6 1 3 b22 1 3 ee4 1 1
- 72. www.tantusdata.com • Know your data! • Fix the data? • Improve the logic? • Add salt? Skewed Join - recap Photo credit: shamakern.com
- 80. www.tantusdata.com Cache • Transformations are lazy! • Re-using RDD/DF means re-calculation! • Branch in execution plan is a candidate for caching • You cannot control priority - it's LRU • Know the size of your RDDs/DF - check Spark UI.
- 81. www.tantusdata.com Other gotchas • Broadcasting • Sizing executors • Locality • Off heap memory • …
- 82. www.tantusdata.com Challenges ahead • File formats • Data evolution • Jobs orchestration • Monitoring • Anomaly detection • ML models • …
- 84. www.tantusdata.com Challenges ahead OPS DEVELOPERS BUSINESS ANALYTICS Common tools and knowledge
- 85. www.tantusdata.com Challenges ahead Photo credit: thefinancialbrand.com
- 86. www.tantusdata.com Conclusions • Spark can help you with data processing at scale • You should know how it works • Think about big picture from day one