Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Batched To Perfection: Modeling & Solving Business Problems With Apache Spark
Similar to Batched To Perfection: Modeling & Solving Business Problems With Apache Spark (20)
Recently uploaded
Recently uploaded (20)
Batched To Perfection: Modeling & Solving Business Problems With Apache Spark
- 1. BATCHED TO Modeling & Solving Business Problems With Apache Spark February 2021 PERFECTION
- 2. AGENDA 01 The Problem Space 02 Aggregations 03 A Basic Pipeline 04 Incremental State 05 Continuous Refinement 06 After Thoughts
- 3. Eliav Lavi ➔ About 7 years at Riskified ➔ A classically trained musician ➔ A Francophile! eliav@riskified.com ABOUT ME
- 5. MEET RISKIFIED Riskified is an AI platform powering the eCommerce revolution. We have an unparalleled ability to recognize legitimate customers and keep them moving toward conversion. The world’s largest brands trust us to increase revenue, manage risk, and improve customer interactions. transactions daily total employees 130 in R&D in funding to date clients include many publicly traded companies 600+ 1M+ $229M Enterprise Focus
- 7. • BlueNote - an imagined music streaming app • We’re asked to present an aggregation per user, in the app’s UserInfo page • There are many metrics required in this project. We will focus on “top liked genres” THE PROBLEM SPACE
- 12. IN PRAISE OF AGGREGATIONS Query Everything: a big-data anti-pattern
- 13. liked_tracks track_genres IN PRAISE OF AGGREGATIONS Query Everything: a big-data anti-pattern
- 14. IN PRAISE OF AGGREGATIONS Query Everything: a big-data anti-pattern
- 15. IN PRAISE OF AGGREGATIONS Aggregation backed systems
- 16. We gain: • Faster response times • Scalability • Cleaner separation of concerns However, we pay with: • Data latency • Diminished agility • Storage & compute IN PRAISE OF AGGREGATIONS Aggregation trade-offs
- 17. SPARK 101
- 18. SPARK 101 Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. Wikipedia
- 19. • Spark itself is written in Scala • Spark Applications can be written in Scala, Java, Python or R • Dataset vs. Dataframe • Various data formats, Parquet as a default ◦ Columnar format ◦ Compression ◦ Nested data structures SPARK 101
- 20. A BASIC PIPELINE
- 21. To bring in new raw data to work with To transform new raw data into useful aggregation To allow updated aggregation to be accessible for apps A BASIC PIPELINE
- 22. A BASIC PIPELINE
- 23. A BASIC PIPELINE
- 24. A BASIC PIPELINE
- 27. A BASIC PIPELINE
- 28. A BASIC PIPELINE
- 29. A BASIC PIPELINE
- 41. • Database reads might be costly • If the process fails, we will need to pay again for reads & writes Possible caveats INCREMENTAL STATE
- 43. • ACID transactions over HDFS • Updating parquets is slow • Delta allows us to postpone optimization • Integrates perfectly with Spark • We are already using S3 How can Delta Lake help? INCREMENTAL STATE
- 45. Read Delta & join INCREMENTAL STATE
- 46. Combine aggs where applicable INCREMENTAL STATE
- 47. Semigroup is a neat, simple typeclass that allows us to describe how two instances of type A combine into a single A: (A, A) => A The Semigroup instance for Map comes from here! Semigroup for the win INCREMENTAL STATE
- 48. Agg as Semigroup INCREMENTAL STATE
- 49. Combine aggs where applicable INCREMENTAL STATE
- 50. 50 Upsert to Delta INCREMENTAL STATE
- 56. CONTINUOUS REFINEMENT Semigroup for the win, again
- 58. AFTER THOUGHTS
- 59. • Proper data: what feeds our process? • The population stage • Testability • Stack agnosticity • S3 could be GCS or other • Final DB can be anything • Bring your own scheduler AFTER THOUGHTS
- 60. RECAP
- 61. • Batch processing as a fully fledged app • Spark + Delta = ❤ • Identifying the aggregation • Continuous refinement RECAP
- 62. Eliav Lavi eliav@riskfied.com THANK YOU FOR YOUR TIME! medium.com/@eliavlavi @eliavlavi / Twitter