R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk by: Maximo Gurmendez , Saket Mengle, Sunanda Parthasarathy
The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk by: Maximo Gurmendez , Saket Mengle, Sunanda Parthasarathy
Similar to R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk by: Maximo Gurmendez , Saket Mengle, Sunanda Parthasarathy (20)
More from Spark Summit
More from Spark Summit (20)
Recently uploaded
Recently uploaded (20)
R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk by: Maximo Gurmendez , Saket Mengle, Sunanda Parthasarathy
- 1. R&D To Product Pipeline Using Apache Spark in Adtech Maximo Gurmendez Dr. Sunanda Parthasarathy Dr. Saket Mengle DataXu Inc.
- 2. What to expect from this session • Who is DataXu? • Why Apache Spark? • From R&D to product using Apache Spark Demo • Analytics using Apache Spark
- 3. DataXu Make Marketing Smarter Through Data Science • Who • Spun out of MIT Labs • A petabyte scale digital marketing platform • One of the fastest growing companies in Inc. 5000 • What • Help world’s most valuable brands understand and engage with their consumer • Maximize ROI Quick Statistics • Billions of ads served per month • ~10ms round trip response time • 130+ TB logs per day • 3000+ servers powering the platform • 13 regions, 24x7
- 6. Why is this hard? Huge Scale • 2.7 million bid decisions per second • 3 PB of data processed daily • Runs 24 X 7 on 5 Continents • Thousands of ML Models Trained per Day Unattended Operation • Model training and deployment runs automatically every day Changing Industry • Need ability to adapt quickly to new customer requirements
- 7. Demo
- 8. Benchmarks 0 100 200 300 400 500 0 5 10 15 TrainingTime(insec) Number of Training Records Millions Training Time Comparison Logistic Regression Decision Tree Random Forest 0 0.5 1 1.5 2 Current DataXu Model Spark Random Forest Avg. Bidding Latency (milliseconds) 0 20 40 60 80 100 120 140 Random Forests Logistic Regression Naive Bayes Decision Trees DataXu Model Model Size in Memory (KB)
- 9. S3 – meta data Why Apache Spark for Adv. Analytics • Makes Advanced Analytics a reality – accelerated queries, graph processing, streaming analytics • Speaks multiple languages (Python, Scala, SQL) • Makes it easy – Compared to Java/Hadoop complexities • Accelerates the analyst/data scientist workflow Real Time Bidding Engine Adv. Analytics Engine
- 10. S3 – meta data Advanced Analytics at DataXu Real Time Bidding Engine Analytics Engine Partner/Client Data Dashboarding/Reporting +}
- 11. Analytics Demo
- 12. Thank You. smengle@dataxu.com mgurmendez@dataxu.com sparthasarathy@dataxu.com !! We’re hiring !! Data Scientists, Data Science Engineers. FTEs, Interns
Editor's Notes
- Spark. I have to add Apache in front of Spark
- ?
- - Add benchmarks for Spark after demo, and add takeaways. Next steps Best practices