IMCSummit 2015 - Day 1 Developer Track - Building Fast, Scalable Machine Learning Pipelines
•
2 likes•755 views
Learn how at Workday, we're using Apache Spark to build predictive applications. We'll discuss how we accelerated our development and enabled a new class of applications by moving from a disk-based Hadoop platform to an all in-memory Spark-based data pipeline. We'll go over some of the common pitfalls that can affect large scale machine learning projects, particularly when using historical data sets, and how in-memory processing changes the way we reason about our application architecture.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (10)
More from In-Memory Computing Summit
More from In-Memory Computing Summit (20)
Recently uploaded
Recently uploaded (20)
IMCSummit 2015 - Day 1 Developer Track - Building Fast, Scalable Machine Learning Pipelines
- 1. Building Fast, Scalable Machine Learning Pipelines Vlad Giverts Sr Director of Software Engineering, Workday
- 18. 18 Parse Normalize Facebook Data Identified Data Pipeline 1.0
- 19. 19 Parse Normalize Facebook Data Identified Data Pipeline 1.0 Index
- 20. 20 Parse Normalize Facebook Data Identified Data Pipeline 1.0 Index Publish
- 21. Facebook Data Identified Data Pipeline 2.0 Twitter Data Doximity Data
- 22. Parse Facebook Data Identified Data Pipeline 2.0 Parse Twitter Data Parse Doximity Data
- 23. Parse Normalize Facebook Data Identified Data Pipeline 2.0 Parse Normalize Twitter Data Parse Normalize Doximity Data
- 24. Parse Normalize Facebook Data Identified Data Pipeline 2.0 Parse Normalize Twitter Data Parse Normalize Doximity Data Merge (ML)
- 25. Parse Normalize Facebook Data Identified Data Pipeline 2.0 Index Parse Normalize Twitter Data Parse Normalize Doximity Data Merge (ML)
- 26. Parse Normalize Facebook Data Identified Data Pipeline 2.0 Index Publish Parse Normalize Twitter Data Parse Normalize Doximity Data Merge (ML)
- 31. Retention Risk
- 40. Data and “Features” Tenure Time in Current Function Pay Range Penetration Manager Attrition Rate Num Promotions Avg Time Between Promotions
- 44. Training and Validation Barry Raise: $1,000 2014 2016 Ravi Left :( John Left :( Albert Promoted! Yury Hired Tejas Changed Teams
- 45. Training and Validation Q1 ‘14 Q2 ‘14 Q3 ‘14 Q4 ‘14 Q1 ‘15 Q2 ‘15
- 46. Training and Validation Q1 ‘14 Q2 ‘14 Q3 ‘14 Q4 ‘14 Q1 ‘15 Q2 ‘15 TRAINING VALIDATION
- 49. Evaluation Q1 ‘14 Q2 ‘14 Q3 ‘14 Q4 ‘14 Q1 ‘15 Q2 ‘15
- 50. Evaluation Q1 ‘14 Q2 ‘14 Q3 ‘14 Q4 ‘14 Q1 ‘15 Q2 ‘15 Q3 ‘15 Q4 ‘15
- 51. Evaluation Q1 ‘14 Q2 ‘14 Q3 ‘14 Q4 ‘14 Q1 ‘15 Q2 ‘15 Q3 ‘15 Q4 ‘15 PREDICTION