Download as PDF, PPTX
Uploaded byDatabricks
Fully Utilizing Spark for Data Validation
The document discusses data validation using various frameworks including Great Expectations, Pandera, and Fugue. It covers their features, advantages, and how they can be applied in Spark for data validation in a fictitious food delivery company case study. Key takeaways include the importance of validation, especially by partitioning, to accommodate geographic differences in data.
![제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계](https://cdn.slidesharecdn.com/ss_thumbnails/3-1boaz23rdconferencemboax-260203102709-9d519923-thumbnail.jpg?width=640&height=640&fit=bounds)
Fully Utilizing Spark for Data Validation
- 1. Fully Utilizing Spark forData Validation Kevin Kho Open Source Community Engineer at Prefect
- 2. Agenda ▪ Starting DataValidation ▪ Great Expectations ▪ Pandera ▪ Fugue ▪ Validation by Partition
- 3. Case Study • FictitiousCompany • Food Delivery • Demand Pricing per Location • Update rate every 10 minutes
- 4. Data Validation Data Validation LoadingData From Source Model Training Data Validation Loading Data From Source Infer New Price Update App Price Model Training - Weekly pipeline Price Inference - 10 minute pipeline
- 5. Common Validations • NullValues • Correct Schema • DataFrame Shape • Numeric values within range
- 6. Validation in Sparkwith Great Expectations
- 7. Great Expectations • Mostlyparameter • Same interface with Pandas • Different result formats
- 8. Great Expectations -Detailed Results
- 9. Great Expectations -Data Documentation
- 10. Is there amore lightweight framework?
- 11. Pandera • Pandas only •Built-in validations • Statistical validations • Easily extensible
- 12. Pandera - SampleCode
- 13. Comparison of ValidationFrameworks ▪ Spark support ▪ Flexible Success Criteria ▪ Detailed Outputs ▪ Data Documentation ▪ Notifications ▪ CLI ▪ Pandas only ▪ Lightweight ▪ Hypothesis Testing ▪ Decorators to Wrap Code • Pandera • Great Expectations
- 14. Using Pandera (andPandas libraries) in Spark
- 15. Fugue • Friendlier interfacethan UDF • Decouple logic and execution • Write code once and scale seamlessly
- 16.
- 17. Fugue - BasicCode
- 18. Combining Fugue andPandera
- 19.
- 20. Motivation - FoodSloth’sExpansion
- 21. Example Data -FoodSloth’s Pricing AsOfTime Location Sublocation Price 12:30 FL Tampa 9 12:30 FL Miami 7.5 12:30 FL Orlando 8.5 12:30 CA San Francisco 14.1 12:30 CA Los Angeles 12.3 12:30 CA San Mateo 11.75 12:30 CA San Diego 12.5
- 22.
- 23.
- 24. Takeaways • Data Validation •Great Expectations • Pandera • Fugue • Partition by Validation
- 25.
- 26. The need formultiple valida • Motivation • Geographic differences • Bullet 2 • Sub-bullet • Sub-bullet
- 27.
- 28. Validation by Partition •Motivation • Geographic differences • Bullet 2 • Sub-bullet • Sub-bullet
- 29.
- 30.
- 31.
- 32.
- 33. Basic Slide • Bullet1 • Sub-bullet • Sub-bullet • Bullet 2 • Sub-bullet • Sub-bullet
- 34. Reduce Long Titles •Bullet 1 • Sub-bullet • Sub-bullet • Bullet 2 • Sub-bullet • Sub-bullet By splitting them into a short title, and a more detailed subtitle using this slide format that includes a subtitle area
- 35. Two Columns ▪ Bulletedlist format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format • Headline Format Headline Format
- 36.
- 37. Two Box ▪ Bulletedlist ▪ Bulleted list ▪ Bulleted list ▪ Bulleted list • Category • Category
- 38. Three Box ▪ Bulletedlist ▪ Bulleted list • Bulleted list • Bulleted list • Category • Category • Bulleted list • Bulleted list • Category
- 39. Four Box ▪ Bulletedlist ▪ Bulleted list ▪ Bulleted list ▪ Bulleted list • Category • Category ▪ Bulleted list ▪ Bulleted list • Category ▪ Bulleted list ▪ Bulleted list • Category
- 40.
- 41. Shapes Rounded corner rectangleDouble corner rectangle Double corner rectangle
- 42.
- 43. Table Column Column Column RowValue Value Value Row Value Value Value Row Value Value Value Row Value Value Value Row Value Value Value Row Value Value Value Row Value Value Value
- 44. Bar chart 0 1 2 3 4 5 6 Category 1Category 2 Category 3 Category 4 Chart Title Series 1 Series 2 Series 3 Series 4 Series 5
- 45. Quotes and TextCallouts
- 46. Attribution Format Second lineof attribution This is a template for a quote slide. This is where the quote goes. Attribute the source below.
- 47. Line chart 0 1 2 3 4 5 6 7 Category 1Category 2 Category 3 Category 4 Chart Title Series 1 Series 2 Series 3 Series 4 Series 5 Series 6
- 48. Pie Chart Sales 1st Qtr2nd Qtr 3rdQtr 4th Qtr
- 49.
- 50. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.
- 51. Data + AISummit Logos
- 52.
- 53.
- 54.