Building Data Quality pipelines with Apache Spark and Delta Lake
Technical Leads and Databricks Champions Darren Fuller & Sandy May will give a fast paced view of how they have productionised Data Quality Pipelines across multiple enterprise customers. Their vision to empower business decisions on data remediation actions and self healing of Data Pipelines led them to build a library of Data Quality rule templates and accompanying reporting Data Model and PowerBI reports. With the drive for more and more intelligence driven from the Lake and less from the Warehouse, also known as the Lakehouse pattern, Data Quality at the Lake layer becomes pivotal. Tools like Delta Lake become building blocks for Data Quality with Schema protection and simple column checking, however, for larger customers they often do not go far enough. Notebooks will be shown in quick fire demos how Spark can be leverage at point of Staging or Curation to apply rules over data. Expect to see simple rules such as Net sales = Gross sales + Tax, or values existing with in a list. As well as complex rules such as validation of statistical distributions and complex pattern matching. Ending with a quick view into future work in the realm of Data Compliance for PII data with generations of rules using regex patterns and Machine Learning rules based on transfer learning.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building Data Quality pipelines with Apache Spark and Delta Lake
Similar to Building Data Quality pipelines with Apache Spark and Delta Lake (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Building Data Quality pipelines with Apache Spark and Delta Lake
- 1. Building Data Quality pipelines with Apache Spark and Delta Lake Sandy May & Darren Fuller Lead Data Engineers Elastacloud
- 2. Speaker Bio Sandy May - @spark-spartan Databricks Champion Data Science London Co-Organizer Tech speaker across the UK Passionate about Apache Spark, Databricks, AI, Data Security and Reporting platforms in Microsoft Azure
- 3. Speaker Bio Darren Fuller - @dazfuller Databricks Champion Tech speaker across the UK Passions include Apache Spark, Microsoft Azure, Raspberry Pi
- 4. Agenda Sandy May What is the problem? What do we need? How can we make it easy to use? Darren Fuller How can we investigate? Where do we go from here? What have we learnt
- 6. What is the problem? • Harvard Business review suggested Dirty Data cost US companies $3 trillion in 2017 • Business data is hard to clean generically, it often requires domain knowledge • Dirty Data can be frustrating for Data Scientists and BI Engineers • In the worst case, Dirty Data can provide incorrect reports and predictions leading to potential significant losses @dazfuller @spark_spartan
- 7. Should we Build or Buy? § Own the IP § Prioritise the features you want § Built for your use case § No licence fees § Use your core technology § May have track record § Bugs fixed by vendor § Features not thought about by business § Service Level Agreements Buy Build @dazfuller @spark_spartan
- 8. Key Design Decisions ▪ Support to Run Cross Cloud ▪ Use Native tools in Azure and AWS ▪ Easy for SQL Devs to write rules ▪ Single Reporting Platform ▪ Capability to reuse custom business rules ▪ Run as part of our Data Ingestion Pipelines with Delta Lake @dazfuller @spark_spartan
- 9. Enterprise Data Warehouse @dazfuller @spark_spartan
- 10. Let’s Build it! @dazfuller @spark_spartan
- 12. Conclusions ▪ Building can be quick and effective ▪ Prioritise your own business needs, you know your data best ▪ Can be used as a stop gap while you create a service for an off the shelf product ▪ Easy to run as part of ingestion pipelines ▪ Business value in reports and reuse of rules ▪ Use Delta Lake for Schema Evolution @dazfuller @spark_spartan
- 13. Thanks for listening! Questions? @dazfuller @spark_spartan
- 14. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.