Downloaded 21 times
Uploaded byGrid Dynamics
Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov
The document discusses the implementation of real-time data quality enforcement using machine learning technologies at Grid Dynamics. It highlights the importance of trust in data for effective decision-making and outlines various methods for detecting and preventing data corruption. Additionally, it emphasizes the need for automated data profiling, anomaly detection, and governance to enhance data security and regulatory compliance.
More Related Content
Similar to Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov
Dynamic Talks: "Implementing data quality automation with open source stack" -Max Martynov
- 1. 2 Data Quality Automatic enforcementin real-time with machine learning. Max Martynov, VP of Technology
- 2. Introducing Grid Dynamicstechnology services Digital transformation Big data, real time analytics, ML & AI Microservices replatforming DevOps & cloud enablement Open Source Cloud-ready Scalable Automated
- 3. 12 years of experiencein digital transformation.
- 5. 9.8 9.3 9.6 9.4 10.1 17.5 16.9 8.99.6 9.2 10.3 9.8 5.1 4.1 3.9 4 3.9 4.5 7.5 7.1 3.8 4.1 4 4.5 4.2 2.3 0 5 10 15 20 7/1/19 7/2/19 7/3/19 7/4/19 7/5/19 7/6/19 7/7/19 7/8/19 7/9/19 7/10/19 7/11/19 7/12/19 7/13/19 7/14/19 Retailer X, Daily Sales – Executive Summary Revenue, $M Gross Profit, $M weekend weekend
- 6.
- 7.
- 8. Cloud Data Lake EDW DBDBDB FileFileFileMQ CloudCloud AppAppAPI
- 9. Cloud EDW DBDBDB FileFileFile MQCloudCloud AppAppAPI AppAppApp Data Lake
- 10. 1 0 11 0 1 1 0 1 0 1 0 1 0 0 1 1. Trust is hard to build and easy to lose. 2. Distrust in data slows down decisions. 3. Slow decisions prevent agility.
- 11. Data corruption reasons 1.Code 2. Data Sources 3. Infrastructure
- 12. Test environment Input Actual Expected ETL code compareTest suite runtest Traditional approach to testing
- 13. Data quality goals Detect datacorruption Prevent it from spreading Alert support team
- 14. Production data lake Real-timedata quality enforcement DBDBDB FileFileFile MQ AppAppAPI data processing job
- 15. Production data lake Real-timedata quality enforcement DBDBDB FileFileFile MQ AppAppAPI data processing job data quality job
- 16. Production data lake Real-timedata quality enforcement DBDBDB FileFileFile MQ AppAppAPI data processing job data quality job alert & stop pipeline alert & continue x
- 17. Data Lake Data source data 1. Comparewith SoR 2. Validate business rules 3. Data profiling and anomaly detection Main data processing pipeline confidence data confidence
- 18. 1. Control divergencefrom SoR Data Lake Data source Imported dataset Compare data in SoR and data lake 1. Validate correctness of import. 2. Prevent stale data. 3. Prevent corruption accumulation in stream processing use cases. 4. Check data before it gets in the lake.
- 19. 2. Validate businessrules Data Lake Dataset Check for nulls and data ranges 1. Enforce schema. 2. Check for nulls. 3. Validate data ranges. 4. Specify and enforce data invariants.
- 20. 3. Anomaly detection DataLake Dataset 1. Fully automatic data quality enforcement. 2. Collect data profile, metrics and statistics. 3. Train ML models. 4. Find anomalies in data. Data profiling and anomaly detection
- 21. Catalog Inventory Orders Data Lake DataQuality Reporting & Alerting Data Profile Demo setup
- 22.
- 23.
- 24. Anomaly detection example:zooming in
- 25. Capabilities for enterprisedata quality and governance Enables widespread adoption. Enforces enterprise-level controls and data usage policies. Increases consistency and confidence in decision making. Decreases the risk of regulatory fines. Improves data security. Facilitates accountability for information quality. Minimizes or eliminates efforts duplication. Data Governance Platform Metadata Management Full-text Search Data Quality Status Schema / Summary Data Profiling Mapping to Glossary Change Log Dependency Detection Consumers Flow Visualization Glossary Portal Knowledge Base Fields Fingerprinting Data Catalog Dataset Profile Lineage Dashboard Data Glossary Data Quality Access and Security Business Rules Anomaly Detection Alerting Access Rules Compliance Policies Policy Engine
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33. Anomaly detection: returnto normal
- 34.
- 35. Anomaly detection (counts):anomaly & return
- 36.
- 37.
- 38. Uniqueness: return tonormal
- 39.
- 40.
- 41.
- 42. Nulls: return tonormal
- 43.
- 44.
- 45.