DataOps - Production ML
•
0 likes•232 views
Do compilers look anything like a data pipeline? How do you do data testing to ensure end to end provenance and enforce engineering guarantees for your data products? What babysteps should you consider when assembling your team?
Recommended
More Related Content
What's hot
What's hot (19)
Similar to DataOps - Production ML
Similar to DataOps - Production ML (20)
Recently uploaded
Recently uploaded (20)
DataOps - Production ML
- 1. DataOps: The Key to Exponential Data Impact Luis Vaquero
- 4. Luis Vaquero Director of Data at Goco Reader in Computer Science at Univ of Bristol Dyson, HPE, HP Labs, Telefonica/O2
- 6. “20% of my time building my model, and 80% of my time cleaning data” Every Data Scientist in 2020
- 7. “20% of my time building my app, and 80% of my time fighting the browser/the infra/Spring config...” Every Web developer in 2000
- 8. https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4
- 9. Pipeline Compiler “Data pipelines are simply compilers that use data as the source code.” Ryan Gross
- 10. Each Data Team Write their Own “Compiler” https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4 data quality, metadata, raw or slightly modelled data Lexical and Syntactic Analysis
- 11. Each Data Team Write their Own “Compiler” https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4 Lexical and Syntactic Analysis Semantic Analysis business understanding, model creation and experiments data quality, metadata, raw or slightly modelled data
- 12. Each Data Team Write their Own “Compiler” https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4 Lexical and Syntactic Analysis Semantic Analysis Code Generation business understanding, model creation and experiments data quality, metadata, raw or slightly modelled data
- 13. Each Data Team Write their Own “Compiler” https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4 further tests, robustness Lexical and Syntactic Analysis Semantic Analysis Code Generation Optimisation business understanding, model creation and experiments data quality, metadata, raw or slightly modelled data
- 14. The Data Pipeline “Compiler”
- 19. http://cidrdb.org/cidr2020/papers/p8-agrawal-cidr20.pdf Providing complete and usable third-party solutions is non-trivial
- 21. Where do we test here? Pipeline ML Model Model Output Raw Data Training/ serving Data
- 24. What Kind of Tests? 1. data validator: schema conformance and evolution a. also a way to document new features used in the pipeline b. Also about trends and anomalies https://blog.acolyer.org/2019/06/05/data-validation-for-machine-learning/
- 26. What Kind of Tests? 1. data validator: schema conformance and evolution a. also a way to document new features used in the pipeline 2. data analyser: basic statistics a. Bias b. Feature/distribution skew c. ... https://blog.acolyer.org/2019/06/05/data-validation-for-machine-learning/
- 27. New Feature Engineering: “Instead of deriving the math before feeding the model, we ensure our features comply with certain properties so that the NN can do the math effectively by itself” -- Airbnb https://arxiv.org/pdf/1810.09591.pdf
- 29. What Kind of Tests? 1. data validator: schema conformance and evolution a. also a way to document new features used in the pipeline 2. data analyser: basic statistics a. Bias b. Feature/distribution skew c. ... 3. model unit tester looks for errors in the training code using synthetic data (schema-led fuzzing) https://blog.acolyer.org/2019/06/05/data-validation-for-machine-learning/
- 31. Model Analysis https://arxiv.org/pdf/1810.09591.pdf “Smooth distributions coming from the lower layers ensure that the upper layers can correctly ‘interpolate’ the behavior for unseen values…. ensure the input features have a smooth distribution” -- Airbnb
- 33. What Kind of Tests? 1. data validator: schema conformance and evolution a. also a way to document new features used in the pipeline 2. data analyser: basic statistics a. Bias b. Feature/distribution skew c. ... 3. model unit tester looks for errors in the training code using synthetic data (schema-led fuzzing) 4. monitoring tests check the output of the model to trigger alerts https://blog.acolyer.org/2019/06/05/data-validation-for-machine-learning/
- 34. Monitoring Model performance dashboard 1. model output metrics through training, validation, testing, and deployment. 2. data input metrics 3. operational telemetry Image from: https://www.parallelm.com/
- 38. What is Special about ML? 1. New Artefacts to Manage a. Data b. Metadata: Hyperparameters c. Code: architecture d. Model: executable software “built from the data” e. Experiment: Data + metadata + Hyperparams + Code -> Model 2. Different Process a. Trial and error: Scientific method b. Reproducibility - traceability c. Explainability
- 39. https://medium.com/thelaunchpad/retracing-your-steps-in-machine-learning-ml-versioning-74d19a66bd08 Using Excel to Track your versions of data, metadata, hyperparameters, and architecture?
- 41. The Data Version Control Tipping Point ● Datasets can be versioned, branched, acted upon by versioned code to create new data sets ● Test and fill bugs against data ● Enable quality control for compiler steps ● Automated lineage and schema change deceting ● Make guarantees about system components
- 47. Ad Hoc Exploration1 Tools RelationshipsProcesses - Isolated efforts - No repeatability - Siloed data - Local dev - No business buy-in - transactional - Ivory tower Tools Relationships Processes
- 48. Reproducible, but limited2 Tools RelationshipsProcesses - Repeatability is patchy - Poor governance - Shy centralisation - Static reports - Heavy transactional - rapport - Team management support only Tools Relationships Processes
- 49. Defined, Controlled3 Tools RelationshipsProcesses - Formal but manually enforced- Good centralization (metadata, access) - Live retrospective reports - Incipient experimentation - Empathy Tools Relationships Processes
- 50. Automated4 Tools RelationshipsProcesses - Automated and searchable - Broad accessibility - Best practice - Event-driven - Wide experimentation - Analytics is the business Tools Relationships Processes