The document details a presentation by Premal Shah from Microsoft on creating continuous integration pipelines using Azure Databricks and Azure DevOps. It covers the fundamentals of DevOps, the functionality of Azure Databricks, and the integration of CI/CD pipelines for model management and deployment. Key takeaways include utilizing Azure tools for secure collaboration and increased productivity in data processing and machine learning implementation.

1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics

2. Premal Shah, Microsoft
Creating continuous integration pipelines on
Azure using Azure Databricks and Azure DevOps
#UnifiedAnalytics #SparkAISummit

3. What is DevOps?
3#UnifiedAnalytics #SparkAISummit

4. What is DevOps?
4#UnifiedAnalytics #SparkAISummit
People. Process. Products.
DevOps is the union of
people, process, and
products to enable
continuous delivery of
value to your end users.
“
”
Build
&
Test
Continuous
Delivery
Deploy
Operate
Monitor
&
Learn
Plan
&
Track
Develop
Donovan Brown, MSFT PM

5. What is Azure Databricks?
5#UnifiedAnalytics #SparkAISummit
Increase productivity
Build on a secure, trusted cloud
Scale without limits
Built with your needs in mind
Enterprise grade Azure security
Native integration with Azure services
Live collaboration
Enterprise-grade SLAs
E2E data pipelines using ADF
Integrated billing

6. Azure DevOps
6#UnifiedAnalytics #SparkAISummit

7. 7#UnifiedAnalytics #SparkAISummit
Azure DevOps: Choose what you like
Any Language, Any Platform

8. 8#UnifiedAnalytics #SparkAISummit
Azure
Databricks
Dev WS
Push
notebooks
to Azure
Devops
Azure
DevOps
Repo
Build Pipeline
Artifact
Release
Pipeline
Deploy
Notebook
to staging
Azure
Databricks
Staging WS
Execute
Tests
Deploy
Notebook
to prod
Azure
Databricks
Prod WS
Implementation in Azure Databricks Notebooks

9. 9#UnifiedAnalytics #SparkAISummit
Azure
Databricks
Dev WS
Azure
DevOps
Repo
Build Pipeline
Artifact
Release
Pipeline
Run with
staging
cluster
Azure
Databricks
Staging WS
Execute
Tests
Run with
Prod
cluster
Azure
Databricks
Prod WS
Implementation in IDE (PyCharm, IntelliJ)
DB Connect

10. Azure Databricks REST API/CLI
• Provides an easy-to-use interface to the Azure
Databricks platform. CLI (open source project) is
built on top of the REST APIs
– Workspace API
• Deploy notebooks from Azure DevOps to Azure Databricks
– DBFS API
• Deploy libraries from Azure DevOps to Azure Databricks
– Jobs API
• Execute notebooks and Spark code once deployed
10#UnifiedAnalytics #SparkAISummit

11. Demo
#UnifiedAnalytics #SparkAISummit

12. DevOps for ML: Goals
• Repeatability of model creation & behavior
• Evaluation of model predictions
• Managing different model versions and files
• Operationalization of the model
• Monitoring of training and scoring pipelines
12#UnifiedAnalytics #SparkAISummit

13. 13#UnifiedAnalytics #SparkAISummit

14. Demo
#UnifiedAnalytics #SparkAISummit

15. Summary
• Two approaches
• Implementation in Azure Notebooks
• Implementation in IDE
• Azure DevOps to build CI/CD pipelines (you can selectively use)
• REST/CLI APIs
• Model CI/CD on Azure
• Azure Databricks: Data preparation and model training
• Azure ML: Model deployment and management
• Azure DevOps: CI/CD pipeline
15#UnifiedAnalytics #SparkAISummit

16. Call to action
• Build a CI/CD pipeline using Azure DevOps
• Azure Databricks documentation
• Azure DevOps pipelines
• Incorporate DevOps in your Azure Databricks
implementation
16#UnifiedAnalytics #SparkAISummit

17. THANK YOU!
“It is not the answer that enlightens, but the
question”
Eugene Ionesco
#UnifiedAnalytics #SparkAISummit

18. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT