Azure Data Factory (ADF) CICD experiences. Demo on how to set up CI and CD using ADF provisioned through Terraform and with deployment and git in Azure DevOps. Presented at Microsoft Data Platform User Group Oslo, Norway January 2020

1. From 0 to Hero
Azure Data Factory CI/CD experiences
#MDPUGOslo
Halvar Trøyel Nerbø

3. Session Abstract
Do you want to do CI/CD using Azure Data Factory? I have.
This session will cover how to do it, showcasing the dos and don'ts and
experience using Terraform and Azure DevOps to deploy and
successfully use Continuous Integration (CI) and Continuous Delivery
(CD) using Azure Data Factory.
We will set up a development, test and production Azure Data Factory
using variables and release pipelines in Azure DevOps. Fun times
guaranteed!

4. About Me
• Name: Halvar Trøyel Nerbø
• Occupation: Consultant
• Employer: RAV Norge AS
• Twitter & LinkedIn : @troyel

5. Problem – Copy data from source to blob
Blob Storage
Resource Group
Data Factory
Number of Air BNB reviews per day for
Vienna

6. Problem – Be able to develop while in prod
Blob Storage
Resource Group
Data Factory
Development
Testing
Blob Storage
Resource Group
Data Factory
Production
Blob Storage
Resource Group
Data Factory
Number of Air BNB reviews
per day for Vienna

7. Problem – We want lots of stuff
Blob Storage
Resource Group
Data Factory
Development
Databricks
Databases
Analysis
Services
Key Vault
Testing
Databricks
Databases
Analysis
Services
Key Vault
Blob Storage
Resource Group
Data Factory
Production
Databricks
Databases
Analysis
Services
Key Vault
Blob Storage
Resource Group
Data Factory

8. Solution

9. Meet the players!

10. Meet the players!
Azure Data Factory
- Pipelines to orchestrate data movement and
transformations
- Git integration (DevOps and Git Hub)
- Serverless, low-code, ++

11. Meet the players!
Terraform by Hashicorp
- Infrastructure as Code
- Manages resources in all major cloud providers
- Open Source

12. Meet the players!
Azure DevOps
- Pipelines to build and deploy applications,
configurations, databases ++
- Git Repos to manage version control and code
collaboration
- Project planning, testing ++

13. Meet the players!
“I will help you build and
manage your resources
in Azure!”
“I will help you test and deploy
safely and automatically from
development to production!”
“I will move data in
different environments”

14. Terraform / DevOps – optional tools
Principles:
• Infrastructure as Code
• Naming conventions
• Variables

15. # Create Azure Data Factory
resource "azurerm_data_factory" “wien" {
name = "wien-adf-${var.environment}"
location = azurerm_resource_group.wien.location
resource_group_name = azurerm_resource_group.wien.name
(…)
# Create a resource group
resource "azurerm_resource_group" "wien" {
name = "wien-rg-${var.environment}"
location = "westeurope"
tags = {
Environment = "wien-${var.environment}"
}
}

16. Give ADF access to KV
Create secret using BLOB connection string
“AZURE_BLOB_connectionString”

19. • Instructions:
• https://www.terraform.io/docs/cloud/vcs/azure-devops-services.html

20. DevOps
Minimize time from development to
production
- Scalable
- Scripted
- Automated
Agile Development + Agile Operations

21. CI / CD
Plan Code Build Test Release Deploy Operate
Continuous Integration Continuous Delivery
<>
<>
<>
<>
<>
<>
<>

22. Why CI / CD

23. CI / CD Azure Data Factory – way
• Everything in ADF is described in JSON
• ADF has built in version control integration
• ADF helps us do CI
• For Continous Delivery we need to set up pipelines for deploying
JSON

27. 2. PUBLISH in ADF4.Startrelease
5b. Deploy ARM
6. Run automated tests
7. Run step 5a,b,c,d in prod
5d. Start triggers

28. ARM template – generated at “Publish”

29. ARM template <- -> dataset.json
ADF_PUBLISH MASTER / FEATURE

30. ARM parameters – adf_publish branch
Parameters file :
THIS IS WHERE THE MAGIC
HAPPENS

31. Demo

32. Important stuff
• (ARM) Template
• (ARM) Template parameters
• (ARM) Override template parameters
• Variables:
• factoryName
• resourceGroup

33. Important points to note
• Git set up only necessary in Dev ADF
• Variable template and variable-override template
• ARM deployment are incremental, clean up

34. NB’s – learnings and experiences
• Deleting pipelines with dependencies
MS pre- and post deployment powershell script will fail when trying to remove
pipelines that have dependencies.
• Timing of deployments and triggers
When deployment happens there is a window where ADF is ‘off-line’ and
triggers are stopped. This needs to be considered when deploying.
• Publishing into adf_publish -> filter
ADF publishes to adf_publish branch for technical reasons, deploys should be
filtered.