Can we store our Connectionstrings or BlobStorageKeys or other Secretvalues somewhere else then in Azure Data Factory(ADF)? Yes you can! You can store these valuable secrets in Azure Key Vault(AKV).
But how can we achieve this in ADF? And finally how do we deploy our DataFactories in Azure Dev Ops to Test, Acceptance and Production environments with these Secrets ? Can this be setup dynamically?
During this session I will give answers on all of these questions. You will learn how to setup your Azure Key Vault, connect these secrets in ADF and finally deploy these secrets dynamically in Azure Dev Ops. As you can see a lot to talk about during this session.
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory how do these Azure Services work perfectly together!
1. Erwin de Kreuk
Microsoft Solution Architect
Azure Key Vault, Azure Dev Ops and AzureDataFactory
how do these Azure Services work perfectly together!
Date: October 8th
Room: Alcazar
Time: 16:45
4. WHAT IS AZURE DATA FACTORY?
Hybrid data integration service
With visual tools, you can build, debug, deploy, operationalize and
monitor your (big) data pipelines
Provides a way to transform data at scale without any coding required.
ELT Platform
5. “DevOps is the union of people, process,
and products to enable continuous
delivery of value to your end users”
Donovan Brown
8. WHAT IS AZURE KEYVAULT?
The safehouse to safeguard the cryptographic keys and secrets that are
used by your applications, servers and cloud applications
Fully integrated with Azure Active Directory
Secret management: Securely store and tightly control access to tokens, passwords,
certificates, API keys, and other secrets.
Key management: Create and control encryption keys that encrypt your data.
Certificate management: Provision, manage, and deploy public and private Secure Sockets
Layer/Transport Layer Security (SSL/TLS) certificates for use with Azure and your internal
connected resources.
9. ROLES IN AZURE KEYVAULT?
Deploys Application
Tells de application the URI of the key /
secret
Configures the application to use key /
secret (and may abuse), but never sees
the keys
AuditorDeveloperSecurity Operations
Manages Keys
Creates a Key Vault in Azure
Add keys / secrets to the Key Vault
Grants Permission to specific
applications to perform specific
operations using keys
Enables usage logs
Monitors access to keys
Reviews usage logs to confirm
proper key use and compliance
with data security standards
10. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
Create a Data Factory for every environment
Define naming conventions
customername-azu-environment-azureservice-department
$SubscriptionName = “XXXXXXXXXXXXXXXXX"
$ResourceGroupName = "DEMOEDK-EUW-BICC-DVLM-RG-ENVIRONMENT"
$DataFactoryNameName="demoedkazudvlmadfbicc01"
$Location="westeurope"
Connect-AzAccount
Select-AzSubscription -SubscriptionName $SubscriptionName
New-AzDataFactoryV2 -Name $DataFactoryNameName -
ResourceGroupName $ResourceGroupName -Location $Location
21. CONNECTING KEYVAULT IN ADF
Grant the Azure Data Factory access to your Azure KeyVault.
Be aware that you need to publish the Azure KeyVault before you start using secrets !
22. CREATING SECRETS IN AZURE KEY VAULT
DVLM-ABLB-ENV01
TEST-ABLB-ENV01
DVLM-ASQL-ENV01-WWI
TEST-ASQL-ENV01-WWI
The use of naming conventions ensures that we can
configure everything more dynamically in the release
process later on !
ENVIRONMENT-CONNECTION-REFERENCE
Environment
Development DVLM
Test TEST
Acceptance ACPT
Production PROD
Linked Service
Azure Blob Storage ABLB
Azure Cosmos DB SQL API ACSA
Azure Cosmos DB MongDB API ACMA
Azure Data Explorer ADEX
Azure Data Lake Storage Gen1 ADLS
Azure Data Lake Storage Gen2 ADLS
Azure Database for MariaDB AMDB
Azure Database for MySQL AMYS
Azure Database for PostgreSQL APOS
Azure File Storage AFIL
Azure Search ASER
Azure SQL Database ASQL
Azure SQL Database Managed Instance ASQM
Azure SQL Data Warehouse ASDW
Azure Table Storage ATBL
SQL Server MSQL
Reference
DatabaseConnection LogicalServer(Short)-DatabaseName
Storage
StorageName like
Audit/SSIS/Staging/DataLake
23.
24. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
27. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
28. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Test
Acceptance
Production
29. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
30. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment
31. CREATE RELEASE IN AZURE DEV OPS
Create a Release based on an Empty Job
Create a Stage for each Environment
Add an Artifact
Enable Continuous Deployment
42. CREATING YOUR AZURE DATA FACTORY
demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
43. CUSTOM PARAMETERS
Not all the properties have parameters by default.
arm-template-parameters-definition.json
Adding accountkey for DataLake with secrets from Azure Key Vault
“accountkey": “|",
Download file from:
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#default-parameterization-
template
= means keep the current value as the default value for the parameter.
- means don't keep the default value for the parameter.
| is a special case for secrets from Azure Key Vault for connection strings or
keys.
44. REMARKS
You cannot publish while SSIS Integration Runtime is running
Triggers needs to be disabled and enabled afterwards
Every time you change something to your connection you need to update the
Template parameter
Deletes are handled manually or through Custom PS script or Marketplace extension
You may run into the Azure Resource Manager template limits as your factory
grows bigger. => Linked Resource Manager templates usually have a master
template and a set of child templates linked to the master.
Resource:
https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#linked-resource-manager-
templates
48. What do you think?
10/8/2019Datamindstestpresentation
4
8
http://bit.ly/dataMindsConnectSession
1.Open the form
2.Provide constructive feedback
3.Be eligible for an amazing prize!
bit.ly is CASE SENSITIVE!