Learn about enablers/features that can unblock and accelerate legacy SSIS migrations into ADF with no/minimal changes to existing packages and tools, e.g. Azure-enabled SSDT and SSMS, Package Deployment Model support, SSIS Integration Runtime (IR) package store, dtutil command prompt utility, Azure SQL Managed Instance (MI) Agent, SSIS scheduling feature and SSIS Job Migration Wizard on SSMS, Azure-enabled DTExec (AzureDTExec) command prompt utility, Virtual Network (VNet) injection of SSIS IR, Self-Hosted IR (SHIR) as a proxy for SSIS IR via ConnectByProxy property, Windows authentication feature, Azure Key Vault (AKV) integration, Azure Active Directory (AAD) authentication with ADF managed identity via ConnectUsingManagedIdentity property and OLEDB driver for SQL Server (MSOLEDBSQL), Azure Monitor integration, etc.
9. SSIS + ADF
Analogy
• Running traditional ETL workloads on premises is like being
on the ground floor
• Running modern ELT workloads in the cloud on a fully
managed Platform as a Service (PaaS) is like being on the 7th
floor, so jumping 7 stories straight up is virtually impossible
10. SSIS + ADF = SSIS in ADF
Analogy
• Running traditional ETL workloads on premises is like being
on the ground floor
• Running modern ELT workloads in the cloud on a fully
managed Platform as a Service (PaaS) is like being on the 7th
floor, so jumping 7 stories straight up is virtually impossible
• Running SSIS in ADF is like taking an escalator that
seamlessly carries you to the 7th floor step-by-step with the
option to get on/off anytime at any floor.
20. Essential On-Premises SSIS Features to Preserve
Use Project/Package Deployment Model – Store packages in
SSISDB/MSDB/file system and run them on SQL Server
Use SSDT/SSMS tools and dtutil/dtexec command prompt utilities to
design/debug/deploy/configure/execute packages
Connect to on-premises data stores using Windows authentication
SQL Server Agent 3rd party tools to
Enterprises/ISVs can build their custom/licensed components on SSIS
Bundled and licensed
On-premises data stores
Cloud
On-premises
SQL Server/
File System
MSDB
SSISDB
21. Project/Package Deployment Models
Azure Files
SSIS IR
Azure-enabled SSDT/SSMS tools and dtutil/dtexec
DF UI
assess
VNet
proxy
Windows authentication
AKV AAD
Azure SQL MI Agent, 3rd party tools, or ADF pipelines, activities,
and triggers
custom setups 3rd party licensing
Pay per hour BYOL
AHB including the 4:1 SE-to-EE/GP-to-BC
exchange ratio, on both maximize
savings
Brown Field Features to Unblock/Accelerate Migration
On-premises data stores
Azure SQL DB/Managed
Instance (MI)/Azure Files
VNet/
Proxy
Azure Data Factory (ADF)
Cloud data stores
Cloud
On-premises
SQL Server/
File System
MSDB
SSISDB
MSDB
SSISDB
Brown Field
22. first-class
embed them directly
SSIS Execute Process/SQL Tasks
multi-cloud destinations
Green Field Features to Complement ADF
On-premises data stores
Azure SQL DB/Managed
Instance (MI)/Azure Files
VNet/
Proxy
Azure Data Factory (ADF)
Cloud data stores
Cloud
On-premises
SQL Server/
File System
MSDB
SSISDB
MSDB
SSISDB
Green Field
23. SSISDB
SSIS IR package store
VNet
injection w/ BYOIP/VNet NAT (Optional)
AAD ADF
managed identity
Azure SQL
MI Agent
dual standby SSIS IR pair
BCDR
T-SQL
BYOL to get AHB, including the 4:1 SE-to-EE/GP-to-BC exchange ratio,
on both maximize savings
Deep Integration Features for Azure SQL and SSIS in ADF
On-premises data stores
Azure SQL DB/Managed
Instance (MI)/Azure Files
VNet/
Proxy
Azure Data Factory (ADF)
Cloud data stores
Cloud
On-premises
SQL Server/
File System
MSDB
SSISDB
MSDB
SSISDB
26. Enterprise-Friendly Features for SSIS in ADF
• At any time, only one can access the primary SSISDB to fetch and execute packages, as well as write package
execution logs (primary role), while the other can only do the same for packages deployed somewhere else, for
example in Azure Files (secondary role)
Dual Standby SSIS IR Pair for Business Continuity & Disaster Recovery (BCDR)
41. SSIS IR node
Container
ISV Setup
1. Specify Product Key in setup script ISV Activation Server
2. Get Activation Key by submitting Cluster ID + Product Key
Local Store
(e.g. Registry)
3. Write Activation Key
SSIS Executor
ISV Extension
4. Read Activation Key and
validate it with Cluster ID
Setup
Runtime
4. Get Cluster ID
4. Report on Node Count (Optional)
SSIS Runtime
2. Activation Key
3rd Party Extensibility – Implementation
46. Batch Assess Batch Migrate
Edit
Test
Assess
DBA
BI Developer
SSDT
Database Migration
Assistant (DMA)/SSDT
Azure Files/file system
Database Migration
Service
(DMS)/Deployment
Wizard/dtutil
utility/manual copy
SSISDB hosted by Azure
SQL DB/MI, MSDB
hosted by Azure SQL MI
47. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
48. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
File System • Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• SQL Agent to ADF via SSMS
• Recreate in ADF via SSMS
• Recreate in ADF
49. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
File System • Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• SQL Agent to ADF via SSMS
• Recreate in ADF via SSMS
• Recreate in ADF
MSDB • Same as above • MSDB to MSDB via DMS (TBD)
• Redeploy to MSDB via dtutil
• Reconfigure w/ SSIS IR
package store
• Same as above • Export to Azure Files/file
system via SSMS/dtutil
• Recreate in ADF via SSMS
• Recreate in ADF
50. Source: SQL Server + SQL Agent Destination: Azure SQL MI + MI Agent Destination: Azure SQL DB + ADF
Storage Types Package Assessment Package Migration Job Migration Package Migration Job Migration
SSISDB • DMA
• SSDT
• SSISDB to SSISDB via DMS • SQL Agent to MI Agent via
DMS (TBD)
• SQL Agent to MI Agent via
PowerShell/T-SQL/C# script
• Recreate in MI Agent via
SSMS
• SSISDB to SSISDB via
DMS/SSMS (TBD)
• Redeploy to SSISDB via
SSDT/SSMS
• SQL Agent to ADF via SSMS
(TBD)
• Recreate in ADF via SSMS
• Recreate in ADF
File System • Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• SQL Agent to ADF via SSMS
• Recreate in ADF via SSMS
• Recreate in ADF
MSDB • Same as above • MSDB to MSDB via DMS (TBD)
• Redeploy to MSDB via dtutil
• Reconfigure w/ SSIS IR
package store
• Same as above • Export to Azure Files/file
system via SSMS/dtutil
• Recreate in ADF via SSMS
• Recreate in ADF
SSIS Package
Store
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Redeploy to MSDB via dtutil
• Reconfigure w/ SSIS IR
package store
• Same as above • Redeploy to Azure Files via
dtutil/manual copy
• Keep in file system to access
via VNet/Self-Hosted IR
• Export to Azure Files/file
system via SSMS/dtutil
• Same as above
60. Why Should You Care? Better TCO & ROI!
Source: The Business Value of Microsoft Azure SQL Database Services, IDC, Mar 2015
The Total Economic Impact of Microsoft Azure SQL Database Managed Instance, Forrester Consulting, Sep 2018
61. Why Should You Care? BYOL to Get AHB!
Maximize the value of on-premises SQL Server license investments by migrating
SSIS in ADF/Azure-SSIS Integration Runtime (IR) Editions
SQL Server with Software Assurance
62. Why Should You Care? Huge Opportunity!
Source: Worldwide Database Server Forecast, IDC, Dec 2016
Worldwide Database Server Forecast
Modernizing
opportunity
with SQL
Server
37% of all units
run Microsoft,
>50% of them
run 2008/R2
or older versions
Non-Relational
Open Source
PaaS & IaaS
Microsoft
Oracle
IBM
SAP
New SQL
Other
MySQL
No
SQL
63. Why Should You Care? Do The Math!
Source: Feb 2019 snapshot of active SSIS instances extrapolated from SSIS 2016/2017 telemetry & CSS case volume for SQL 2008-2017
SSIS 2008-2017 on
premises/Azure VMs/AWS VMs
(100%)
SSIS 2008-2017 on premises
(88%)
SSIS 2008/R2 EOS w/o SSISDB
(13%)
SSIS 2012-2017 w/ SSISDB
(21%)
Total Addressable Market
Present Lift & Shift
Target
Immediate
Target
Past Target
65. How Should You Care? Fear is a Great Motivator!
Fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
66. Fear of losing investments
Too many SSIS packages to convert from using OLEDB connection via Win Auth to using ADO.NET connection via AAD Auth w/ ADF
managed identity
Too many SSIS packages to convert from Package Deployment Model w/o SSISDB to Project Deployment Model w/ SSISDB
Too many SSIS jobs scheduled by 3rd party orchestrators (e.g. Active Batch, Control-M, etc.) invoking dtexec
Too many packages using absolute paths w/ hard-coded drives (e.g. C:)
SQL MI Agent doesn’t support SSIS environment/32-bit runtime/property override
SQL MI Agent can’t invoke dtexec to execute SSIS packages stored in MSDB/file system/SSIS package stores
Network configuration/policy is too complex/strict to access on-premises data via VNet
Custom setups w/ script are too complex/error-prone
How Should You Care? Fear is a Great Motivator!
Fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
67. How Should You Care? Fear is a Great Motivator!
Fear of losing investments
Too many SSIS packages to convert from using OLEDB connection via Win Auth to using ADO.NET connection via AAD Auth w/ ADF
managed identity
– Support OLEDB connection via AAD Auth w/ ADF managed identity, switch at run-time via property overrides
Too many SSIS packages to convert from Package Deployment Model w/o SSISDB to Project Deployment Model w/ SSISDB
– Support file system/Azure Files/MSDB hosted by SQL MI to store packages, turn SSISDB from a blocker into an upsell
Too many SSIS jobs scheduled by 3rd party orchestrators (e.g. Active Batch, Control-M, etc.) invoking dtexec
– Azure-enable dtexec to invoke Execute SSIS Package activities in ADF pipelines
Too many packages using absolute paths w/ hard-coded drives (e.g. C:)
– Replace absolute paths w/ UNC paths at run-time via property overrides
SQL MI Agent doesn’t support SSIS environment/32-bit runtime/property override
– Achieve feature parity w/ SQL Server Agent
SQL MI Agent can’t invoke dtexec to execute SSIS packages stored in MSDB/file system/SSIS package stores
– ADF-enable SQL MI Agent to invoke executions of packages stored in file system/Azure Files/SSIS IR package stores
Network configuration/policy is too complex/strict to access on-premises data via VNet
– Support Self-Hosted IR as proxy for Azure-SSIS IR to access on-premises data
Custom setups w/ script are too complex/error-prone
– Support custom setups w/o script
Fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
68. How Should You Care? Fear is a Great Motivator!
Minimizing fear of losing investments
– No more lift & shift blockers
Rebalancing your risks/fears
Maximizing fear of missing out
Separation of storage/DB engine from compute/ETL engine
High Availability (HA) via multi-node cluster
Disaster Recovery (DR) via standby SSIS IR in a paired region
Dynamic scalability (in/out/up/down)
ADO.NET/OLEDB/Storage connection via AAD Auth w/ ADF managed identity
AKV support on Execute SSIS Package activity
SSIS ADF connectors (80+)
PQ Source
DQS migration to cloud alternatives running on SSIS in ADF
More cloud-first/cloud-only features, etc.
71. Provisioning Azure-SSIS IR Using UI: https://docs.microsoft.com/azure/data-factory/tutorial-deploy-ssis-packages-azure
Using PSH: https://docs.microsoft.com/azure/data-factory/tutorial-deploy-ssis-packages-azure-powershell
Using ARM template: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime#azure-
resource-manager-template
Provisioning Azure-SSIS IR using Azure SQL
DB/Managed Instance w/ VNet service
endpoint/private endpoint to host SSISDB
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime
Joining a VNet: https://docs.microsoft.com/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
Provisioning Azure-SSIS IR using AAD Auth w/
ADF managed identity
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime
Enabling AAD Auth: https://docs.microsoft.com/azure/data-factory/enable-aad-authentication-azure-ssis-ir
Provisioning Azure-SSIS IR to access data on
premises
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/create-azure-ssis-integration-runtime
Joining a VNet: https://docs.microsoft.com/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
Using Self-Hosted IR as a proxy: https://docs.microsoft.com/azure/data-factory/self-hosted-integration-runtime-proxy-
ssis
Monitoring Azure-SSIS IR and SSIS package
executions
Using UI/PSH: https://docs.microsoft.com/azure/data-factory/monitor-integration-runtime#azure-ssis-integration-
runtime
Using Azure Monitor: https://docs.microsoft.com/azure/data-factory/monitor-using-azure-monitor#monitor-ssis-
operations-with-azure-monitor
Documentations
72. Configuring Azure-SSIS IR Using UI/PSH: https://docs.microsoft.com/azure/data-factory/manage-azure-ssis-integration-runtime
Configuring Azure-SSIS IR to use Enterprise Edition Using UI/PSH: https://docs.microsoft.com/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition
Configuring Azure-SSIS IR to use custom setups Using UI/PSH: https://docs.microsoft.com/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup
Configuring Azure-SSIS IR for high performance Using UI/PSH: https://docs.microsoft.com/azure/data-factory/configure-azure-ssis-integration-runtime-
performance
Configuring Azure-SSIS IR for disaster recovery Using UI/PSH: https://docs.microsoft.com/azure/data-factory/configure-bcdr-azure-ssis-integration-runtime
Deploying projects/packages to run on Azure-SSIS IR Connecting SSMS to SSISDB: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-connect-to-
catalog-database?view=sql-server-2017
Using UI/PSH: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-deploy-run-monitor-
tutorial?view=sql-server-2017
Using dtutil: https://docs.microsoft.com/azure/data-factory/azure-ssis-integration-runtime-package-
store#deploying-multiple-packages-with-dtutil
Documentations
73. Documentations
Connecting to data stores using Win Auth on Azure-
SSIS IR
Using 4 methods: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-connect-with-windows-
auth?view=sql-server-2017
Connecting to data stores using AAD Auth w/ ADF
managed identity on Azure-SSIS IR
Using Azure Storage connections: https://docs.microsoft.com/sql/integration-services/connection-
manager/azure-storage-connection-manager#managed-identities-for-azure-resources-authentication
Using ADO.NET connections: https://docs.microsoft.com/sql/integration-services/connection-manager/ado-net-
connection-manager#managed-identities-for-azure-resources-authentication
Using OLEDB connections: https://docs.microsoft.com/sql/integration-services/connection-manager/ole-db-
connection-manager#managed-identities-for-azure-resources-authentication
Connecting to file shares and processing files on Azure-
SSIS IR
Using local file system/file shares: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-files-
file-shares?view=sql-server-2017
Using Win Auth: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-connect-with-windows-
auth?view=sql-server-2017
Running packages on Azure-SSIS IR From SSDT: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-ssdt
Using SSMS/stored procedures/scripts/code: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-
azure-run-packages?view=sql-server-2017
Using AzureDTExec: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-azure-enabled-
dtexec
As Execute SSIS Package activities in ADF pipelines: https://docs.microsoft.com/azure/data-factory/how-to-
invoke-ssis-package-ssis-activity
As Stored Procedure activities in ADF pipelines: https://docs.microsoft.com/azure/data-factory/how-to-invoke-
ssis-package-stored-procedure-activity
Viewing data lineage w/ Azure Purview (Data Catalog) Connect ADF to Azure Purview: https://docs.microsoft.com/azure/data-factory/connect-data-factory-to-azure-
purview
Connect Azure Purview to ADF: https://docs.microsoft.com/azure/purview/how-to-link-azure-data-factory
74. Documentations
Scheduling package executions on Azure-SSIS IR Using SSMS/ADF/other methods: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-
schedule-packages?view=sql-server-2017
Using SSMS: https://docs.microsoft.com/sql/integration-services/lift-shift/ssis-azure-schedule-packages-
ssms?view=sql-server-2017
As Execute SSIS Package activities in ADF pipelines: https://docs.microsoft.com/azure/data-factory/how-to-
invoke-ssis-package-ssis-activity
Using Azure SQL MI Agent: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-
managed-instance-agent
Scheduling Azure-SSIS IR provisioning on demand/just
in time
Using Web activities: https://docs.microsoft.com/azure/data-factory/how-to-schedule-azure-ssis-integration-
runtime
Cleaning up package execution logs in SSISDB Using Elastic Jobs: https://docs.microsoft.com/azure/data-factory/how-to-clean-up-ssisdb-logs-with-elastic-jobs
Assessing SSIS packages for migration Using DMA: https://docs.microsoft.com/sql/dma/dma-assess-ssis?view=sql-server-ver15
Using SSDT: https://docs.microsoft.com/azure/data-factory/how-to-invoke-ssis-package-ssdt#assess-ssis-
projectpackages-for-executions-in-azure
Migrating SSIS packages SSIS Migration Playbook: https://docs.microsoft.com/azure/data-factory/scenario-ssis-migration-overview
Using DMS: https://docs.microsoft.com/azure/dms/how-to-migrate-ssis-packages-managed-instance
Practicing SSIS CI/CD Using Azure DevOps: https://docs.microsoft.com/sql/integration-services/devops/ssis-devops-overview?view=sql-
server-ver15
Developing premium/licensed components for Azure-
SSIS IR
Using Cluster ID: https://docs.microsoft.com/azure/data-factory/how-to-develop-azure-ssis-ir-licensed-
components
Procuring 3rd party components for Azure-SSIS IR Partner list: https://techcommunity.microsoft.com/t5/SQL-Server-Integration-Services/Enterprise-Edition-Custom-
Setup-and-3rd-Party-Extensibility-for/ba-p/388360
Continuous and increased product investments for different customer cohorts.
Continuous and increased product investments for different customer cohorts.
Continuous and increased product investments for different customer cohorts.
Typical SQL Server estate including primary Value-Added Services (VAS), excluding secondary ones (MDS/DQS).
Full compatibility rehost for DB + ETL engines.
Full compatibility rehost vs. partial rehost/complete replacement.
SSIS in ADF as glue between disparate resources on premises and in the cloud.
Pop quiz: What is AWS ETL service called?
ADF complements/fills gaps in SSIS and vice versa.
We have cloud alternatives of DQS built as extensions running on SSIS in ADF.
2017 is our break-out year.
Lift & shift motivation/migration drive from the economic/financial angle for enterprises.
Lift & shift motivation/migration drive from the economic/financial angle for enterprises.
Pop quiz: If you have on-premises SQL Server license for 100 cores and you’ve traded in 50 cores for Azure SQL DB/MI, how many cores can you use for SQL on IaaS/SSIS in ADF?
Lift & shift motivation/migration drive from the business opportunity angle for System Integrators (SIs) and DB/BI consultancies.
Lift & shift motivation/migration drive from the business opportunity angle for System Integrators (SIs) and DB/BI consultancies.
Lift & shift motivation/migration drive from the timing angle for all.
Pop quiz: What kind of insect is in my photo?
Lift & shift motivation/migration drive from the risk/fear angle (pros & cons) for enterprises.
Pros = fear of missing out.
Lift & shift motivation/migration drive from the risk/fear angle (pros & cons) for enterprises.
In the past, pros = fear of missing out < cons = fear of losing investments.
We’ve addressed/are addressing each fear of losing investments.
Now, pros = fear of missing out > cons = fear of losing investments.