Techorama 2019 session - Migration SSIS packages and projects to Azure Data Factory.

1. Migrating SSIS to the Cloud
Koen Verbeeck – Senior BI Consultant

2. Koen Verbeeck
contact information
koen dot verbeeck at ae dot be
@Ko_Ver
LinkedIn

3. agenda
what is the Azure-SSIS IR?
when migrate?
setting up the Azure-SSIS IR
doing a custom setup
creating an ETL/ELT flow in ADF
3

4. what is the Azure-SSIS IR?
or “the runtime” for short
4

5. 5
Cloud
On-premises
SSISDB
SSDT SSMS
author & deploy manage & schedule
ADF
provisioning &
management
SSISDB
Azure SQL DB
or Managed
Instance
Azure-
SSIS IR
(Azure
Compute)

6. Azure-SSIS IR
• one of three Integration Runtimes
• Azure IR – data movement & activity dispatch
• Self-hosted IR – data management gateway
• Azure-SSIS IR – lift & shift of SSIS projects
• “SSIS scale-out cluster on VMs in the cloud”
6

7. the SSIS catalog
Azure SQL DB
• easy to set-up
• cost effective (can be set to low tier)
• limited SQL Server Agent compatibility
• supported by latest versions of SSMS
7
Managed Instance
• “full-blown” SQL Server
• not the cheapest option
• but can be used for other
purposes as well
• full SQL Server Agent compatibility
• supported by latest versions of
SSMS

8. DEMO

9. when migrate?
to the cloud!
9

10. 10
when NOT to migrate

11. when NOT to migrate
11

12. alternatives
12

13. ADF v2 with mapping data flows (preview)
13

14. ADF v2 with wrangling data flows (preview)
14
Screenshot by Cathrine Wilhelmsen

15. when to migrate
• existing SSIS projects (lift & shift)
• no budget/time to rework existing code base
• desire to move to the cloud
• VMs are not an option
• good knowledge of SSIS, but little of the cloud
15

16. setting up the Azure-SSIS IR
16

17. Step 1
Choose location of the SSISDB
17

18. Step 2
Add some PowerShell and
create an ADF v2 instance
18

19. Step 3
Choose the Azure-SSIS node
size
19

20. Step 4
Configure connection to SQL
Server
20

21. Step 5
If necessary, set up AAD
authentication
21
CREATE USER [SSISIrGroup]
FROM EXTERNAL PROVIDER;
ALTER ROLE dbmanager ADD
MEMBER [SSISIrGroup];

22. Step 6
Test connection and set
number of parallel executions
22

23. Step 7
Go grab some coffee
23

24. Step 8
Deploy your SSIS project to
the SSIS catalog
24

25. Step 9
Schedule your SSIS packages
25

26. custom setup
26

27. Step 1
Create an Azure Blob
container
27

28. Step 2
Create a custom set-up script.
Use gacutil to load external dll
into the IR
28

29. Step 3
Get the Shared Access
Signature
29

30. Step 3
Configure the custom setup
SAS URI
30

31. DEMO

32. create an ETL/ELT flow in ADF
data pipelines for the data engineers
32

33. lift & shift?
• replace UNC paths by Azure Files
• use stored procs to kick off
parallel package execution
• either use a vnet to access on-premises data or
use ADF with self-hosted IR to pull data into
Azure
33

34. using Azure Files
34

35. typical flow
35
automatically start/stop IR
coffee time!do non-SSIS stuff
while IR is starting
run SSIS packages

36. typical flow since a couple of weeks
36
still coffee time!

37. DEMO

38. conclusion
• setting up an Azure-SSIS IR is not that complex
• maybe networking and custom setup
• most SSIS projects can be deployed without many changes
• the solution is however not that cost-effective
38

39. BONUS
39

40. tips
Configure an Azure SQL Server Integration Services Integration Runtime
Customized Setup for the Azure-SSIS Integration Runtime
Executing Integration Services Packages in the Azure-SSIS Integration Runtime
Parallel package execution in Azure-SSIS Runtime
Connect to On-premises Data in Azure Data Factory with the Self-hosted Integration Runtime - Part 1
Connect to On-premises Data in Azure Data Factory with the Self-hosted Integration Runtime - Part 2
Start an Azure Logic App Workflow on Demand or by an Event
Transfer Files from SharePoint To Blob Storage with Azure Logic Apps
Automate the Azure-SSIS Integration Runtime Start-up and Shutdown - Part 1
Automate the Azure-SSIS Integration Runtime Start-up and Shutdown - Part 2
40

41. QUESTIONS?