Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Pipelines and Packages:
Introduction to Azure Data Factory
Cathrine Wilhelmsen
DATA:Scotland · September 13th, 2019
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipelines and Packages: Introduction to Azure Data Factory
@cathrinew
cathrinew.net
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and ...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and ...
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Collect
Store
Transform
Integrate
Prepare
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
10 years ago…
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
SSIS
SQL Server Integration Services
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Then…
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ADF v1
Azure Data Factory Version 1
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Today…
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
ADF v2
Azure Data Factory Version 2
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why?
How?
What?
Azure
Data Factory
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Data Factory?
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is inside Azure Data Factory?
Pipelines
Activities
Datasets
Linked
Serv...
DEMO
Let's look inside
Azure Data Factory!
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Wait…
I already have
thousands of SSIS
packages!
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
And…
You told me to use
this Biml thing!
SSIS
Lift and Shift
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What does Lift and Shift mean?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
*
Why should you Lift and Shift SSIS?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
1. Configure Azure-SSIS Integration Runtime
2. Deploy SSIS Packages to SSISD...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
1. Configure Azure-SSIS Integration Runtime
2. Deploy SSIS Packages to SSISD...
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Azure-SSIS Integration Runtime
like all VMs
DEMO
Let’s lift and shift
some SSIS Packages!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
≈
≈
≈
≈
≈
≈
ADF vs SSIS
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Pipeline
Linked Service
Source
Sink
Activity
Data Flow
≈
≈
≈
≈
≈
≈
Package
C...
Mapping
Data Flows
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Mapping Data Flows?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How do Mapping Data Flows work?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use Mapping Data Flows?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Schema Drift?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Schema Drift in SSIS
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Schema Drift in ADF
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Oh no!
DEMO
Let's transform
some data!
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lessons Learned
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!
© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Upcoming SlideShare
Loading in …5
×

of

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 1 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 2 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 3 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 4 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 5 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 6 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 7 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 8 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 9 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 10 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 11 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 12 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 13 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 14 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 15 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 16 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 17 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 18 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 19 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 20 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 21 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 22 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 23 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 24 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 25 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 26 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 27 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 28 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 29 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 30 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 31 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 32 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 33 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 34 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 35 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 36 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 37 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 38 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 39 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 40 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 41 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 42 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 43 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 44 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 45 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 46 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 47 Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019) Slide 48
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

4 Likes

Share

Download to read offline

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

Download to read offline

Pipelines and Packages: Introduction to Azure Data Factory (Presented at DATA:Scotland on September 13th, 2019)

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

  1. 1. Pipelines and Packages: Introduction to Azure Data Factory Cathrine Wilhelmsen DATA:Scotland · September 13th, 2019
  2. 2. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  3. 3. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Pipelines and Packages: Introduction to Azure Data Factory
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  6. 6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  7. 7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Collect Store Transform Integrate Prepare
  8. 8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  9. 9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) 10 years ago…
  10. 10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS SQL Server Integration Services
  11. 11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  12. 12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Then…
  13. 13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v1 Azure Data Factory Version 1
  14. 14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  15. 15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  16. 16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Today…
  17. 17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v2 Azure Data Factory Version 2
  18. 18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  19. 19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why? How? What?
  20. 20. Azure Data Factory
  21. 21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Azure Data Factory?
  22. 22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  23. 23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Data Flows Templates Triggers
  24. 24. DEMO Let's look inside Azure Data Factory!
  25. 25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Wait… I already have thousands of SSIS packages!
  26. 26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) And… You told me to use this Biml thing!
  27. 27. SSIS Lift and Shift
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What does Lift and Shift mean?
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) * Why should you Lift and Shift SSIS?
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
  32. 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Azure-SSIS Integration Runtime like all VMs
  33. 33. DEMO Let’s lift and shift some SSIS Packages!
  34. 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ≈ ≈ ≈ ≈ ≈ ≈ ADF vs SSIS
  35. 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
  36. 36. Mapping Data Flows
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What are Mapping Data Flows?
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do Mapping Data Flows work?
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why use Mapping Data Flows?
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is Schema Drift?
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in SSIS
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in ADF
  43. 43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Oh no!
  44. 44. DEMO Let's transform some data!
  45. 45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lessons Learned
  46. 46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  47. 47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!
  48. 48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  • ignacio_18

    Mar. 2, 2021
  • AlpaBuddhabhatti

    Feb. 11, 2021
  • MohammedWaseemAS

    Nov. 10, 2020
  • CarolineLavecchia

    May. 1, 2020

Pipelines and Packages: Introduction to Azure Data Factory (Presented at DATA:Scotland on September 13th, 2019)

Views

Total views

3,220

On Slideshare

0

From embeds

0

Number of embeds

60

Actions

Downloads

270

Shares

0

Comments

0

Likes

4

×