Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

305 views

Published on

Pipelines and Packages: Introduction to Azure Data Factory (Presented at DATA:Scotland on September 13th, 2019)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)

  1. 1. Pipelines and Packages: Introduction to Azure Data Factory Cathrine Wilhelmsen DATA:Scotland · September 13th, 2019
  2. 2. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  3. 3. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Pipelines and Packages: Introduction to Azure Data Factory
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  6. 6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
  7. 7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Collect Store Transform Integrate Prepare
  8. 8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  9. 9. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) 10 years ago…
  10. 10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS SQL Server Integration Services
  11. 11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  12. 12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Then…
  13. 13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v1 Azure Data Factory Version 1
  14. 14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  15. 15. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  16. 16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Today…
  17. 17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v2 Azure Data Factory Version 2
  18. 18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  19. 19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why? How? What?
  20. 20. Azure Data Factory
  21. 21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is Azure Data Factory?
  22. 22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What can you do in Azure Data Factory? Copy Data Transform Data
  23. 23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Data Flows Templates Triggers
  24. 24. DEMO Let's look inside Azure Data Factory!
  25. 25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Wait… I already have thousands of SSIS packages!
  26. 26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) And… You told me to use this Biml thing!
  27. 27. SSIS Lift and Shift
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What does Lift and Shift mean?
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) * Why should you Lift and Shift SSIS?
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
  32. 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Azure-SSIS Integration Runtime like all VMs
  33. 33. DEMO Let’s lift and shift some SSIS Packages!
  34. 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ≈ ≈ ≈ ≈ ≈ ≈ ADF vs SSIS
  35. 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
  36. 36. Mapping Data Flows
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What are Mapping Data Flows?
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do Mapping Data Flows work?
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why use Mapping Data Flows?
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) What is Schema Drift?
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in SSIS
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in ADF
  43. 43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Oh no!
  44. 44. DEMO Let's transform some data!
  45. 45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lessons Learned
  46. 46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  47. 47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!
  48. 48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

×