Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

768 views

Published on

Pipelines and Packages: Introduction to Azure Data Factory (Presented at 24 Hours of PASS on April 4th, 2019)

Published in: Data & Analytics
  • Be the first to comment

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

  1. 1. Cathrine Wilhelmsen, Senior BI Consultant, Inmeta Moderated By: Giuliana Grecco Pipelines and Packages: Introduction to Azure Data Factory
  2. 2. If you require assistance during the session, type your inquiry into the question pane on the right side. Maximize your screen with the zoom button on the top of the presentation window. Please fill in the short evaluation following the session. It will appear in your web browser. Technical Assistance
  3. 3. Thank you to our Presenting Sponsors
  4. 4. PASS’ flagship event November 5-8 Seattle, Washington Free 1-day local training events Local user groups around the world Online special interest user groups Business analytics training Get involved Explore everything PASS has to offer Free Online Resources Newsletters PASS.org
  5. 5. Cathrine Wilhelmsen Senior BI Consultant, Inmeta Work Things Senior BI Consultant and Microsoft Data Platform Tech Lead in Inmeta, Norway. Specialties: ADF, SSIS, Biml and T-SQL Geeky Things Fun Things Active in the PASS community as a speaker, blogger and chronic volunteer. Likes: coffee, chocolate and cat gifs :) /cathrinewilhelmsen @cathrinew cathrinew.net hi@cathrinew.net
  6. 6. Cathrine Wilhelmsen, Senior BI Consultant, Inmeta Moderated By: Giuliana Grecco Pipelines and Packages: Introduction to Azure Data Factory
  7. 7. Past Learnings and Future Visions
  8. 8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 10 years ago…
  9. 9. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS SQL Server Integration Services
  10. 10. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  11. 11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Then…
  12. 12. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v1 Azure Data Factory Version 1
  13. 13. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  14. 14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  15. 15. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Today…
  16. 16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ADF v2 Azure Data Factory Version 2
  17. 17. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  18. 18. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why? How? What?
  19. 19. Azure Data Factory
  20. 20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Azure Data Factory
  21. 21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Data Movement and Integration Azure Data Factory Pipelines Activities Datasets Data Flows Triggers Integration RuntimesLinked Services Templates
  22. 22. DEMO Azure Data Factory
  23. 23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Wait… I already have thousands of SSIS packages!
  24. 24. SSIS Lift and Shift
  25. 25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why Lift and Shift SSIS?
  26. 26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) 1. 2. 3. 4. How to Lift and Shift SSIS
  27. 27. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS Integration Runtime
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) https://sqlbits.com/Sessions/Event18/Deeper_Integration _and_New_Transformation_for_SSIS_in_ADF SSIS Integration Runtime Deep Dive
  29. 29. DEMO SSIS Lift and Shift
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) SSIS Lift and Shift: Lessons Learned like all VMs
  31. 31. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ≈ ≈ ≈ ≈ ≈ ≈ ADF vs SSIS
  32. 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
  33. 33. Mapping Data Flows
  34. 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Mapping Data Flows
  35. 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Mapping Data Flows Example
  36. 36. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Kamil Nowinski's Cheat Sheet github.com/SQLPlayer/CheatSheets/blob/master/ADFDF-Cheat-Sheet-sqlplayer.pdf
  37. 37. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in SSIS
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Schema Drift in ADF
  40. 40. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Oh no!
  41. 41. DEMO Mapping Data Flows
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lessons Learned
  43. 43. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck! Thank you :)
  44. 44. Questions?
  45. 45. Thank you for attending! @sqlpass #sqlpass @PASScommunity Learn more from Cathrine Wilhelmsen @cathrinew hi@cathrinew.net

×