Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

556 views

Published on

Building Dynamic Pipelines in Azure Data Factory (Presented at SQLSaturday Oslo on August 31st, 2019)

Published in: Data & Analytics
  • Be the first to comment

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

  1. 1. Building Dynamic Pipelines in Azure Data Factory Cathrine Wilhelmsen SQLSatOslo · August 31st, 2019
  2. 2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Building Dynamic Data Pipelines in Azure Data Factory
  3. 3. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Our sponsorsPlatinum and event Gold Global Silver Bronze Raffle
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) You are an Azure Data Factory developer
  6. 6. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  7. 7. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …this is getting pretty tedious…
  8. 8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) There has to be a better way!
  9. 9. Dynamic Data Pipelines
  10. 10. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why use dynamic solutions?
  11. 11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why not use dynamic solutions?
  12. 12. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How dynamic should the solution be?
  13. 13. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How dynamic should the solution be?
  14. 14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do you make dynamic solutions? 1. 2. 3. 4.
  15. 15. Parameters
  16. 16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters
  17. 17. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters
  18. 18. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service
  19. 19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service pipeline
  20. 20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service data flow
  21. 21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service activity
  22. 22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service pipeline activity data flow
  23. 23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  24. 24. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  25. 25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  26. 26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  27. 27. Expressions and Functions
  28. 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Expressions @
  29. 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Functions
  30. 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Expressions with Functions @
  31. 31. Combining Strings
  32. 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Combining Strings @concat @{…}
  33. 33. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) String Concatenation @concat('First Name: ', pipeline().parameters.FirstName, ', Last Name: ', pipeline().parameters.LastName)
  34. 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) String Interpolation First Name: @{pipeline().parameters.FirstName}, Last Name: @{pipeline().parameters.LastName}
  35. 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Combining Strings String Concatenation String Interpolation
  36. 36. DEMO Let's parameterize some pipelines!
  37. 37. ForEach Loops
  38. 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ForEach Loops
  39. 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ForEach Loops @item()
  40. 40. Array Items
  41. 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items
  42. 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items colors @item()
  43. 43. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items @item parts @item colors @item sets
  44. 44. Object Items
  45. 45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items
  46. 46. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items { "Name": "colors", "Extension": "csv" } @item()
  47. 47. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items { "Name": "colors", "Extension": "csv" } @item().Name
  48. 48. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items @item().Name parts @item().Name colors @item().Name sets
  49. 49. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Limitations when using Foreach Loops Nested ForEach Loops are not supported Maximum number of items: 100 000 Maximum number of parallel executions: 50 Default number of parallel executions: 20
  50. 50. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Best Practices for using ForEach Loops Executing one activity? Add directly in ForEach Executing multiple activities? Create new pipeline and execute from ForEach
  51. 51. DEMO Let's create some loops!
  52. 52. Lookups
  53. 53. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup
  54. 54. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output
  55. 55. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output { "firstRow" : { "Column1" : "Value", "Column2" : "Value" } } { "count" : "2", "value" : [ { "Column1" : "Value", "Column2" : "Value" }, { "Column1" : "Value", "Column2" : "Value" } ] }
  56. 56. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: First Row Only { "firstRow" : { "Column1" : "Value", "Column2" : "Value" } } firstRow
  57. 57. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: First Row Only { "firstRow" : { "Column1" : "Value", "Column2" : "Value" } } firstRow .Column1
  58. 58. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: All Rows { "count" : "2", "value" : [ { "Column1" : "Value", "Column2" : "Value" }, { "Column1" : "Value", "Column2" : "Value" } ] } value
  59. 59. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output @{activity('Lookup').output.firstRow.Column1} @{activity('Lookup').output.value}
  60. 60. DEMO Let's add a lookup to control our loop!
  61. 61. Controls and Conditionals
  62. 62. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Controls and Conditionals
  63. 63. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Branching and Chaining Success Failure Completion Skipped
  64. 64. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) If Condition
  65. 65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Variables
  66. 66. DEMO Let's create some if statements!
  67. 67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Controls and Conditionals
  68. 68. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Until
  69. 69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Get Metadata
  70. 70. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Get Metadata
  71. 71. DEMO Let's check for file existence!
  72. 72. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  73. 73. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!

×