Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)

149 views

Published on

Building Dynamic Pipelines in Azure Data Factory (Presented at Data Saturday Holland on October 5th, 2019)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Building Dynamic Pipelines in Azure Data Factory (Data Saturday Holland)

  1. 1. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  2. 2. Dynamic Data Pipelines in Azure Data Factory Cathrine Wilhelmsen Data Saturday Holland · Oct 5, 2019
  3. 3. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Building Dynamic Data Pipelines in Azure Data Factory
  4. 4. @cathrinew cathrinew.net
  5. 5. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) You are an Azure Data Factory developer
  6. 6. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
  7. 7. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) …this is getting pretty tedious…
  8. 8. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) There has to be a better way!
  9. 9. Dynamic Data Pipelines
  10. 10. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why use dynamic solutions?
  11. 11. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Why not use dynamic solutions?
  12. 12. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How dynamic should the solution be?
  13. 13. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How dynamic should the solution be?
  14. 14. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) How do you make dynamic solutions? 1. 2. 3. 4.
  15. 15. Parameters
  16. 16. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters
  17. 17. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters
  18. 18. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service
  19. 19. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service pipeline
  20. 20. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service data flow
  21. 21. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service activity
  22. 22. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service pipeline activity data flow
  23. 23. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  24. 24. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  25. 25. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  26. 26. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  27. 27. Expressions and Functions
  28. 28. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Expressions @
  29. 29. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Functions
  30. 30. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Expressions with Functions @
  31. 31. Combining Strings
  32. 32. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Combining Strings @concat @{…}
  33. 33. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) String Concatenation @concat( ' TRUNCATE TABLE dbo. ', pipeline().parameters.TableName )
  34. 34. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) String Interpolation TRUNCATE TABLE dbo. @{pipeline().parameters.TableName}
  35. 35. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Combining Strings String Concatenation String Interpolation
  36. 36. DEMO Let's parameterize some pipelines!
  37. 37. ForEach Loops
  38. 38. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ForEach Loops
  39. 39. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) ForEach Loops @item()
  40. 40. Array Items
  41. 41. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items
  42. 42. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items colors @item()
  43. 43. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items @item() parts @item() colors @item() sets
  44. 44. Object Items
  45. 45. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items
  46. 46. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items { "Name": "colors", "Extension": "csv" } @item()
  47. 47. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items { "Name": "colors", "Extension": "csv" } @item().Name
  48. 48. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items @item().Name parts @item().Name colors @item().Name sets
  49. 49. DEMO Let's create some loops!
  50. 50. Lookups
  51. 51. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup
  52. 52. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output
  53. 53. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output { "firstRow" : { "Name" : "colors", "Extension" : "csv" } } { "count" : "2", "value" : [ { "Name" : "parts", "Extension" : "csv" }, { "Name" : "sets", "Extension" : "csv" } ] }
  54. 54. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: First Row Only { "firstRow" : { "Name" : "colors", "Extension" : "csv" } } firstRow
  55. 55. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: First Row Only { "firstRow" : { "Name" : "colors", "Extension" : "csv" } } firstRow .Name
  56. 56. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: All Rows { "count" : "2", "value" : [ { "Name" : "parts", "Extension" : "csv" }, { "Name" : "sets", "Extension" : "csv" } ] } value
  57. 57. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: All Rows { "count" : "2", "value" : [ { "Name" : "parts", "Extension" : "csv" }, { "Name" : "sets", "Extension" : "csv" } ] } value[1]
  58. 58. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: All Rows { "count" : "2", "value" : [ { "Name" : "parts", "Extension" : "csv" }, { "Name" : "sets", "Extension" : "csv" } ] } value[1] Name
  59. 59. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output @{activity('Lookup').output.firstRow.ColumnName} @{activity('Lookup').output.value}
  60. 60. DEMO Let's add a lookup to control our loop!
  61. 61. Controls and Conditionals
  62. 62. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Controls and Conditionals
  63. 63. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Branching and Chaining Success Failure Completion Skipped
  64. 64. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) If Condition
  65. 65. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Variables
  66. 66. DEMO Let's create some if statements!
  67. 67. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Controls and Conditionals
  68. 68. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Until
  69. 69. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Get Metadata
  70. 70. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Wait
  71. 71. DEMO Let's check for file existence!
  72. 72. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  73. 73. © 2019 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!

×