Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)

80 views

Published on

Building Dynamic Data Pipelines in Azure Data Factory (Presented at Microsoft Ignite on November 4th, 2019)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)

  1. 1. @cathrinew | cathrinew.net
  2. 2. Session Abstract
  3. 3. @cathrinew cathrinew.net
  4. 4. You are an Azure Data Factory developer
  5. 5. This is getting pretty tedious…
  6. 6. There has to be a better way!
  7. 7. Dynamic Solutions
  8. 8. Why would you use dynamic solutions?
  9. 9. How dynamic should the solution be?
  10. 10. How dynamic should the solution be?
  11. 11. What can make a solution dynamic? Parameters and Variables: Pass input values and set or update values during runtime Expressions and Functions: Modify the content of values during runtime Loops and Lookups: Control logic and executions based on external configuration values
  12. 12. Parameters and Variables
  13. 13. What are Parameters?
  14. 14. How are Parameters passed? trigger pipeline dataset user activity linked service pipeline
  15. 15. Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  16. 16. Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  17. 17. Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  18. 18. What are Variables?
  19. 19. How are variables controlled? Set Variable Append Variable
  20. 20. Variables @variables('VariableName') @first(variables('VariableName')) @last(variables('VariableName'))
  21. 21. Variables @variables('VariableName') @first(variables('VariableName')) @last(variables('VariableName'))
  22. 22. Expressions and Functions
  23. 23. What are Expressions? the @ symbol "@toUpper(pipeline().parameters.FileName)" "COLORS.CSV"
  24. 24. What are Functions? String: Date: Collection: Logical: Conversion: Math: concat, substring, startswith, endswith … adddays, addhours, formatDateTime … co s, f s , s , g h … f, , o , qu s, ss, g … createArray, y, jso , xm , … add, sub, mul, v, m , m x, mo , …
  25. 25. Combining Strings
  26. 26. How to combine strings? @concat() @{…}
  27. 27. What is String Concatenation? @concat( ' TRUNCATE TABLE dbo. ', pipeline().parameters.TableName )
  28. 28. What is String Interpolation? TRUNCATE TABLE dbo. @{pipeline().parameters.TableName}
  29. 29. DEMO Let's add some parameters!
  30. 30. Loops
  31. 31. ForEach Loop Input @item()
  32. 32. Array Items
  33. 33. Array Items
  34. 34. Array Items colors @item()
  35. 35. Array Items o ch o ch o y o y o y o y o y o y @item() parts @item() colors @item() sets
  36. 36. Object Items
  37. 37. Object Items
  38. 38. Object Items { "Name": "colors", "Extension": "csv" } @item()
  39. 39. Object Items { "Name": "colors", "Extension": "csv" } @item().Name
  40. 40. Object Items o ch o ch o y o y o y o y o y o y @item().Name parts @item().Name colors @item().Name sets
  41. 41. DEMO Let's add a loop!
  42. 42. Lookups
  43. 43. Lookup
  44. 44. Lookup Output
  45. 45. Lookup Output
  46. 46. Lookup Output: First Row Only { "Name" : "colors", "Extension" : "csv" } @{activity('Lookup') .output.firstRow}
  47. 47. Lookup Output: First Row Only { "Name" : "colors", "Extension" : "csv" } @{activity('Lookup') .output.firstRow .Name}
  48. 48. Lookup Output: All Rows @{activity('Lookup') .output.value} [ { "Name" : "parts", "Extension" : "csv" }, { "Name" : "sets", "Extension" : "csv" } ]
  49. 49. Lookup Output First Row Only: Use to pass single values to another activity @{activity('Lookup').output.firstRow.ColumnName} All Rows: Use to pass entire collection to another activity @{activity('Lookup').output.value}
  50. 50. DEMO Let's add a lookup!
  51. 51. How do you make dynamic solutions? 1. 2. with parameters 3. solution in a loop 4. from configuration table
  52. 52. Good luck!
  53. 53. @cathrinew cathrinew.net hi@cathrinew.net thank you!

×