Building Dynamic Pipelines in Azure Data Factory (Presented at SQLSaturday Oslo on August 31st, 2019)

1. Building Dynamic Pipelines
in Azure Data Factory
Cathrine Wilhelmsen
SQLSatOslo · August 31st, 2019

2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Building Dynamic Data Pipelines in Azure Data Factory

3. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Our sponsorsPlatinum and event
Gold
Global
Silver
Bronze Raffle

4. @cathrinew
cathrinew.net

5. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
You are an
Azure Data Factory
developer

6. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)

7. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…this is getting
pretty tedious…

8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
There has to be
a better way!

9. Dynamic Data
Pipelines

10. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use dynamic solutions?

11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why not use dynamic solutions?

12. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?

13. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?

14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you make dynamic solutions?
1.
2.
3.
4.

15. Parameters

16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters

17. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters

18. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service

19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline

20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
data flow

21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
activity

22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline
activity
data flow

23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

24. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName

25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables

27. Expressions
and Functions

28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions
@

29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Functions

30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions with Functions
@

31. Combining Strings

32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
@concat
@{…}

33. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
String Concatenation
@concat('First Name: ',
pipeline().parameters.FirstName,
', Last Name: ',
pipeline().parameters.LastName)

34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
String Interpolation
First Name:
@{pipeline().parameters.FirstName},
Last Name:
@{pipeline().parameters.LastName}

35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
String Concatenation String Interpolation

36. DEMO
Let's parameterize
some pipelines!

37. ForEach Loops

38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops

39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops
@item()

40. Array Items

41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items

42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
colors
@item()

43. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
@item
parts
@item
colors
@item
sets

44. Object Items

45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items

46. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item()

47. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item().Name

48. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
@item().Name
parts
@item().Name
colors
@item().Name
sets

49. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Limitations when using Foreach Loops
Nested ForEach Loops are not supported
Maximum number of items: 100 000
Maximum number of parallel executions: 50
Default number of parallel executions: 20

50. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Best Practices for using ForEach Loops
Executing one activity?
Add directly in ForEach
Executing multiple activities?
Create new pipeline and execute from ForEach

51. DEMO
Let's create
some loops!

52. Lookups

53. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup

54. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output

55. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
{
"firstRow" :
{
"Column1" : "Value",
"Column2" : "Value"
}
}
{
"count" : "2",
"value" : [
{
"Column1" : "Value",
"Column2" : "Value"
},
{
"Column1" : "Value",
"Column2" : "Value"
}
]
}

56. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Column1" : "Value",
"Column2" : "Value"
}
}
firstRow

57. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Column1" : "Value",
"Column2" : "Value"
}
}
firstRow
.Column1

58. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Column1" : "Value",
"Column2" : "Value"
},
{
"Column1" : "Value",
"Column2" : "Value"
}
]
}
value

59. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
@{activity('Lookup').output.firstRow.Column1}
@{activity('Lookup').output.value}

60. DEMO
Let's add a lookup to
control our loop!

61. Controls and
Conditionals

62. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals

63. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Branching and Chaining
Success Failure Completion Skipped

64. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
If Condition

65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Variables

66. DEMO
Let's create some
if statements!

67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals

68. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Until

69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Get Metadata

70. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Get Metadata

71. DEMO
Let's check for
file existence!

72. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!

73. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!