SlideShare a Scribd company logo
1 of 73
Download to read offline
Building Dynamic Pipelines
in Azure Data Factory
Cathrine Wilhelmsen
SQLSatOslo · August 31st, 2019
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Building Dynamic Data Pipelines in Azure Data Factory
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Our sponsorsPlatinum and event
Gold
Global
Silver
Bronze Raffle
@cathrinew
cathrinew.net
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
You are an
Azure Data Factory
developer
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
…this is getting
pretty tedious…
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
There has to be
a better way!
Dynamic Data
Pipelines
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use dynamic solutions?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Why not use dynamic solutions?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How dynamic should the solution be?
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you make dynamic solutions?
1.
2.
3.
4.
Parameters
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
data flow
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
activity
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Passing Parameters
trigger
pipeline dataset
user
activity linked service
pipeline
activity
data flow
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@dataset().
@linkedService().
ParameterName
ParameterName
ParameterName
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Parameters
@pipeline().parameters.
@pipeline().DataFactory
@pipeline().TriggerTime
ParameterName
and System Variables
Expressions
and Functions
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions
@
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Functions
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Expressions with Functions
@
Combining Strings
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
@concat
@{…}
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
String Concatenation
@concat('First Name: ',
pipeline().parameters.FirstName,
', Last Name: ',
pipeline().parameters.LastName)
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
String Interpolation
First Name:
@{pipeline().parameters.FirstName},
Last Name:
@{pipeline().parameters.LastName}
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Combining Strings
String Concatenation String Interpolation
DEMO
Let's parameterize
some pipelines!
ForEach Loops
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
ForEach Loops
@item()
Array Items
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
colors
@item()
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Array Items
@item
parts
@item
colors
@item
sets
Object Items
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item()
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
{ "Name": "colors", "Extension": "csv" }
@item().Name
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Object Items
@item().Name
parts
@item().Name
colors
@item().Name
sets
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Limitations when using Foreach Loops
Nested ForEach Loops are not supported
Maximum number of items: 100 000
Maximum number of parallel executions: 50
Default number of parallel executions: 20
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Best Practices for using ForEach Loops
Executing one activity?
Add directly in ForEach
Executing multiple activities?
Create new pipeline and execute from ForEach
DEMO
Let's create
some loops!
Lookups
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
{
"firstRow" :
{
"Column1" : "Value",
"Column2" : "Value"
}
}
{
"count" : "2",
"value" : [
{
"Column1" : "Value",
"Column2" : "Value"
},
{
"Column1" : "Value",
"Column2" : "Value"
}
]
}
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Column1" : "Value",
"Column2" : "Value"
}
}
firstRow
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: First Row Only
{
"firstRow" :
{
"Column1" : "Value",
"Column2" : "Value"
}
}
firstRow
.Column1
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output: All Rows
{
"count" : "2",
"value" : [
{
"Column1" : "Value",
"Column2" : "Value"
},
{
"Column1" : "Value",
"Column2" : "Value"
}
]
}
value
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup Output
@{activity('Lookup').output.firstRow.Column1}
@{activity('Lookup').output.value}
DEMO
Let's add a lookup to
control our loop!
Controls and
Conditionals
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Branching and Chaining
Success Failure Completion Skipped
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
If Condition
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Variables
DEMO
Let's create some
if statements!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Controls and Conditionals
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Until
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Get Metadata
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Get Metadata
DEMO
Let's check for
file existence!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
Good luck!
© 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!

More Related Content

What's hot

Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...Edureka!
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra
 
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesPutting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesDATAVERSITY
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshJeffrey T. Pollock
 
Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Mark Kromer
 
Azure Storage Services - Part 01
Azure Storage Services - Part 01Azure Storage Services - Part 01
Azure Storage Services - Part 01Neeraj Kumar
 
Azure Data Factory V2; The Data Flows
Azure Data Factory V2; The Data FlowsAzure Data Factory V2; The Data Flows
Azure Data Factory V2; The Data FlowsThomas Sykes
 
Introduction to Azure Data Factory
Introduction to Azure Data FactoryIntroduction to Azure Data Factory
Introduction to Azure Data FactorySlava Kokaev
 
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Apache Kafka and the Data Mesh | Michael Noll, ConfluentApache Kafka and the Data Mesh | Michael Noll, Confluent
Apache Kafka and the Data Mesh | Michael Noll, ConfluentHostedbyConfluent
 
Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)James Serra
 
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks
 
Azure data factory
Azure data factoryAzure data factory
Azure data factoryDavid Giard
 
Azure Data Factory
Azure Data FactoryAzure Data Factory
Azure Data FactoryHARIHARAN R
 
Snowflake for Data Engineering
Snowflake for Data EngineeringSnowflake for Data Engineering
Snowflake for Data EngineeringHarald Erb
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Cathrine Wilhelmsen
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)James Serra
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureDatabricks
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationDenodo
 

What's hot (20)

Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
Azure Data Factory | Moving On-Premise Data to Azure Cloud | Microsoft Azure ...
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data PipelinesPutting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
Putting the Ops in DataOps: Orchestrate the Flow of Data Across Data Pipelines
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
 
Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005Azure Data Factory Data Flows Training v005
Azure Data Factory Data Flows Training v005
 
Azure Storage Services - Part 01
Azure Storage Services - Part 01Azure Storage Services - Part 01
Azure Storage Services - Part 01
 
Azure Data Factory V2; The Data Flows
Azure Data Factory V2; The Data FlowsAzure Data Factory V2; The Data Flows
Azure Data Factory V2; The Data Flows
 
Introduction to Azure Data Factory
Introduction to Azure Data FactoryIntroduction to Azure Data Factory
Introduction to Azure Data Factory
 
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Apache Kafka and the Data Mesh | Michael Noll, ConfluentApache Kafka and the Data Mesh | Michael Noll, Confluent
Apache Kafka and the Data Mesh | Michael Noll, Confluent
 
Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)
 
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptx
 
Azure data factory
Azure data factoryAzure data factory
Azure data factory
 
Azure Data Factory
Azure Data FactoryAzure Data Factory
Azure Data Factory
 
Snowflake for Data Engineering
Snowflake for Data EngineeringSnowflake for Data Engineering
Snowflake for Data Engineering
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 

Similar to Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)Cathrine Wilhelmsen
 
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)Cathrine Wilhelmsen
 
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)Cathrine Wilhelmsen
 
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)Cathrine Wilhelmsen
 
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Cathrine Wilhelmsen
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Cathrine Wilhelmsen
 
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)Cathrine Wilhelmsen
 
Biml for Beginners: Script and Automate SSIS development (Hybrid VC)
Biml for Beginners: Script and Automate SSIS development (Hybrid VC)Biml for Beginners: Script and Automate SSIS development (Hybrid VC)
Biml for Beginners: Script and Automate SSIS development (Hybrid VC)Cathrine Wilhelmsen
 
Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)Cathrine Wilhelmsen
 
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)Cathrine Wilhelmsen
 
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)Cathrine Wilhelmsen
 
Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...
Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...
Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...Amazon Web Services
 
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...Amazon Web Services
 

Similar to Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo) (13)

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)
 
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
 
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)
 
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
Creating Visual Transformations in Azure Data Factory (dataMinds Connect)
 
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
Understanding Azure Data Factory: The What, When, and Why (NIC 2020)
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
 
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
Pipelines and Packages: Introduction to Azure Data Factory (24HOP)
 
Biml for Beginners: Script and Automate SSIS development (Hybrid VC)
Biml for Beginners: Script and Automate SSIS development (Hybrid VC)Biml for Beginners: Script and Automate SSIS development (Hybrid VC)
Biml for Beginners: Script and Automate SSIS development (Hybrid VC)
 
Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)Azure Data Factory for the SSIS Developer (SentryOne Webinar)
Azure Data Factory for the SSIS Developer (SentryOne Webinar)
 
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)
Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)
 
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
Level Up Your Biml: Best Practices and Coding Techniques (PASS Summit 2018)
 
Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...
Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...
Petabyte-Scale Migration to Amazon S3 Building Photobox's Data Lake (STG393) ...
 
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
The Theory and Math Behind Data Privacy and Security Assurance (SEC301) - AWS...
 

More from Cathrine Wilhelmsen

Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...Cathrine Wilhelmsen
 
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)Cathrine Wilhelmsen
 
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...Cathrine Wilhelmsen
 
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)Cathrine Wilhelmsen
 
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)Cathrine Wilhelmsen
 
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)Cathrine Wilhelmsen
 
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Cathrine Wilhelmsen
 
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Cathrine Wilhelmsen
 
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)Cathrine Wilhelmsen
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Cathrine Wilhelmsen
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Cathrine Wilhelmsen
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Cathrine Wilhelmsen
 
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ..."I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...Cathrine Wilhelmsen
 
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Cathrine Wilhelmsen
 
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)Cathrine Wilhelmsen
 
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Cathrine Wilhelmsen
 
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)Cathrine Wilhelmsen
 
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...Cathrine Wilhelmsen
 
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Cathrine Wilhelmsen
 

More from Cathrine Wilhelmsen (20)

Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
Getting Started: Data Factory in Microsoft Fabric (Microsoft Fabric Community...
 
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
Website Analytics in My Pocket using Microsoft Fabric (SQLBits 2024)
 
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
Data Integration using Data Factory in Microsoft Fabric (ESPC Microsoft Fabri...
 
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
Choosing between Fabric, Synapse and Databricks (Data Left Unattended 2023)
 
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
 
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)
 
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
 
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...
 
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
Website Analytics in my Pocket using Microsoft Fabric (AdaCon 2023)
 
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (D...
 
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...
 
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ..."I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...
 
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...
 
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)
 
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
Lessons Learned: Understanding Pipeline Pricing in Azure Data Factory and Azu...
 
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
Azure Synapse Analytics Teaser (Microsoft TechX Oslo 2019)
 
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
Lessons Learned: Understanding Azure Data Factory Pricing (Microsoft Ignite 2...
 
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
Pipelines and Packages: Introduction to Azure Data Factory (Techorama NL 2019)
 

Recently uploaded

办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degreeyuu sss
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改yuu sss
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...Amil Baba Dawood bangali
 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024thyngster
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanMYRABACSAFRA2
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
 
GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]📊 Markus Baersch
 
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样vhwb25kk
 
Semantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxSemantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxMike Bennett
 
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort servicejennyeacort
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档208367051
 
Advanced Machine Learning for Business Professionals
Advanced Machine Learning for Business ProfessionalsAdvanced Machine Learning for Business Professionals
Advanced Machine Learning for Business ProfessionalsVICTOR MAESTRE RAMIREZ
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一F La
 
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDRafezzaman
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...Boston Institute of Analytics
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our WorldEduminds Learning
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Boston Institute of Analytics
 
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degreeyuu sss
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesTimothy Spann
 
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhThiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhYasamin16
 

Recently uploaded (20)

办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
 
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
专业一比一美国俄亥俄大学毕业证成绩单pdf电子版制作修改
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
 
Identifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population MeanIdentifying Appropriate Test Statistics Involving Population Mean
Identifying Appropriate Test Statistics Involving Population Mean
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
 
GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]GA4 Without Cookies [Measure Camp AMS]
GA4 Without Cookies [Measure Camp AMS]
 
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
1:1定制(UQ毕业证)昆士兰大学毕业证成绩单修改留信学历认证原版一模一样
 
Semantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxSemantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptx
 
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
 
Advanced Machine Learning for Business Professionals
Advanced Machine Learning for Business ProfessionalsAdvanced Machine Learning for Business Professionals
Advanced Machine Learning for Business Professionals
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
 
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTDINTERNSHIP ON PURBASHA COMPOSITE TEX LTD
INTERNSHIP ON PURBASHA COMPOSITE TEX LTD
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our World
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
 
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
毕业文凭制作#回国入职#diploma#degree澳洲中央昆士兰大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhThiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh
 

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

  • 1. Building Dynamic Pipelines in Azure Data Factory Cathrine Wilhelmsen SQLSatOslo · August 31st, 2019
  • 2. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Building Dynamic Data Pipelines in Azure Data Factory
  • 3. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Our sponsorsPlatinum and event Gold Global Silver Bronze Raffle
  • 5. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) You are an Azure Data Factory developer
  • 6. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net)
  • 7. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) …this is getting pretty tedious…
  • 8. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) There has to be a better way!
  • 10. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why use dynamic solutions?
  • 11. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Why not use dynamic solutions?
  • 12. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How dynamic should the solution be?
  • 13. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How dynamic should the solution be?
  • 14. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) How do you make dynamic solutions? 1. 2. 3. 4.
  • 16. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters
  • 17. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters
  • 18. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service
  • 19. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service pipeline
  • 20. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service data flow
  • 21. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service activity
  • 22. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Passing Parameters trigger pipeline dataset user activity linked service pipeline activity data flow
  • 23. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  • 24. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @dataset(). @linkedService(). ParameterName ParameterName ParameterName
  • 25. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  • 26. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Parameters @pipeline().parameters. @pipeline().DataFactory @pipeline().TriggerTime ParameterName and System Variables
  • 28. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Expressions @
  • 29. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Functions
  • 30. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Expressions with Functions @
  • 32. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Combining Strings @concat @{…}
  • 33. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) String Concatenation @concat('First Name: ', pipeline().parameters.FirstName, ', Last Name: ', pipeline().parameters.LastName)
  • 34. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) String Interpolation First Name: @{pipeline().parameters.FirstName}, Last Name: @{pipeline().parameters.LastName}
  • 35. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Combining Strings String Concatenation String Interpolation
  • 38. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ForEach Loops
  • 39. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) ForEach Loops @item()
  • 41. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items
  • 42. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items colors @item()
  • 43. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Array Items @item parts @item colors @item sets
  • 45. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items
  • 46. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items { "Name": "colors", "Extension": "csv" } @item()
  • 47. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items { "Name": "colors", "Extension": "csv" } @item().Name
  • 48. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Object Items @item().Name parts @item().Name colors @item().Name sets
  • 49. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Limitations when using Foreach Loops Nested ForEach Loops are not supported Maximum number of items: 100 000 Maximum number of parallel executions: 50 Default number of parallel executions: 20
  • 50. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Best Practices for using ForEach Loops Executing one activity? Add directly in ForEach Executing multiple activities? Create new pipeline and execute from ForEach
  • 53. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup
  • 54. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output
  • 55. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output { "firstRow" : { "Column1" : "Value", "Column2" : "Value" } } { "count" : "2", "value" : [ { "Column1" : "Value", "Column2" : "Value" }, { "Column1" : "Value", "Column2" : "Value" } ] }
  • 56. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: First Row Only { "firstRow" : { "Column1" : "Value", "Column2" : "Value" } } firstRow
  • 57. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: First Row Only { "firstRow" : { "Column1" : "Value", "Column2" : "Value" } } firstRow .Column1
  • 58. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output: All Rows { "count" : "2", "value" : [ { "Column1" : "Value", "Column2" : "Value" }, { "Column1" : "Value", "Column2" : "Value" } ] } value
  • 59. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Lookup Output @{activity('Lookup').output.firstRow.Column1} @{activity('Lookup').output.value}
  • 60. DEMO Let's add a lookup to control our loop!
  • 62. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Controls and Conditionals
  • 63. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Branching and Chaining Success Failure Completion Skipped
  • 64. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) If Condition
  • 65. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Variables
  • 67. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Controls and Conditionals
  • 68. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Until
  • 69. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Get Metadata
  • 70. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Get Metadata
  • 72. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) Good luck!
  • 73. © 2018 Cathrine Wilhelmsen (hi@cathrinew.net) @cathrinew cathrinew.net hi@cathrinew.net thank you!