SlideShare a Scribd company logo
Sponsored by
Is there a way that we can build our Azure Synapse
Pipelines all with parameters based on MetaData?
Erwin de Kreuk
Lead Data and AI @InSpark
13 May 10:00 - 11:00 (UTC)
13 May 12:00 - 13:00 (UTC+01:00) CET
All sessions available on www.cloudlunchlearn.com/marathon
<
Erwin de Kreuk
Lead Data and AI @InSpark
@erwindekreuk
https://www.linkedin.com/in/erwindekreuk/
https://erwindekreuk.com
We help organizations
accelerating their digital
transformation with impactful
Microsoft solutions & expertise
We Are InSpark
Azure Synapse Analytics
Scottish
Summit
Template
 Pipeline Parameter
 Dataset Parameter
 Notebook Parameter
 Linked Service Parameter
 Dataflow Parameter
Linked Services Parameters
 Connect to different Database
on same Server
 Connect to different Logical
Servers
Dataset Parameters
 Create 1 dataset for all your
Linked Services activities
FileSystem Directory FileName
Dataset Parameters
 Create 1 dataset for all your
Linked Services activities
FileSystem Directory FileName
Dataflow Parameters
 Pass parameters to Dataflow
Notebook Parameters
 Pass Parameters from
Synapse to Databricks
Pipeline Parameters
 Can be used across all your
Pipelines
• Notebooks
• Pipeline
• DataSet
• Linked Service
Parameters
• Dataflow
Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData?
DEMO
Innovate
to
accelerate
Metadata
Load your pipelines
dynamically
Can we get answers on the following
questions?
Can we build Synapse Pipelines
dynamically?
Can we extract data from my
sources based on MetaData?
Can we load the active(current) or
historical records to a DataStore?
Can we build history from
extracted data based on
MetaData?
Can we log the execution of the
Pipelines?
Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData?
Source Name
Source Schema
DataLake
Catalog
Table
Destination
Schema
IsIncremental LastLoadTime
Table
Destination
Name
IsActive
IsIncremental
Column
Metadata
Source Parameter table
Lookup
Get Source data
ForEach
For Each
Execute Pipeline
Load
Lookup
Get LastLoadDate
Copy
Copy Source to
ADLS
Stored Procedure
Set LastLoadDate
Command
Execute
SELECT [PipelineParameterId]
,[SourceName]
,[SourceSchema]
,[SelectQuery]
,[SelectLastLoaddate]
,[FilePath]
,[FileName]
,[TableDestinationName]
,[ProcessType]
,[IsActive]
,[IsIncremental]
,[IsIncrementalColumn]
,[LastLoadtime]
FROM [execution].[Pipeline_DataLake_Files]
SELECT case when 1=1 then
convert(varchar,max(LasteditedWhen),120) else
convert(varchar,getdate(),120) end as LastLoadDate
FROM SourceSchema.SourceTable
Metadata
Source Parameter table
 Log Start and End Time of
records
 Log Extracted Records
 Log Execution Failure
 Create Pipeline_ExecutionLog
table
[audit].[Event_Pipeline_OnBegin] [audit].[Event_Pipeline_OnEnd]
[audit].[Event_Pipeline_OnError]
PIPELINE ACTIVITY
Logging
Logging  Log Start and End Time of
records
 Log Extracted Records
 Log Execution Failure
 Create Pipeline_ExecutionLog
table
Pipeline_ExecutionLog
BEGIN
Insert new Record
Insert Metadata
Insert Start time
END
End Time
Status(1)
Row Counts
Pipeline Details
ERROR
End Time
Status(2)
Failure Message
Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData?
DEMO
Innovate
to
accelerate
Can we get answers on the following
questions?
Can we build Synapse Pipelines
dynamically?
Can we extract data from my
sources based on MetaData?
Can we load the active(current) or
historical records to a DataStore?
Can we build history from
extracted data based on
MetaData?
Can we log the execution of the
Pipelines?
Integration
runtime
on premises
datasources
Databricks
Azure
Synapse
Datapipelines
Azure SQL
Azure SQL
Database Data Lake
Intermediate Zone
Parquet
Azure
Synapse
Analytics
Azure
Synapse
Datapipelines
Data Lake
Raw Zone
Parquet
Data Store
Delta Lake
Data Lake
EXTRACT PREP LOAD
Auditing, Logging, Metadata and Execution
Power BI
HIGH OVERVIEW ARCHITECTURE
NITROGEN Data Integration
Process Flow
For each
Daily Run
Data Lake
Command
Delta Lake
Command
Data Store
Command
Execute
Data Lake
Execute
For each
Execute
Delta Lake
Execute
For each
Execute
Data Store
Execute
Auditing
Pipeline_DataLake
Pipeline_ExecutionLog
Pipeline_DeltaLake Pipeline_DataStore
Begin End Error
Auditing
Begin End Error
Auditing
Begib End Error
Command
Execute
Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData?
DEMO
Innovate
to
accelerate
Can we get answers on the following
questions?
Can we build Synapse Pipelines
dynamically?
Can we extract data from my
sources based on MetaData?
Can we load the active(current) or
historical records to a DataStore?
Can we build history from
extracted data based on
MetaData?
Can we log the execution of the
Pipelines?
@erwindekreuk
https://www.linkedin.com/in/erwindekreuk/
https://erwindekreuk.com
Slides will be available on my blog
Share your Love with the speaker and team on social, using the hashtag #cllmarathon and @CloudLunchLearn
And follow us 😊
youtube.com/c/CloudLunchandLearn @CloudLunchLearn
In collaboration with

More Related Content

What's hot

Plsql programs
Plsql programsPlsql programs
Ch08
Ch08Ch08
AstraZeneca - The promise of graphs & graph-based learning in drug discovery
AstraZeneca - The promise of graphs & graph-based learning in drug discoveryAstraZeneca - The promise of graphs & graph-based learning in drug discovery
AstraZeneca - The promise of graphs & graph-based learning in drug discovery
Neo4j
 
Bertelsmann: BeTrend – Building a Trend Aggregation Machine.pdf
Bertelsmann: BeTrend – Building a Trend Aggregation Machine.pdfBertelsmann: BeTrend – Building a Trend Aggregation Machine.pdf
Bertelsmann: BeTrend – Building a Trend Aggregation Machine.pdf
Neo4j
 
Graphs in Data Structure
 Graphs in Data Structure Graphs in Data Structure
Graphs in Data Structure
hafsa komal
 
Application Of Graph Data Structure
Application Of Graph Data StructureApplication Of Graph Data Structure
Application Of Graph Data Structure
Gaurang Dobariya
 
AstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & Management
AstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & ManagementAstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & Management
AstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & Management
Neo4j
 
Azure Synapse Analytics
Azure Synapse AnalyticsAzure Synapse Analytics
Azure Synapse Analytics
WinWire Technologies Inc
 
Operator Precedence and Associativity
Operator Precedence and AssociativityOperator Precedence and Associativity
Operator Precedence and Associativity
Nicole Ynne Estabillo
 

What's hot (9)

Plsql programs
Plsql programsPlsql programs
Plsql programs
 
Ch08
Ch08Ch08
Ch08
 
AstraZeneca - The promise of graphs & graph-based learning in drug discovery
AstraZeneca - The promise of graphs & graph-based learning in drug discoveryAstraZeneca - The promise of graphs & graph-based learning in drug discovery
AstraZeneca - The promise of graphs & graph-based learning in drug discovery
 
Bertelsmann: BeTrend – Building a Trend Aggregation Machine.pdf
Bertelsmann: BeTrend – Building a Trend Aggregation Machine.pdfBertelsmann: BeTrend – Building a Trend Aggregation Machine.pdf
Bertelsmann: BeTrend – Building a Trend Aggregation Machine.pdf
 
Graphs in Data Structure
 Graphs in Data Structure Graphs in Data Structure
Graphs in Data Structure
 
Application Of Graph Data Structure
Application Of Graph Data StructureApplication Of Graph Data Structure
Application Of Graph Data Structure
 
AstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & Management
AstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & ManagementAstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & Management
AstraZeneca - Re-imagining the Data Landscape in Compound Synthesis & Management
 
Azure Synapse Analytics
Azure Synapse AnalyticsAzure Synapse Analytics
Azure Synapse Analytics
 
Operator Precedence and Associativity
Operator Precedence and AssociativityOperator Precedence and Associativity
Operator Precedence and Associativity
 

Similar to Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData?

Is there a way that we can build our Azure Data Factory all with parameters b...
Is there a way that we can build our Azure Data Factory all with parameters b...Is there a way that we can build our Azure Data Factory all with parameters b...
Is there a way that we can build our Azure Data Factory all with parameters b...
Erwin de Kreuk
 
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
VMware Tanzu
 
SQL Server 2008 for Developers
SQL Server 2008 for DevelopersSQL Server 2008 for Developers
SQL Server 2008 for Developers
ukdpe
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
Guido Schmutz
 
Practical OData
Practical ODataPractical OData
Practical OData
Vagif Abilov
 
Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft Private Cloud
 
Visualizing Big Data in Realtime
Visualizing Big Data in RealtimeVisualizing Big Data in Realtime
Visualizing Big Data in Realtime
DataWorks Summit
 
Zeppelin at Twitter
Zeppelin at TwitterZeppelin at Twitter
Zeppelin at Twitter
Prasad Wagle
 
Microsoft SQL Server - SQL Server Migrations Presentation
Microsoft SQL Server - SQL Server Migrations PresentationMicrosoft SQL Server - SQL Server Migrations Presentation
Microsoft SQL Server - SQL Server Migrations Presentation
Microsoft Private Cloud
 
An Architect's guide to real time big data systems
An Architect's guide to real time big data systemsAn Architect's guide to real time big data systems
An Architect's guide to real time big data systems
Raja SP
 
SQL Server - High availability
SQL Server - High availabilitySQL Server - High availability
SQL Server - High availability
Peter Gfader
 
01 Architecture Of Integration Services
01 Architecture Of Integration Services01 Architecture Of Integration Services
01 Architecture Of Integration Services
Slava Kokaev
 
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data AnalyticsStrata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
SingleStore
 
SharePoint Intelligence Introduction To Share Point Designer Workflows
SharePoint Intelligence Introduction To Share Point Designer WorkflowsSharePoint Intelligence Introduction To Share Point Designer Workflows
SharePoint Intelligence Introduction To Share Point Designer Workflows
Ivan Sanders
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...
Databricks
 
Data Science on Google Cloud Platform
Data Science on Google Cloud PlatformData Science on Google Cloud Platform
Data Science on Google Cloud Platform
Virot "Ta" Chiraphadhanakul
 
MuleSoft London Community February 2020 - MuleSoft and OData
MuleSoft London Community February 2020 - MuleSoft and ODataMuleSoft London Community February 2020 - MuleSoft and OData
MuleSoft London Community February 2020 - MuleSoft and OData
Pace Integration
 
How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...
How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...
How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...
serge luca
 
Azure data analytics platform - A reference architecture
Azure data analytics platform - A reference architecture Azure data analytics platform - A reference architecture
Azure data analytics platform - A reference architecture
Rajesh Kumar
 
Presto
PrestoPresto
Presto
Knoldus Inc.
 

Similar to Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData? (20)

Is there a way that we can build our Azure Data Factory all with parameters b...
Is there a way that we can build our Azure Data Factory all with parameters b...Is there a way that we can build our Azure Data Factory all with parameters b...
Is there a way that we can build our Azure Data Factory all with parameters b...
 
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
 
SQL Server 2008 for Developers
SQL Server 2008 for DevelopersSQL Server 2008 for Developers
SQL Server 2008 for Developers
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
 
Practical OData
Practical ODataPractical OData
Practical OData
 
Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview Presentation
 
Visualizing Big Data in Realtime
Visualizing Big Data in RealtimeVisualizing Big Data in Realtime
Visualizing Big Data in Realtime
 
Zeppelin at Twitter
Zeppelin at TwitterZeppelin at Twitter
Zeppelin at Twitter
 
Microsoft SQL Server - SQL Server Migrations Presentation
Microsoft SQL Server - SQL Server Migrations PresentationMicrosoft SQL Server - SQL Server Migrations Presentation
Microsoft SQL Server - SQL Server Migrations Presentation
 
An Architect's guide to real time big data systems
An Architect's guide to real time big data systemsAn Architect's guide to real time big data systems
An Architect's guide to real time big data systems
 
SQL Server - High availability
SQL Server - High availabilitySQL Server - High availability
SQL Server - High availability
 
01 Architecture Of Integration Services
01 Architecture Of Integration Services01 Architecture Of Integration Services
01 Architecture Of Integration Services
 
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data AnalyticsStrata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
Strata+Hadoop 2015 NYC End User Panel on Real-Time Data Analytics
 
SharePoint Intelligence Introduction To Share Point Designer Workflows
SharePoint Intelligence Introduction To Share Point Designer WorkflowsSharePoint Intelligence Introduction To Share Point Designer Workflows
SharePoint Intelligence Introduction To Share Point Designer Workflows
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...
 
Data Science on Google Cloud Platform
Data Science on Google Cloud PlatformData Science on Google Cloud Platform
Data Science on Google Cloud Platform
 
MuleSoft London Community February 2020 - MuleSoft and OData
MuleSoft London Community February 2020 - MuleSoft and ODataMuleSoft London Community February 2020 - MuleSoft and OData
MuleSoft London Community February 2020 - MuleSoft and OData
 
How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...
How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...
How to choose between SharePoint lists, SQL Azure, Microsoft Dataverse with D...
 
Azure data analytics platform - A reference architecture
Azure data analytics platform - A reference architecture Azure data analytics platform - A reference architecture
Azure data analytics platform - A reference architecture
 
Presto
PrestoPresto
Presto
 

More from Erwin de Kreuk

Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...
Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...
Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...
Erwin de Kreuk
 
Lake Database Database Template Map Data in Azure Synapse Analytics
Lake Database  Database Template  Map Data in Azure Synapse AnalyticsLake Database  Database Template  Map Data in Azure Synapse Analytics
Lake Database Database Template Map Data in Azure Synapse Analytics
Erwin de Kreuk
 
Dealing with different Synapse Roles in Azure Synapse Analytics Erwin de Kreuk
Dealing with different Synapse Roles in Azure Synapse Analytics Erwin de KreukDealing with different Synapse Roles in Azure Synapse Analytics Erwin de Kreuk
Dealing with different Synapse Roles in Azure Synapse Analytics Erwin de Kreuk
Erwin de Kreuk
 
DataMinds 2022 Azure Purview Erwin de Kreuk
DataMinds 2022 Azure Purview Erwin de KreukDataMinds 2022 Azure Purview Erwin de Kreuk
DataMinds 2022 Azure Purview Erwin de Kreuk
Erwin de Kreuk
 
Data weekender4.2 azure purview erwin de kreuk
Data weekender4.2  azure purview erwin de kreukData weekender4.2  azure purview erwin de kreuk
Data weekender4.2 azure purview erwin de kreuk
Erwin de Kreuk
 
Data saturday Oslo Azure Purview Erwin de Kreuk
Data saturday Oslo Azure Purview Erwin de KreukData saturday Oslo Azure Purview Erwin de Kreuk
Data saturday Oslo Azure Purview Erwin de Kreuk
Erwin de Kreuk
 
Datasaturday Pordenone Azure Purview Erwin de Kreuk
Datasaturday Pordenone Azure Purview Erwin de KreukDatasaturday Pordenone Azure Purview Erwin de Kreuk
Datasaturday Pordenone Azure Purview Erwin de Kreuk
Erwin de Kreuk
 
Azure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de KreukAzure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de Kreuk
Erwin de Kreuk
 
SQL KONFERENZ 2020 Azure Key Vault, Azure Dev Ops and Azure Data Factory how...
SQL KONFERENZ 2020  Azure Key Vault, Azure Dev Ops and Azure Data Factory how...SQL KONFERENZ 2020  Azure Key Vault, Azure Dev Ops and Azure Data Factory how...
SQL KONFERENZ 2020 Azure Key Vault, Azure Dev Ops and Azure Data Factory how...
Erwin de Kreuk
 
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...
Erwin de Kreuk
 
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...Help, I need to migrate my On Premise Database to Azure, which Database Tier ...
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...
Erwin de Kreuk
 
DataSaturdayNL 2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory h...
DataSaturdayNL 2019  Azure Key Vault, Azure Dev Ops and Azure Data Factory  h...DataSaturdayNL 2019  Azure Key Vault, Azure Dev Ops and Azure Data Factory  h...
DataSaturdayNL 2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory h...
Erwin de Kreuk
 
TechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data Factor
TechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data FactorTechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data Factor
TechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data Factor
Erwin de Kreuk
 

More from Erwin de Kreuk (13)

Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...
Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...
Azure Key Vault, Azure Dev Ops and Azure Synapse - how these services work pe...
 
Lake Database Database Template Map Data in Azure Synapse Analytics
Lake Database  Database Template  Map Data in Azure Synapse AnalyticsLake Database  Database Template  Map Data in Azure Synapse Analytics
Lake Database Database Template Map Data in Azure Synapse Analytics
 
Dealing with different Synapse Roles in Azure Synapse Analytics Erwin de Kreuk
Dealing with different Synapse Roles in Azure Synapse Analytics Erwin de KreukDealing with different Synapse Roles in Azure Synapse Analytics Erwin de Kreuk
Dealing with different Synapse Roles in Azure Synapse Analytics Erwin de Kreuk
 
DataMinds 2022 Azure Purview Erwin de Kreuk
DataMinds 2022 Azure Purview Erwin de KreukDataMinds 2022 Azure Purview Erwin de Kreuk
DataMinds 2022 Azure Purview Erwin de Kreuk
 
Data weekender4.2 azure purview erwin de kreuk
Data weekender4.2  azure purview erwin de kreukData weekender4.2  azure purview erwin de kreuk
Data weekender4.2 azure purview erwin de kreuk
 
Data saturday Oslo Azure Purview Erwin de Kreuk
Data saturday Oslo Azure Purview Erwin de KreukData saturday Oslo Azure Purview Erwin de Kreuk
Data saturday Oslo Azure Purview Erwin de Kreuk
 
Datasaturday Pordenone Azure Purview Erwin de Kreuk
Datasaturday Pordenone Azure Purview Erwin de KreukDatasaturday Pordenone Azure Purview Erwin de Kreuk
Datasaturday Pordenone Azure Purview Erwin de Kreuk
 
Azure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de KreukAzure Purview Data Toboggan Erwin de Kreuk
Azure Purview Data Toboggan Erwin de Kreuk
 
SQL KONFERENZ 2020 Azure Key Vault, Azure Dev Ops and Azure Data Factory how...
SQL KONFERENZ 2020  Azure Key Vault, Azure Dev Ops and Azure Data Factory how...SQL KONFERENZ 2020  Azure Key Vault, Azure Dev Ops and Azure Data Factory how...
SQL KONFERENZ 2020 Azure Key Vault, Azure Dev Ops and Azure Data Factory how...
 
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...
DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory ho...
 
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...Help, I need to migrate my On Premise Database to Azure, which Database Tier ...
Help, I need to migrate my On Premise Database to Azure, which Database Tier ...
 
DataSaturdayNL 2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory h...
DataSaturdayNL 2019  Azure Key Vault, Azure Dev Ops and Azure Data Factory  h...DataSaturdayNL 2019  Azure Key Vault, Azure Dev Ops and Azure Data Factory  h...
DataSaturdayNL 2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory h...
 
TechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data Factor
TechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data FactorTechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data Factor
TechnoramaNL Azure Key Vault, Azure Dev Ops and Azure Data Factor
 

Recently uploaded

Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
LINAT
 
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
fatima shekh$A17
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
huseindihon
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
44annissa
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata
 
Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
doctorzlife786
 
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
6459astrid
 
Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)
Alireza Kamrani
 
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
revolutionary575
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
janvikumar4133
 
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
dizzycaye
 
transgenders community data in india by govt
transgenders community data in india by govttransgenders community data in india by govt
transgenders community data in india by govt
palanisamyiiiier
 
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured DataFine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
kevig
 
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
gargnatasha985
 
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
sheetal singh$A17
 
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdfWhy_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Alexander Teggin
 
ch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ssch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ss
MinThetLwin1
 
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
revolutionary575
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
birajmohan012
 
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
NABLAS株式会社
 

Recently uploaded (20)

Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
 
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
 
Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
 
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Premium Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)
 
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
Celebrity Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servi...
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
 
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
 
transgenders community data in india by govt
transgenders community data in india by govttransgenders community data in india by govt
transgenders community data in india by govt
 
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured DataFine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
 
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
 
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
Exclusive Girls Call Noida 🎈🔥9873940964 🔥💋🎈 Provide Best And Top Girl Service...
 
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdfWhy_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
 
ch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ssch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ss
 
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
 
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
社内勉強会資料_TransNeXt: Robust Foveal Visual Perception for Vision Transformers
 

Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData?

  • 1. Sponsored by Is there a way that we can build our Azure Synapse Pipelines all with parameters based on MetaData? Erwin de Kreuk Lead Data and AI @InSpark 13 May 10:00 - 11:00 (UTC) 13 May 12:00 - 13:00 (UTC+01:00) CET All sessions available on www.cloudlunchlearn.com/marathon <
  • 2. Erwin de Kreuk Lead Data and AI @InSpark @erwindekreuk https://www.linkedin.com/in/erwindekreuk/ https://erwindekreuk.com
  • 3. We help organizations accelerating their digital transformation with impactful Microsoft solutions & expertise We Are InSpark
  • 5. Scottish Summit Template  Pipeline Parameter  Dataset Parameter  Notebook Parameter  Linked Service Parameter  Dataflow Parameter
  • 6. Linked Services Parameters  Connect to different Database on same Server  Connect to different Logical Servers
  • 7. Dataset Parameters  Create 1 dataset for all your Linked Services activities FileSystem Directory FileName
  • 8. Dataset Parameters  Create 1 dataset for all your Linked Services activities FileSystem Directory FileName
  • 9. Dataflow Parameters  Pass parameters to Dataflow
  • 10. Notebook Parameters  Pass Parameters from Synapse to Databricks
  • 11. Pipeline Parameters  Can be used across all your Pipelines
  • 12. • Notebooks • Pipeline • DataSet • Linked Service Parameters • Dataflow
  • 14. DEMO
  • 17. Can we get answers on the following questions? Can we build Synapse Pipelines dynamically? Can we extract data from my sources based on MetaData? Can we load the active(current) or historical records to a DataStore? Can we build history from extracted data based on MetaData? Can we log the execution of the Pipelines?
  • 19. Source Name Source Schema DataLake Catalog Table Destination Schema IsIncremental LastLoadTime Table Destination Name IsActive IsIncremental Column Metadata Source Parameter table
  • 20. Lookup Get Source data ForEach For Each Execute Pipeline Load Lookup Get LastLoadDate Copy Copy Source to ADLS Stored Procedure Set LastLoadDate Command Execute SELECT [PipelineParameterId] ,[SourceName] ,[SourceSchema] ,[SelectQuery] ,[SelectLastLoaddate] ,[FilePath] ,[FileName] ,[TableDestinationName] ,[ProcessType] ,[IsActive] ,[IsIncremental] ,[IsIncrementalColumn] ,[LastLoadtime] FROM [execution].[Pipeline_DataLake_Files] SELECT case when 1=1 then convert(varchar,max(LasteditedWhen),120) else convert(varchar,getdate(),120) end as LastLoadDate FROM SourceSchema.SourceTable Metadata Source Parameter table
  • 21.  Log Start and End Time of records  Log Extracted Records  Log Execution Failure  Create Pipeline_ExecutionLog table [audit].[Event_Pipeline_OnBegin] [audit].[Event_Pipeline_OnEnd] [audit].[Event_Pipeline_OnError] PIPELINE ACTIVITY Logging
  • 22. Logging  Log Start and End Time of records  Log Extracted Records  Log Execution Failure  Create Pipeline_ExecutionLog table Pipeline_ExecutionLog BEGIN Insert new Record Insert Metadata Insert Start time END End Time Status(1) Row Counts Pipeline Details ERROR End Time Status(2) Failure Message
  • 24. DEMO
  • 26. Can we get answers on the following questions? Can we build Synapse Pipelines dynamically? Can we extract data from my sources based on MetaData? Can we load the active(current) or historical records to a DataStore? Can we build history from extracted data based on MetaData? Can we log the execution of the Pipelines?
  • 27. Integration runtime on premises datasources Databricks Azure Synapse Datapipelines Azure SQL Azure SQL Database Data Lake Intermediate Zone Parquet Azure Synapse Analytics Azure Synapse Datapipelines Data Lake Raw Zone Parquet Data Store Delta Lake Data Lake EXTRACT PREP LOAD Auditing, Logging, Metadata and Execution Power BI HIGH OVERVIEW ARCHITECTURE NITROGEN Data Integration
  • 28. Process Flow For each Daily Run Data Lake Command Delta Lake Command Data Store Command Execute Data Lake Execute For each Execute Delta Lake Execute For each Execute Data Store Execute Auditing Pipeline_DataLake Pipeline_ExecutionLog Pipeline_DeltaLake Pipeline_DataStore Begin End Error Auditing Begin End Error Auditing Begib End Error Command Execute
  • 30. DEMO
  • 32. Can we get answers on the following questions? Can we build Synapse Pipelines dynamically? Can we extract data from my sources based on MetaData? Can we load the active(current) or historical records to a DataStore? Can we build history from extracted data based on MetaData? Can we log the execution of the Pipelines?
  • 34. Share your Love with the speaker and team on social, using the hashtag #cllmarathon and @CloudLunchLearn And follow us 😊 youtube.com/c/CloudLunchandLearn @CloudLunchLearn In collaboration with