Is there a way that we can build our Synapse Data Pipelines all with parameters all based on MetaData? Yes there's and I will show you how to. During this session I will show how you can load Incremental or Full datasets from your sql database to your Azure Data Lake. The next step is that we want to track our history from these extracted tables. We will do using Delta Lake. The last step that we want, is to make this data available in Azure SQL Database or Azure Synapse Analytics. Oh and we want to have some logging as well from our processes A lot to talk and to demo about during this session.

1. Sponsored by
Is there a way that we can build our Azure Synapse
Pipelines all with parameters based on MetaData?
Erwin de Kreuk
Lead Data and AI @InSpark
13 May 10:00 - 11:00 (UTC)
13 May 12:00 - 13:00 (UTC+01:00) CET
All sessions available on www.cloudlunchlearn.com/marathon
<

2. Erwin de Kreuk
Lead Data and AI @InSpark
@erwindekreuk
https://www.linkedin.com/in/erwindekreuk/
https://erwindekreuk.com

3. We help organizations
accelerating their digital
transformation with impactful
Microsoft solutions & expertise
We Are InSpark

4. Azure Synapse Analytics

5. Scottish
Summit
Template
 Pipeline Parameter
 Dataset Parameter
 Notebook Parameter
 Linked Service Parameter
 Dataflow Parameter

6. Linked Services Parameters
 Connect to different Database
on same Server
 Connect to different Logical
Servers

7. Dataset Parameters
 Create 1 dataset for all your
Linked Services activities
FileSystem Directory FileName

8. Dataset Parameters
 Create 1 dataset for all your
Linked Services activities
FileSystem Directory FileName

9. Dataflow Parameters
 Pass parameters to Dataflow

10. Notebook Parameters
 Pass Parameters from
Synapse to Databricks

11. Pipeline Parameters
 Can be used across all your
Pipelines

12. • Notebooks
• Pipeline
• DataSet
• Linked Service
Parameters
• Dataflow

14. DEMO

15. Innovate
to
accelerate

16. Metadata
Load your pipelines
dynamically

17. Can we get answers on the following
questions?
Can we build Synapse Pipelines
dynamically?
Can we extract data from my
sources based on MetaData?
Can we load the active(current) or
historical records to a DataStore?
Can we build history from
extracted data based on
MetaData?
Can we log the execution of the
Pipelines?

19. Source Name
Source Schema
DataLake
Catalog
Table
Destination
Schema
IsIncremental LastLoadTime
Table
Destination
Name
IsActive
IsIncremental
Column
Metadata
Source Parameter table

20. Lookup
Get Source data
ForEach
For Each
Execute Pipeline
Load
Lookup
Get LastLoadDate
Copy
Copy Source to
ADLS
Stored Procedure
Set LastLoadDate
Command
Execute
SELECT [PipelineParameterId]
,[SourceName]
,[SourceSchema]
,[SelectQuery]
,[SelectLastLoaddate]
,[FilePath]
,[FileName]
,[TableDestinationName]
,[ProcessType]
,[IsActive]
,[IsIncremental]
,[IsIncrementalColumn]
,[LastLoadtime]
FROM [execution].[Pipeline_DataLake_Files]
SELECT case when 1=1 then
convert(varchar,max(LasteditedWhen),120) else
convert(varchar,getdate(),120) end as LastLoadDate
FROM SourceSchema.SourceTable
Metadata
Source Parameter table

21.  Log Start and End Time of
records
 Log Extracted Records
 Log Execution Failure
 Create Pipeline_ExecutionLog
table
[audit].[Event_Pipeline_OnBegin] [audit].[Event_Pipeline_OnEnd]
[audit].[Event_Pipeline_OnError]
PIPELINE ACTIVITY
Logging

22. Logging  Log Start and End Time of
records
 Log Extracted Records
 Log Execution Failure
 Create Pipeline_ExecutionLog
table
Pipeline_ExecutionLog
BEGIN
Insert new Record
Insert Metadata
Insert Start time
END
End Time
Status(1)
Row Counts
Pipeline Details
ERROR
End Time
Status(2)
Failure Message

24. DEMO

25. Innovate
to
accelerate

26. Can we get answers on the following
questions?
Can we build Synapse Pipelines
dynamically?
Can we extract data from my
sources based on MetaData?
Can we load the active(current) or
historical records to a DataStore?
Can we build history from
extracted data based on
MetaData?
Can we log the execution of the
Pipelines?

27. Integration
runtime
on premises
datasources
Databricks
Azure
Synapse
Datapipelines
Azure SQL
Azure SQL
Database Data Lake
Intermediate Zone
Parquet
Azure
Synapse
Analytics
Azure
Synapse
Datapipelines
Data Lake
Raw Zone
Parquet
Data Store
Delta Lake
Data Lake
EXTRACT PREP LOAD
Auditing, Logging, Metadata and Execution
Power BI
HIGH OVERVIEW ARCHITECTURE
NITROGEN Data Integration

28. Process Flow
For each
Daily Run
Data Lake
Command
Delta Lake
Command
Data Store
Command
Execute
Data Lake
Execute
For each
Execute
Delta Lake
Execute
For each
Execute
Data Store
Execute
Auditing
Pipeline_DataLake
Pipeline_ExecutionLog
Pipeline_DeltaLake Pipeline_DataStore
Begin End Error
Auditing
Begin End Error
Auditing
Begib End Error
Command
Execute

30. DEMO

31. Innovate
to
accelerate

32. Can we get answers on the following
questions?
Can we build Synapse Pipelines
dynamically?
Can we extract data from my
sources based on MetaData?
Can we load the active(current) or
historical records to a DataStore?
Can we build history from
extracted data based on
MetaData?
Can we log the execution of the
Pipelines?

33. @erwindekreuk
https://www.linkedin.com/in/erwindekreuk/
https://erwindekreuk.com
Slides will be available on my blog

34. Share your Love with the speaker and team on social, using the hashtag #cllmarathon and @CloudLunchLearn
And follow us 😊
youtube.com/c/CloudLunchandLearn @CloudLunchLearn
In collaboration with