Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?
Similar to Is there a way that we can build our Azure Data Factory all with parameters based on MetaData? (20)
More from Erwin de Kreuk
More from Erwin de Kreuk (9)
Recently uploaded
Recently uploaded (20)
Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?
- 1. #ScottishSummit2021 E r w i n d e K r e u k A z u r e D a t a F a c t o r y E l o n 1 7 : 0 0 G M T Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?
- 2. I n S p a r k L e a d D a t a & A I @ e r w i n d e k r e u k Erwin De Kreuk Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?
- 3. We help organizations accelerating their digital transformation with impactful Microsoft solutions & expertise We Are InSpark
- 4. Our Sponsors
- 6. • Hybrid data integration service • With visual tools, you can build, debug, deploy, operationalize and monitor your (big) data pipelines • Provides a way to transform data at scale without any coding required ELT Platform What is Azure Data Factory?
- 7. Scottish Summit Template Global Parameter Pipeline Parameter Dataset Parameter Notebook Parameter Linked Service Parameter Dataflow Parameter
- 8. Global Parameters Can be used across all your Pipelines Can be deployment in CI/CD pipeline().globalParameters.<parameterName>.
- 9. Can be used across all your Pipelines Can be deployment in CI/CD Global parameters - Azure Data Factory | Microsoft Docs Disabled Global Parameters
- 10. Enabled Global Parameters Can be used across all your Pipelines Can be deployment in CI/CD Global parameters - Azure Data Factory | Microsoft Docs
- 11. Dataset Parameters Create 1 dataset for all your Linked Services activities You can’t use Global Parameters FileSystem Directory FileName
- 12. Dataset Parameters Create 1 dataset for all your Linked Services activities You can’t use Global Parameters FileSystem Directory FileName
- 13. Pipeline Parameters Can be used across all your Pipelines
- 14. Notebook Parameters Pass Parameters from ADF to Databricks
- 15. Linked Services Parameters Connect to different Database on same Server Connect to different Logical Servers
- 16. • Pipeline • Global(Only ADF) • Linked Service Parameters • DataSet • Notebooks
- 18. DEMO
- 21. Can we get answers on the following questions? Can we build ADF Pipelines dynamically? Can we extract data from my sources based on MetaData? Can we load the active(current) or historical records to a DataStore? Can we build history from extracted data based on MetaData? Can we log the execution of the Pipelines?
- 23. Source Name Source Schema DataLake Catalog Table Destination Schema IsIncremental LastLoadTime Table Destination Name IsActive IsIncremental Column Metadata Source Parameter table
- 24. Lookup Get Source data ForEach For Each Execute Pipeline Load Lookup Get LastLoadDate Copy Copy Source to ADLS Stored Procedure Set LastLoadDate Command Execute SELECT [PipelineParameterId] ,[SourceName] ,[SourceSchema] ,[SelectQuery] ,[SelectLastLoaddate] ,[FilePath] ,[FileName] ,[TableDestinationName] ,[ProcessType] ,[IsActive] ,[IsIncremental] ,[IsIncrementalColumn] ,[LastLoadtime] FROM [execution].[Pipeline_DataLake_Files] SELECT case when 1=1 then convert(varchar,max(LasteditedWhen),120) else convert(varchar,getdate(),120) end as LastLoadDate FROM SourceSchema.SourceTable Metadata Source Parameter table
- 25. Logging Log Start and End Time of records Log Extracted Records Log Execution Failure Create Pipeline_ExecutionLog table [audit].[Event_Pipeline_OnBegin] [audit].[Event_Pipeline_OnEnd] [audit].[Event_Pipeline_OnError] PIPELINE ACTIVITY
- 26. Logging Log Start and End Time of records Log Extracted Records Log Execution Failure Create Pipeline_ExecutionLog table Pipeline_ExecutionLog BEGIN Insert new Record Insert Metadata Insert Start time END End Time Status(1) Row Counts Pipeline Details ERROR End Time Status(2) Failure Message
- 28. DEMO
- 30. Can we get answers on the following questions? Can we build ADF Pipelines dynamically? Can we extract data from my sources based on MetaData? Can we load the active(current) or historical records to a DataStore? Can we build history from extracted data based on MetaData? Can we log the execution of the Pipelines?
- 31. Integration runtime on premises datasources Databricks Data Factory Azure SQL Azure SQL Database Data Lake Intermediate Zone Parquet Azure Synapse Analytics Data Factory Data Lake Raw Zone Parquet Data Store Delta Lake Data Lake EXTRACT PREP LOAD Auditing, Logging, MetaData and Execution Power BI HIGH OVERVIEW ARCHITECTURE NITROGEN Data Accelerator
- 32. Process Flow For each Daily Run Data Lake Command Delta Lake Command Data Store Command Data Lake Execute For each Delta Lake Execute For each Data Store Execute Auditing Pipeline_DataLake Pipeline_ExecutionLog Pipeline_DeltaLake Pipeline_DataStore Begin End Error Auditing Begin End Error Auditing Begib End Error Command Execute
- 34. DEMO
- 36. Can we get answers on the following questions? Can we build ADF Pipelines dynamically? Can we extract data from my sources based on MetaData? Can we load the active(current) or historical records to a DataStore? Can we build history from extracted data based on MetaData? Can we log the execution of the Pipelines?
Editor's Notes
- Hallo and Welcome to my session about Is there a way that we can build our Azure Data Factory all with parameters based on MetaData?
- My name is Erwin de Kreuk and I’m working as a Lead Data and AI for InSpark a Microsoft Partner in the Netherlands
- Azure Data Factory is a Hybrid data integration During the session today I will explain how you can use Parameters within Azure DataFactory How you can replace these parameters with MetaData How we can log these dynamic pipelines And a quick walk through of a complete solution with DataBricks and Azure SQL Database as endpoint
- Hybrid data integration service where you easily extract data from On Prem Sources, Cloud Sources, SaaS application with more then 120 different DataConnectors With visual tools, you can build, debug, deploy, operationalize and monitor your data pipelines or big data Pipelines Provides an easy way to transform data at scale without any coding required ELT Platform. With Parameters you build a complete dynamically solution and this is what I’m going
- Passing parameters to ADF or Azure Synapse is quite important as it provides the flexibility required to create dynamic pipelines. To reference a parameter, you must provide the fully qualified name of the parameter. It is worth noting that parameter names are case sensitive. A parameter could be a user input, which means that the parameter is passed from the pipeline layer or could be an input coming from an activity within the pipeline.
- Global parameters can be used in any pipeline expression. If a pipeline is referencing another resource such as a dataset or data flow, you can pass down the global parameter value via that resource's parameters. Global Parameters are only available in Azure Data Factory and not in Azure Synapse Analytics You can Global Parameters in the Management Hub in ADF You must define the datatype of the Global Parameter Global parameters are referenced as pipeline().globalParameters.<parameterName>.
- There are two ways to integrate global parameters in your continuous integration and deployment solution: Include global parameters in the ARM template Deploy global parameters via a PowerShell script Or you can enable this box I’m going not that much in detail, but you can find details in the added link
- Or you can enable this box I will show you later in the Demo how you deploy these parameters to a next environment
- Create 1 Dataset for all your Activities per Linked Service
- Create 1 Dataset for all your Activities per Linked Service Explain parameter name Explain add dynamic content.
- You can define pipeline parameters to pass through values to your dataset example. How this works I will explain in the upcoming demo
- Use widgets in Databricks notebooks to use these as a Parameter in ADF
- Make your Linked Service Dynamiccaly, fe if you want to extract data from the same server but from different databases
- Samenvatting
- Linked Service DataBase Parameter Create DataBase Parameter Dataset Create Source Parameter SourceDatabase SourceTable SourceSchema Dataset create Datalake Folder FilePath Filename Pipeline Assign Pipeline Parameters
- Now we have learned to implement a Dynamically set up Pipeline is it time for the next stage How can we fill all the parameters based on metadata
- Source Name => The name of Source Table without schema name Source Schema The name of Source Schema DataLake Catalog Folder in the DataLake Table Destination Schema Table Destination Table IsActive IsIncremental IsIncremental LastLoadDataTime
- With the auditing we have created 3 Stored Proc. 1 where we will start the execution 1 where we will end the execution if successful 1 where we will end the execution if not successful
- With the auditing we have created 3 Stored Proc. 1 where we will start the execution 1 where we will end the execution if successful 1 where we will end the execution if not successful
- Show Source Parameters table Show [execution].[Pipeline_DataLake_Files] Show Execution pipeline Show Command Pipeline Get Files For Each Pipeline Execution Run Pipeline DEMO 3 Show SP On Begin, On End, On Error Explain Parent Log Id Show Execution Show Table
- Show Source Parameters table Show [execution].[Pipeline_DataLake_Files] Show Execution pipeline Show Command Pipeline Get Files For Each Pipeline Execution Run Pipeline DEMO 3 Show SP On Begin, On End, On Error Explain Parent Log Id Show Execution Show Table