SlideShare a Scribd company logo
Next Generation of Data Integration
with Azure Data Factory
Tom Kerkhove
Azure Consultant at Codit, MSFT Azure MVP
Expo Sponsors
Event Sponsors
Expo Light Sponsors
Hi!
Tom Kerkhove
• Azure Consultant at Codit
• Microsoft Azure MVP & Advisor
• Belgian Azure User Group (AZUG)
blog.tomkerkhove.be
@TomKerkhove
tomkerkhove
Azure Serverless
Azure Logic AppsAzure Functions Azure Event Grid
Azure Serverless
Azure Logic AppsAzure Functions Azure Event GridAzure Data Factory
Disclaimer
Azure Data Factory 2.0 Preview
https://bit.ly/adf-v1-vs-v2
➔ Managed data orchestration service
➔ Allows you to run pipelines
➔ Execute SSIS packages
➔ Support for hybrid scenarios
➔ Data movement-as-a-service with 70+ connectors
➔ Visual tooling & programmability
➔ .NET, Python, REST, ARM
What is Azure Data Factory?
What is Azure Data Factory?
Trigger(s) Activity ActivityActivity
Activity
Activity
Pipeline
➔ A pipeline represents a business process with multiple “steps”
which are represented by activities and is started by a trigger
➔ Activities represent a steps in a business process that perform
a specific action.
➔ This is based on the outcome of the previous step and can be on success,
failure, skipped or completion
What is Azure Data Factory?
➔ Different types of triggers
➔ On-Demand (Via REST API, .NET, etc.)
• Azure API Management can make this easier
➔ Scheduled / Wall-clock
➔ Tumbling Windows (aka “data slicing”)
➔ Event-based (New file is added to blob storage)
➔ Support for passing parameters
Triggers
What is Azure Data Factory?
Trigger(s) Activity ActivityActivity
Activity
Activity
Pipeline
➔ Data Movement
➔ Azure, Databases, NoSQL, File, SaaS, Web, etc
➔ Data Transformation
➔ Pig, Hive, Stored Procedure, U-SQL, ML, Spark, MapReduce, etc.
➔ Control Flow
➔ Web call, Lookup, Get Metadata, If, Wait, ForEach, Execute Pipeline, etc
➔ Custom
➔ Run commands on an Azure Batch cluster
➔ Run R scripts on a HDInsight cluster
Activities
➔ An activity can produce or consume a data set. It is a
representation of a data structure in a data store that can be
used as a source or sink.
➔ Linked Services define how an activity can connect to an
external system. This external system can be a data store or
compute resource.
What is Azure Data Factory?
What is Azure Data Factory?
Activity
Data
Set
Linked
Service
Represents data
stored in
Produces
Consumes
➔ Compute infrastructure used by Data Factory
➔ Azure, Azure-SSIS or Self-Hosted (Any cloud or on-prem)
➔ Core capabilities
➔ Data movement
➔ Pipeline activity execution
➔ SSIS package execution
➔ Pipelines issues commands & control, integration runtime executes
➔ Data movement is from IR to IR
➔ All executions are happening in sources & sinks
Integration Runtime (IR)
Integration Runtime (IR)
➔ Stores SSISDB in Azure SQL DB or Managed Instance
➔ Azure-SSIS integration runtime as compute-layer
➔ Compute part for running SSIS
➔ Managed cluster of Azure VMs
➔ Compute-layer
➔ Can be linked to VNET for hybrid scenarios
➔ Lift & shift packages to the cloud
Running SSIS packages in Azure
Running SSIS packages in Azure
➔ Native support for Managed Service Identity (MSI)
➔ Native integration with Azure Key Vault
➔ Encrypted-in-transit via HTTPS
➔ Supports encryption-at-rest with data stores
Security
Show it to me!
➔ Every user should be capable of requesting their data
Using Azure Serverless to become GDPR compliant
User Profile
information
StackExchange
Data Set
Kerkhove.tom
@gmail.com
Using Azure Serverless to become GDPR compliant
➔ Visual monitoring in the portal
➔ Monitoring per pipeline run
➔ Detailed information per activity
➔ Azure Monitor integration
➔ Diagnostic Logs
➔ Metrics
➔ Alerts
Monitoring
➔ Serverless orchestration
➔ Pay for what you use
➔ Data-centric vs Application-centric workflows
➔ Work together seamlessly
How is this different from Logic Apps?
➔ Azure Data Factory is a great way to orchestrate data
processes and build data-integration pipelines
➔ Very powerful for data-centric workloads
➔ Unsung hero in the serverless space
➔ A perfect match with Azure Logic Apps
➔ Allows you to get to market very quickly with the built-in
connectors
Conclusion
28

More Related Content

What's hot

Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Elasticsearch
 
Azure Container Services​
Azure Container Services​Azure Container Services​
Azure Container Services​Pedro Sousa
 
Toyko azure meetup # 1 azure paa s overview
Toyko azure meetup # 1   azure paa s overviewToyko azure meetup # 1   azure paa s overview
Toyko azure meetup # 1 azure paa s overviewTokyo Azure Meetup
 
Lift SSIS package to Azure Data Factory V2
Lift SSIS package to Azure Data Factory V2Lift SSIS package to Azure Data Factory V2
Lift SSIS package to Azure Data Factory V2Manjeet Singh
 
One Azure Monitor to Rule Them All? - Marius Zaharia
One Azure Monitor to Rule Them All? - Marius ZahariaOne Azure Monitor to Rule Them All? - Marius Zaharia
One Azure Monitor to Rule Them All? - Marius ZahariaITCamp
 
Refactoring the Monolith: A Systematic Approach to Application Modernization
Refactoring the Monolith: A Systematic Approach to Application ModernizationRefactoring the Monolith: A Systematic Approach to Application Modernization
Refactoring the Monolith: A Systematic Approach to Application ModernizationDynatrace
 
Mastering Azure Monitor
Mastering Azure MonitorMastering Azure Monitor
Mastering Azure MonitorRichard Conway
 
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch on Azure
Elasticsearch on AzureElasticsearch
 
The hidden secrets of azure networking
The hidden secrets of azure networkingThe hidden secrets of azure networking
The hidden secrets of azure networkingMohamed Wali
 
Configuration in azure done right
Configuration in azure done rightConfiguration in azure done right
Configuration in azure done rightRick van den Bosch
 
Search for All with Elastic Workplace Search
Search for All with Elastic Workplace Search Search for All with Elastic Workplace Search
Search for All with Elastic Workplace Search Elasticsearch
 
The full picture of Openstack in real-time
The full picture of Openstack in real-timeThe full picture of Openstack in real-time
The full picture of Openstack in real-timeDynatrace
 
Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Elasticsearch
 
Azure Container Instance
Azure Container InstanceAzure Container Instance
Azure Container InstanceBishoy Demian
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsBetter Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsElasticsearch
 
Cncf event driven autoscaling with keda
Cncf   event driven autoscaling with kedaCncf   event driven autoscaling with keda
Cncf event driven autoscaling with kedaJurajHantk
 
Search for All with Elastic Enterprise Search
Search for All with Elastic Enterprise Search Search for All with Elastic Enterprise Search
Search for All with Elastic Enterprise Search Elasticsearch
 
Microsoft Azure News - Nov 2016
Microsoft Azure News - Nov 2016Microsoft Azure News - Nov 2016
Microsoft Azure News - Nov 2016Daniel Toomey
 
Offline first with Couchbase Mobile
Offline first with Couchbase MobileOffline first with Couchbase Mobile
Offline first with Couchbase MobileVegard Haugstvedt
 

What's hot (20)

Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
 
Azure Container Services​
Azure Container Services​Azure Container Services​
Azure Container Services​
 
Toyko azure meetup # 1 azure paa s overview
Toyko azure meetup # 1   azure paa s overviewToyko azure meetup # 1   azure paa s overview
Toyko azure meetup # 1 azure paa s overview
 
Lift SSIS package to Azure Data Factory V2
Lift SSIS package to Azure Data Factory V2Lift SSIS package to Azure Data Factory V2
Lift SSIS package to Azure Data Factory V2
 
One Azure Monitor to Rule Them All? - Marius Zaharia
One Azure Monitor to Rule Them All? - Marius ZahariaOne Azure Monitor to Rule Them All? - Marius Zaharia
One Azure Monitor to Rule Them All? - Marius Zaharia
 
Refactoring the Monolith: A Systematic Approach to Application Modernization
Refactoring the Monolith: A Systematic Approach to Application ModernizationRefactoring the Monolith: A Systematic Approach to Application Modernization
Refactoring the Monolith: A Systematic Approach to Application Modernization
 
DBaaS at Scale
DBaaS at ScaleDBaaS at Scale
DBaaS at Scale
 
Mastering Azure Monitor
Mastering Azure MonitorMastering Azure Monitor
Mastering Azure Monitor
 
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch on Azure
Elasticsearch on Azure
 
The hidden secrets of azure networking
The hidden secrets of azure networkingThe hidden secrets of azure networking
The hidden secrets of azure networking
 
Configuration in azure done right
Configuration in azure done rightConfiguration in azure done right
Configuration in azure done right
 
Search for All with Elastic Workplace Search
Search for All with Elastic Workplace Search Search for All with Elastic Workplace Search
Search for All with Elastic Workplace Search
 
The full picture of Openstack in real-time
The full picture of Openstack in real-timeThe full picture of Openstack in real-time
The full picture of Openstack in real-time
 
Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box Building a reliable and cost effect logging system at Box
Building a reliable and cost effect logging system at Box
 
Azure Container Instance
Azure Container InstanceAzure Container Instance
Azure Container Instance
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsBetter Search and Business Analytics at Southern Glazer’s Wine & Spirits
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
 
Cncf event driven autoscaling with keda
Cncf   event driven autoscaling with kedaCncf   event driven autoscaling with keda
Cncf event driven autoscaling with keda
 
Search for All with Elastic Enterprise Search
Search for All with Elastic Enterprise Search Search for All with Elastic Enterprise Search
Search for All with Elastic Enterprise Search
 
Microsoft Azure News - Nov 2016
Microsoft Azure News - Nov 2016Microsoft Azure News - Nov 2016
Microsoft Azure News - Nov 2016
 
Offline first with Couchbase Mobile
Offline first with Couchbase MobileOffline first with Couchbase Mobile
Offline first with Couchbase Mobile
 

Similar to Intelligent Cloud Conference 2018 - Next Generation of Data Integration with Azure Data Factory

NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
NDC Minnesota - Analyzing StackExchange data with Azure Data LakeNDC Minnesota - Analyzing StackExchange data with Azure Data Lake
NDC Minnesota - Analyzing StackExchange data with Azure Data LakeTom Kerkhove
 
Azure Data Factory for Redmond SQL PASS UG Sept 2018
Azure Data Factory for Redmond SQL PASS UG Sept 2018Azure Data Factory for Redmond SQL PASS UG Sept 2018
Azure Data Factory for Redmond SQL PASS UG Sept 2018Mark Kromer
 
Integration Monday - Analysing StackExchange data with Azure Data Lake
Integration Monday - Analysing StackExchange data with Azure Data LakeIntegration Monday - Analysing StackExchange data with Azure Data Lake
Integration Monday - Analysing StackExchange data with Azure Data LakeTom Kerkhove
 
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...Codit
 
Analyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data LakeAnalyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data LakeBizTalk360
 
Cepta The Future of Data with Power BI
Cepta The Future of Data with Power BICepta The Future of Data with Power BI
Cepta The Future of Data with Power BIKellyn Pot'Vin-Gorman
 
Azure fundamental -Introduction
Azure fundamental -IntroductionAzure fundamental -Introduction
Azure fundamental -IntroductionManishK55
 
Azure Data.pptx
Azure Data.pptxAzure Data.pptx
Azure Data.pptxFedoRam1
 
Leveraging cloud database connectors to automate analytics in alteryx
Leveraging cloud database connectors to automate analytics in alteryxLeveraging cloud database connectors to automate analytics in alteryx
Leveraging cloud database connectors to automate analytics in alteryxGrazitti Interactive
 
Azure Data Factory Introduction.pdf
Azure Data Factory Introduction.pdfAzure Data Factory Introduction.pdf
Azure Data Factory Introduction.pdfMaheshPandit16
 
NDC Sydney - Analyzing StackExchange with Azure Data Lake
NDC Sydney - Analyzing StackExchange with Azure Data LakeNDC Sydney - Analyzing StackExchange with Azure Data Lake
NDC Sydney - Analyzing StackExchange with Azure Data LakeTom Kerkhove
 
MongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoT
MongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoTMongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoT
MongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoTMongoDB
 
Azure from scratch part 3 By Girish Kalamati
Azure from scratch part 3 By Girish KalamatiAzure from scratch part 3 By Girish Kalamati
Azure from scratch part 3 By Girish KalamatiGirish Kalamati
 
Microsoft Azure Stack
Microsoft Azure StackMicrosoft Azure Stack
Microsoft Azure StackTudor Damian
 
Global Azure Bootcamp 2018 - Azure Network Security
Global Azure Bootcamp 2018 - Azure Network SecurityGlobal Azure Bootcamp 2018 - Azure Network Security
Global Azure Bootcamp 2018 - Azure Network SecurityScott Hoag
 
Azure Data Factory for Azure Data Week
Azure Data Factory for Azure Data WeekAzure Data Factory for Azure Data Week
Azure Data Factory for Azure Data WeekMark Kromer
 
Adelaide Global Azure Bootcamp 2018 - Azure 101
Adelaide Global Azure Bootcamp 2018 - Azure 101Adelaide Global Azure Bootcamp 2018 - Azure 101
Adelaide Global Azure Bootcamp 2018 - Azure 101Balabiju
 
Introducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseIntroducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseJames Serra
 
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data AnalyticsHow to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data AnalyticsInformatica
 

Similar to Intelligent Cloud Conference 2018 - Next Generation of Data Integration with Azure Data Factory (20)

NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
NDC Minnesota - Analyzing StackExchange data with Azure Data LakeNDC Minnesota - Analyzing StackExchange data with Azure Data Lake
NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
 
Azure Data Factory for Redmond SQL PASS UG Sept 2018
Azure Data Factory for Redmond SQL PASS UG Sept 2018Azure Data Factory for Redmond SQL PASS UG Sept 2018
Azure Data Factory for Redmond SQL PASS UG Sept 2018
 
Integration Monday - Analysing StackExchange data with Azure Data Lake
Integration Monday - Analysing StackExchange data with Azure Data LakeIntegration Monday - Analysing StackExchange data with Azure Data Lake
Integration Monday - Analysing StackExchange data with Azure Data Lake
 
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
 
Analyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data LakeAnalyzing StackExchange data with Azure Data Lake
Analyzing StackExchange data with Azure Data Lake
 
Cepta The Future of Data with Power BI
Cepta The Future of Data with Power BICepta The Future of Data with Power BI
Cepta The Future of Data with Power BI
 
Azure fundamental -Introduction
Azure fundamental -IntroductionAzure fundamental -Introduction
Azure fundamental -Introduction
 
Azure Data.pptx
Azure Data.pptxAzure Data.pptx
Azure Data.pptx
 
Leveraging cloud database connectors to automate analytics in alteryx
Leveraging cloud database connectors to automate analytics in alteryxLeveraging cloud database connectors to automate analytics in alteryx
Leveraging cloud database connectors to automate analytics in alteryx
 
Azure Data Factory Introduction.pdf
Azure Data Factory Introduction.pdfAzure Data Factory Introduction.pdf
Azure Data Factory Introduction.pdf
 
NDC Sydney - Analyzing StackExchange with Azure Data Lake
NDC Sydney - Analyzing StackExchange with Azure Data LakeNDC Sydney - Analyzing StackExchange with Azure Data Lake
NDC Sydney - Analyzing StackExchange with Azure Data Lake
 
MongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoT
MongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoTMongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoT
MongoDB IoT City Tour STUTTGART: The Microsoft Azure Platform for IoT
 
Azure from scratch part 3 By Girish Kalamati
Azure from scratch part 3 By Girish KalamatiAzure from scratch part 3 By Girish Kalamati
Azure from scratch part 3 By Girish Kalamati
 
Azure bootcamp (1)
Azure bootcamp (1)Azure bootcamp (1)
Azure bootcamp (1)
 
Microsoft Azure Stack
Microsoft Azure StackMicrosoft Azure Stack
Microsoft Azure Stack
 
Global Azure Bootcamp 2018 - Azure Network Security
Global Azure Bootcamp 2018 - Azure Network SecurityGlobal Azure Bootcamp 2018 - Azure Network Security
Global Azure Bootcamp 2018 - Azure Network Security
 
Azure Data Factory for Azure Data Week
Azure Data Factory for Azure Data WeekAzure Data Factory for Azure Data Week
Azure Data Factory for Azure Data Week
 
Adelaide Global Azure Bootcamp 2018 - Azure 101
Adelaide Global Azure Bootcamp 2018 - Azure 101Adelaide Global Azure Bootcamp 2018 - Azure 101
Adelaide Global Azure Bootcamp 2018 - Azure 101
 
Introducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseIntroducing Azure SQL Data Warehouse
Introducing Azure SQL Data Warehouse
 
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data AnalyticsHow to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
 

More from Tom Kerkhove

Techorama 2022 - Adventures of building Promitor, an open-source product
Techorama 2022 - Adventures of building Promitor, an open-source productTechorama 2022 - Adventures of building Promitor, an open-source product
Techorama 2022 - Adventures of building Promitor, an open-source productTom Kerkhove
 
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...Tom Kerkhove
 
Introduction to Promitor
Introduction to PromitorIntroduction to Promitor
Introduction to PromitorTom Kerkhove
 
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...Tom Kerkhove
 
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...Tom Kerkhove
 
Global Azure Virtual - Application Autoscaling with KEDA
Global Azure Virtual - Application Autoscaling with KEDAGlobal Azure Virtual - Application Autoscaling with KEDA
Global Azure Virtual - Application Autoscaling with KEDATom Kerkhove
 
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...Tom Kerkhove
 
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...Tom Kerkhove
 
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...Tom Kerkhove
 
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft AzureIglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft AzureTom Kerkhove
 
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...Tom Kerkhove
 
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...Tom Kerkhove
 
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...Tom Kerkhove
 
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key VaultAzure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key VaultTom Kerkhove
 
Next Generation Data Integration with Azure Data Factory
Next Generation Data Integration with Azure Data FactoryNext Generation Data Integration with Azure Data Factory
Next Generation Data Integration with Azure Data FactoryTom Kerkhove
 
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...Tom Kerkhove
 
Techdays Finland 2018 - Building secure cloud applications with Azure Key Vault
Techdays Finland 2018 - Building secure cloud applications with Azure Key VaultTechdays Finland 2018 - Building secure cloud applications with Azure Key Vault
Techdays Finland 2018 - Building secure cloud applications with Azure Key VaultTom Kerkhove
 
TechDays NL 2016 - Building your scalable secure IoT Solution on Azure
TechDays NL 2016 - Building your scalable secure IoT Solution on AzureTechDays NL 2016 - Building your scalable secure IoT Solution on Azure
TechDays NL 2016 - Building your scalable secure IoT Solution on AzureTom Kerkhove
 
ITProceed 2015 - Securing Sensitive Data with Azure Key Vault
ITProceed 2015 - Securing Sensitive Data with Azure Key VaultITProceed 2015 - Securing Sensitive Data with Azure Key Vault
ITProceed 2015 - Securing Sensitive Data with Azure Key VaultTom Kerkhove
 
Securing sensitive data with Azure Key Vault
Securing sensitive data with Azure Key VaultSecuring sensitive data with Azure Key Vault
Securing sensitive data with Azure Key VaultTom Kerkhove
 

More from Tom Kerkhove (20)

Techorama 2022 - Adventures of building Promitor, an open-source product
Techorama 2022 - Adventures of building Promitor, an open-source productTechorama 2022 - Adventures of building Promitor, an open-source product
Techorama 2022 - Adventures of building Promitor, an open-source product
 
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
 
Introduction to Promitor
Introduction to PromitorIntroduction to Promitor
Introduction to Promitor
 
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
 
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
 
Global Azure Virtual - Application Autoscaling with KEDA
Global Azure Virtual - Application Autoscaling with KEDAGlobal Azure Virtual - Application Autoscaling with KEDA
Global Azure Virtual - Application Autoscaling with KEDA
 
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
 
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
 
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
 
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft AzureIglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
 
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
 
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
 
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
 
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key VaultAzure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
 
Next Generation Data Integration with Azure Data Factory
Next Generation Data Integration with Azure Data FactoryNext Generation Data Integration with Azure Data Factory
Next Generation Data Integration with Azure Data Factory
 
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
 
Techdays Finland 2018 - Building secure cloud applications with Azure Key Vault
Techdays Finland 2018 - Building secure cloud applications with Azure Key VaultTechdays Finland 2018 - Building secure cloud applications with Azure Key Vault
Techdays Finland 2018 - Building secure cloud applications with Azure Key Vault
 
TechDays NL 2016 - Building your scalable secure IoT Solution on Azure
TechDays NL 2016 - Building your scalable secure IoT Solution on AzureTechDays NL 2016 - Building your scalable secure IoT Solution on Azure
TechDays NL 2016 - Building your scalable secure IoT Solution on Azure
 
ITProceed 2015 - Securing Sensitive Data with Azure Key Vault
ITProceed 2015 - Securing Sensitive Data with Azure Key VaultITProceed 2015 - Securing Sensitive Data with Azure Key Vault
ITProceed 2015 - Securing Sensitive Data with Azure Key Vault
 
Securing sensitive data with Azure Key Vault
Securing sensitive data with Azure Key VaultSecuring sensitive data with Azure Key Vault
Securing sensitive data with Azure Key Vault
 

Recently uploaded

A Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationA Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationHelp Desk Migration
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAlluxio, Inc.
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j
 
10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdfkalichargn70th171
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityamy56318795
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Krakówbim.edu.pl
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfMehmet Akar
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzisteffenkarlsson2
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfkalichargn70th171
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1KnowledgeSeed
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024vaibhav130304
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.
 
iGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockiGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockSkilrock Technologies
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)Max Lee
 
Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfMeon Technology
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy
 

Recently uploaded (20)

A Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data MigrationA Guideline to Gorgias to to Re:amaze Data Migration
A Guideline to Gorgias to to Re:amaze Data Migration
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 
Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf10 Essential Software Testing Tools You Need to Know About.pdf
10 Essential Software Testing Tools You Need to Know About.pdf
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
iGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by SkilrockiGaming Platform & Lottery Solutions by Skilrock
iGaming Platform & Lottery Solutions by Skilrock
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdf
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 

Intelligent Cloud Conference 2018 - Next Generation of Data Integration with Azure Data Factory

  • 1.
  • 2. Next Generation of Data Integration with Azure Data Factory Tom Kerkhove Azure Consultant at Codit, MSFT Azure MVP
  • 4. Hi! Tom Kerkhove • Azure Consultant at Codit • Microsoft Azure MVP & Advisor • Belgian Azure User Group (AZUG) blog.tomkerkhove.be @TomKerkhove tomkerkhove
  • 5. Azure Serverless Azure Logic AppsAzure Functions Azure Event Grid
  • 6. Azure Serverless Azure Logic AppsAzure Functions Azure Event GridAzure Data Factory
  • 7. Disclaimer Azure Data Factory 2.0 Preview https://bit.ly/adf-v1-vs-v2
  • 8. ➔ Managed data orchestration service ➔ Allows you to run pipelines ➔ Execute SSIS packages ➔ Support for hybrid scenarios ➔ Data movement-as-a-service with 70+ connectors ➔ Visual tooling & programmability ➔ .NET, Python, REST, ARM What is Azure Data Factory?
  • 9. What is Azure Data Factory? Trigger(s) Activity ActivityActivity Activity Activity Pipeline
  • 10. ➔ A pipeline represents a business process with multiple “steps” which are represented by activities and is started by a trigger ➔ Activities represent a steps in a business process that perform a specific action. ➔ This is based on the outcome of the previous step and can be on success, failure, skipped or completion What is Azure Data Factory?
  • 11. ➔ Different types of triggers ➔ On-Demand (Via REST API, .NET, etc.) • Azure API Management can make this easier ➔ Scheduled / Wall-clock ➔ Tumbling Windows (aka “data slicing”) ➔ Event-based (New file is added to blob storage) ➔ Support for passing parameters Triggers
  • 12. What is Azure Data Factory? Trigger(s) Activity ActivityActivity Activity Activity Pipeline
  • 13. ➔ Data Movement ➔ Azure, Databases, NoSQL, File, SaaS, Web, etc ➔ Data Transformation ➔ Pig, Hive, Stored Procedure, U-SQL, ML, Spark, MapReduce, etc. ➔ Control Flow ➔ Web call, Lookup, Get Metadata, If, Wait, ForEach, Execute Pipeline, etc ➔ Custom ➔ Run commands on an Azure Batch cluster ➔ Run R scripts on a HDInsight cluster Activities
  • 14. ➔ An activity can produce or consume a data set. It is a representation of a data structure in a data store that can be used as a source or sink. ➔ Linked Services define how an activity can connect to an external system. This external system can be a data store or compute resource. What is Azure Data Factory?
  • 15. What is Azure Data Factory? Activity Data Set Linked Service Represents data stored in Produces Consumes
  • 16. ➔ Compute infrastructure used by Data Factory ➔ Azure, Azure-SSIS or Self-Hosted (Any cloud or on-prem) ➔ Core capabilities ➔ Data movement ➔ Pipeline activity execution ➔ SSIS package execution ➔ Pipelines issues commands & control, integration runtime executes ➔ Data movement is from IR to IR ➔ All executions are happening in sources & sinks Integration Runtime (IR)
  • 17.
  • 19. ➔ Stores SSISDB in Azure SQL DB or Managed Instance ➔ Azure-SSIS integration runtime as compute-layer ➔ Compute part for running SSIS ➔ Managed cluster of Azure VMs ➔ Compute-layer ➔ Can be linked to VNET for hybrid scenarios ➔ Lift & shift packages to the cloud Running SSIS packages in Azure
  • 21. ➔ Native support for Managed Service Identity (MSI) ➔ Native integration with Azure Key Vault ➔ Encrypted-in-transit via HTTPS ➔ Supports encryption-at-rest with data stores Security
  • 22. Show it to me!
  • 23. ➔ Every user should be capable of requesting their data Using Azure Serverless to become GDPR compliant User Profile information StackExchange Data Set Kerkhove.tom @gmail.com
  • 24. Using Azure Serverless to become GDPR compliant
  • 25. ➔ Visual monitoring in the portal ➔ Monitoring per pipeline run ➔ Detailed information per activity ➔ Azure Monitor integration ➔ Diagnostic Logs ➔ Metrics ➔ Alerts Monitoring
  • 26. ➔ Serverless orchestration ➔ Pay for what you use ➔ Data-centric vs Application-centric workflows ➔ Work together seamlessly How is this different from Logic Apps?
  • 27. ➔ Azure Data Factory is a great way to orchestrate data processes and build data-integration pipelines ➔ Very powerful for data-centric workloads ➔ Unsung hero in the serverless space ➔ A perfect match with Azure Logic Apps ➔ Allows you to get to market very quickly with the built-in connectors Conclusion
  • 28. 28

Editor's Notes

  1. Evangelistisch overkomen – adviseren Gevoel: heeft visie .
  2. Evangelistisch overkomen – adviseren Gevoel: heeft visie .