Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
Michael Stephenson, profile picture
Uploaded byMichael Stephenson
PPTX, PDF163 views

LeedsSharp May 2023 - Azure Integration Services

The document discusses Azure Integration Services, exploring its components like Logic Apps, Functions, and Data Factory, and their use cases in enterprise integration. It provides insights into migration from older Microsoft integration technologies, the evolution of integration trends, and practical examples of combining different Azure services to optimize workflows. Additionally, it addresses common questions regarding the appropriate use of various Azure tools, including comparisons and best practices for scalable implementations.

Technology◦
Related topics:
Microsoft Azure•
Download to read offline
Azure Integration Services – Where should I use what?
Azure Integration Services – Where should I use what?
About me • Microsoft MVP 14 years • Freelance Cloud Architect • Based in Newcastle, UK • https://mikestephenson.me/ • @Michael_Stephen • Product Advisor for Serverless360
What have we been up to Business • Dozens of projects • Multi-Year investment in Microsoft IPaaS • EAI Platform = Microsoft iPaaS • EDMS Platform = Microsoft Data Platform • SAP / Power Platform / Power BI / Workday / IBM Maximo / SuccessFactors Technical • Environments - 5 • Resources (Across all environments) • Logic Apps – 1895 • Function Apps – 116 • Integration Account – 5 • APIM – 5 • Service Bus • Event Grid • Event Hub • Synapse
“What is Azure Integration Services and why should I care”? “Where do I use what”?
History of Microsoft Integration – Way back! BizTalk SSIS Custom - Hard learning curve - Specific skill set - Messaging / Orchestration / ETL based - Complex to implement - DBA driven - Specific skill set - ETL - Most orgs had SQL - .net or script based - Maybe API’s or WCF
History of Integration Trends from yester-year iPaaS has seen customers transition to getting stuff done!
ETL & ELT Security & Governance Enterprise Integration Platform Durable Messaging Data Transformation Helper Functions Storage Hybrid EAI API Enterprise Scenarios Business Rules System Workflow Human Workflow Business Users B2B Integration Management & Monitoring Power BI API Management Data Factory Key Vault Logic Apps Logic Apps Integration Account Integration Account Power Automate Azure Functions Service Bus Logic Apps Azure Functions Advisor Security Centre Storage Azure SQL DB Logic App Inline Code Data Gateway Power App VNet Integration Enterprise Connectors Azure Monitor Application Insights Application Configuration Cosmos DB Azure Functions Log Analytics Synapse Pipelines API Management
Usage Scenarios BizTalk Migration SSIS Migration Custom Developed Integration IT Admin Automation User Automation Application Developers Dynamics / Power Platform Developers SharePoint Developers Citizen Developers Common Skillsets Share resources / Team Members
The challenges are different • I want a product not a platform! • Azure is like Lego how do I know which bricks to use? • Instead of 1 or 2 really challenging skillsets there are now LOTS of skillsets but they are easier to learn
What questions do people usually have? • Logic Apps vs Power Automate • Logic Apps vs Data Factory • Logic Apps vs Functions • Logic Apps Consumption vs Logic Apps Standard • Data Factory vs Synapse • Service Bus vs Event Hub • Service Bus vs Event Grid • Event Hub vs Event Grid • API Management vs Functions
Secret Sauce – Effectively Combining Technologies Combine Examples Logic Apps + Power Automate System Workflow + Human Workflow for approvals Logic Apps + Data Factory Trigger ETL processes as part of an automation Synapse Pipeline + Logic App Simple SharePoint integration to publish data to users Logic Apps + Functions Custom code to support my orchestration API Management + Functions + Logic Apps Centralized management of API’s used by my integration platform Service Bus + Logic Apps Load levelling and pub/sub messaging for complex integrations Event Hub + Functions High scale processing of event data
Real-world Journey of Patterns
Step 1 – Tactical Point to Point Logic Apps Trigger Validate Message Get Existing Data Do some stuff Do some other stuff Order Updates Do more stuff Create record Create other record Return Response
What happened Things that weren’t working: - Doing too much in Logic Apps - Rebuild & Repeat rather than Reuse & Extend - The implementation was getting complex
Step 2 – Enter Service Bus, API Management & Functions Loaded Railcar Message Conversion API Convert Weights etc Enriched and formatted Message Archive Support User Manually Loaded railcars And for testing Loaded Railcar Data to CRM Loaded Railcar Data to SAP Loaded Railcar Data to Transport System Transport System Loaded Railcar Receiver Other interfaces Other interfaces Plant Data Receiver
Step 3 – Add Publish data to Data Lake Loaded Railcar Message Conversion API Convert Weights etc Enriched and formatted Message Synapse Data Platform Archive Support User Power App Power BI Manually Loaded railcars And for testing Loaded Railcar Data to CRM Loaded Railcar Data to SAP Loaded Railcar Data to Transport System Loaded Railcar Data to Synapse Transport System Loaded Railcar Receiver Other interfaces Other interfaces Plant Data Receiver
Step 4 – Add Synapse to build a Data Platform Integration Platform EAI Transactions to EDMS EDMS Platform Event Hub Capture Avro files Data Lake Synapse Analytics Synapse Pipelines
Import Step 4 – Start Building out Data Platform with Synapse Data Platform EAI to EDMS Functions Landing Transform & Orchestrate Serve Data Lake Storage Synapse Pipelines Synapse Spark Capture Avro files Synapse Analytics Power App EAI Transactions Parquet files Bulk Imports Enterprise Reports Power BI Experimentation Power BI
Step 5 – Use the Data Platform to Help your EAI Import Data Platform Landing Transform & Orchestrate Serve Data Lake Storage Synapse Pipelines Synapse Spark Capture Avro files Synapse Analytics Parquet files Bulk Imports Synapse Pipelines Other interfaces
Required Integration Capabilities Messaging Batch Events API Workflow Data
Where to use what?
What questions do people usually have? • Logic Apps vs Power Automate • Logic Apps vs Data Factory • Logic Apps vs Functions • Logic Apps Consumption vs Logic Apps Standard • Data Factory vs Synapse • Service Bus vs Event Hub • Service Bus vs Event Grid • Event Hub vs Event Grid • API Management vs Functions
Logic App vs Power Automate
Logic Apps vs Power Automate Logic Apps Power Automate Key Use Cases - System to System Workflow (EAI) - Process Automation - Automate myself - Human Workflow - Application Workflows - Dynamics / Power Platform - SharePoint - RPA (Power Automate Desktop) Key Differences - Tied to Azure pricing - Integrates with O365 - Pricing - Scalability - Visual Studio / VSCode / Portal - Tied to O365 and O365 / Power Platform licensing model - Integrates with Azure - Web UI Key Overlaps - They are both workflows - API Connectors - Runs on Logic Apps under the hood - API Connectors
Trigger Get Data Get Template Generate Document Send to Customer Power Automate Truck Integration with Dataverse and other systems Customer Document Generation Enterprise Integration Domain Customer Experience Platform Domain
Logic App vs Data Factory
Logic Apps vs Data Factory Logic Apps Data Factory Key Use Cases - System to System Workflow (EAI) - Process Automation - ETL / ELT Key Differences - Process messages - Can do batches but not excessively large - Complex orchestration - Processes large volumes of data - Simple workflow Key Overlaps - Data transformation - Connectors - Data Mapping - Connectors
Azure Data Factory Power BI Pipeline sFTP Server
Send Batch of Data Process Response Batch
Import Controller Logic App Trigger Extract Run Pipeline Data Factory Pipeline Query Dataset Transform Data Integration Account Load to SAP Daily Load to SAP Logic App
Logic App vs Functions
Logic Apps vs Functions Logic Apps Functions Key Use Cases - System to System Workflow (EAI) - Process Automation - Custom code to execute small functions - API Backend Key Differences - Visual designer - Long Running Processes - Cloud Connectors - Code based - Short running (note does have Durable Functions for longer running support) - Language Support eg: C#, Powershell, Python, etc Key Overlaps - Logic App Standard uses Functions Runtime under the hood - Logic App Standard Built In Connectors uses Functions Bindings - Functions Bindings
Loaded Railcar Message Convert Weights etc Enriched and formatted Message Archive Support User Manually Loaded railcars And for testing Loaded Railcar Data to CRM Loaded Railcar Data to SAP Loaded Railcar Data to Transport System Transport System Loaded Railcar Receiver Other interfaces Other interfaces Plant Data Receiver Conversion API
Logic App Standard vs Consumption
Logic Apps Consumption vs Standard Logic Apps Consumption Logic App Standard Key Use Cases - System to System Workflow (EAI) - Process Automation - System to System Workflow (EAI) - Process Automation Key Differences - Management Unit = 1 workflow - Per execution cost - Network connectivity via Data Gateway - Microsoft manage the host 100% - Can scale to zero - Management Unit = 1 App containing multiple workflows - CPU time-based cost model - Network connectivity via VNet integration - You manage the host 50% (like App Service) - Higher Performance - Built in connectors - Can be self hosted in func.exe - BizTalk like features such as Data Mapper, custom code support, etc Key Overlaps - The concept of what they do is the same - How they work under the hood is different and that affects when you would use them
Data Factory vs Synapse
Data Factory vs Synapse Synapse Data Factory Key Use Cases - ETL / ELT - Data Processing – Sparkpool - SQL on top of storage - ETL / ELT Key Differences - Synapse is like Data Factory plus everything else you need to make a data platform - Cosmos/SQL/Dataverse Link - Data Factory is a subset of Synapse Key Overlaps - ETL/ELT - ETL/ELT
Azure Data Factory Power BI Pipeline sFTP Server
SharePoint to Data Lake Sync Data Platform Synapse Data Warehouse Synapse Pipeline Power BI Landing Zone Standard Zone Synapse Pipeline Synapse Pipeline Line of Business Apps
Service Bus vs Event Hub
Service Bus vs Event Hub Service Bus Event Hub Key Use Cases - Pub/Sub - Durable Messaging - Event Stream Key Differences - Transactional Message Processing - Peek / Lock / Delete - Read / Delete - Pub/Sub - Each message is read and completed once - Re-read stream from point in time - Multiple concurrent readers - Capture Feature Key Overlaps - Sender / Receiver concept - Fan Out via Pub/Sub - Sender / Receiver concept - Fan Out via Consumer Groups https://www.mikestephenson.me/2015/03/03/azure-event- hubs-vs-azure-messaging/
Service Bus vs Event Hub Service Bus Event Hub
Service Bus vs Event Grid
Service Bus vs Event Grid Service Bus Event Grid Key Use Cases - Pub/Sub - Durable Messaging - “High value transactional messaging” - Event driven reactive programming model - Pub/Sub - “The state of something has changed, just letting you know if you want to do something about it” Key Differences - Message based - Transactional Message Processing - Peek / Lock / Delete - Read / Delete - Pub/Sub - Receiver pulls the message - Throttling and load levelling - Event based - Subscription can push a message to an endpoint - System Topic / Custom Topic / Event Grid Domains Key Overlaps - Sender / Receiver concept - Fan Out via Pub/Sub - Sender / Receiver concept - Fan Out via Consumer Groups
User uploads a file to a website Load file Logic App Web App writes the file to Blob storage Blob events fire to Event Grid Dataverse Read File Content Load Rows to Dataverse
Queue allows load levelling User uploads a file to a website Load file Logic App Web App writes the file to Blob storage Blob events fire to Event Grid Dataverse Read File Content Load Rows to Dataverse
Event Hub vs Event Grid
Event Hub vs Event Grid Event Grid Event Grid Key Use Cases - Event Stream - Event driven reactive programming model - Pub/Sub - “The state of something has changed, just letting you know if you want to do something about it” Key Differences - Re-read stream from point in time - Multiple concurrent readers - Capture Feature - Event based - Subscription can push a message to an endpoint - System Topic / Custom Topic / Event Grid Domains Key Overlaps - Sender / Receiver concept - Fan Out via Consumer Groups - Sender / Receiver concept - Fan Out via Consumer Groups
Data Platform Event Hub Capture Avro files Data Lake File create event Triggers Pipeline Railcar GPS API Dedicated Data Warehouse When will my railcars be back at the plant?
API Management vs Functions
API Management vs Functions API Management Functions Key Use Cases - Gateway to call HTTP based API’s - Centralized management of API use - Add additional features on top of your API - Security - Policy / Transform requests - Inbound, Outbound and Internal API’s - Good for helper API’s with light weight code - Light weight programming code for on demand execution Key Differences - VNet once for all API - Write transformation code as xml policy - VNet for every app (note differences for ASE) - Supports non HTTP based bindings - C#, Python, Java, Powershell, Javascript, etc Key Overlaps & Similarities - Serverless & Premium SKU’s - Network Integration - Serverless & Premium SKU’s - Network Integration
Data Platform Event Hub Capture Avro files Data Lake File create event Triggers Pipeline Railcar GPS API Dedicated Data Warehouse When will my railcars be back at the plant?
Loaded Railcar Message Convert Weights etc Enriched and formatted Message Send data to 3rd Party Loaded Railcar Receiver Plant Data Receiver Conversion API
Summary
ETL & ELT Security & Governance Enterprise Integration Platform Durable Messaging Data Transformation Helper Functions Storage Hybrid EAI API Enterprise Scenarios Business Rules System Workflow Human Workflow Business Users B2B Integration Management & Monitoring Power BI API Management Data Factory Key Vault Logic Apps Logic Apps Integration Account Integration Account Power Automate Azure Functions Service Bus Logic Apps Azure Functions Advisor Security Centre Storage Azure SQL DB Logic App Inline Code Data Gateway Power App VNet Integration Enterprise Connectors Azure Monitor Application Insights Application Configuration Cosmos DB Azure Functions Log Analytics Synapse Pipelines API Management
Questions