I did a session at the LeedsSharp about azure integration services and where to use which technology - Azure Functions - Logic Apps - Synapse - Data Factory - Event Grid - Service Bus - Event Hub - API Management

1. Azure Integration
Services – Where
should I use what?

2. Azure Integration
Services – Where
should I use what?

3. About me
• Microsoft MVP 14 years
• Freelance Cloud Architect
• Based in Newcastle, UK
• https://mikestephenson.me/
• @Michael_Stephen
• Product Advisor for Serverless360

4. What have we been up to
Business
• Dozens of projects
• Multi-Year investment in Microsoft
IPaaS
• EAI Platform = Microsoft iPaaS
• EDMS Platform = Microsoft Data
Platform
• SAP / Power Platform / Power BI /
Workday / IBM Maximo /
SuccessFactors
Technical
• Environments - 5
• Resources (Across all environments)
• Logic Apps – 1895
• Function Apps – 116
• Integration Account – 5
• APIM – 5
• Service Bus
• Event Grid
• Event Hub
• Synapse

5. “What is Azure Integration Services and why should I
care”?
“Where do I use what”?

6. History of Microsoft Integration – Way back!
BizTalk SSIS Custom
- Hard learning curve
- Specific skill set
- Messaging / Orchestration / ETL
based
- Complex to implement
- DBA driven
- Specific skill set
- ETL
- Most orgs had SQL
- .net or script based
- Maybe API’s or WCF

7. History of Integration Trends from yester-year
iPaaS has seen customers transition to getting stuff done!

8. ETL & ELT Security & Governance
Enterprise Integration Platform
Durable Messaging
Data Transformation
Helper Functions Storage Hybrid
EAI API
Enterprise Scenarios
Business Rules System Workflow Human Workflow Business Users B2B Integration
Management &
Monitoring
Power BI
API
Management
Data Factory Key Vault
Logic Apps
Logic Apps
Integration
Account
Integration
Account
Power
Automate
Azure
Functions
Service Bus
Logic Apps
Azure
Functions
Advisor
Security
Centre
Storage
Azure SQL DB
Logic App
Inline Code
Data Gateway
Power App
VNet
Integration
Enterprise
Connectors
Azure Monitor
Application
Insights
Application
Configuration Cosmos DB
Azure
Functions
Log Analytics
Synapse
Pipelines
API
Management

9. Usage Scenarios
BizTalk Migration SSIS Migration Custom Developed Integration
IT Admin Automation User Automation Application Developers
Dynamics / Power Platform
Developers
SharePoint Developers Citizen Developers
Common
Skillsets
Share resources /
Team Members

10. The challenges are different
• I want a product not a platform!
• Azure is like Lego how do I know which bricks to use?
• Instead of 1 or 2 really challenging skillsets there are now LOTS of
skillsets but they are easier to learn

11. What questions do people usually have?
• Logic Apps vs Power Automate
• Logic Apps vs Data Factory
• Logic Apps vs Functions
• Logic Apps Consumption vs Logic Apps Standard
• Data Factory vs Synapse
• Service Bus vs Event Hub
• Service Bus vs Event Grid
• Event Hub vs Event Grid
• API Management vs Functions

12. Secret Sauce – Effectively Combining Technologies
Combine Examples
Logic Apps + Power Automate System Workflow + Human Workflow for approvals
Logic Apps + Data Factory Trigger ETL processes as part of an automation
Synapse Pipeline + Logic App Simple SharePoint integration to publish data to users
Logic Apps + Functions Custom code to support my orchestration
API Management + Functions + Logic Apps Centralized management of API’s used by my integration
platform
Service Bus + Logic Apps Load levelling and pub/sub messaging for complex integrations
Event Hub + Functions High scale processing of event data

13. Real-world Journey of Patterns

14. Step 1 – Tactical Point to Point Logic Apps
Trigger
Validate
Message
Get Existing
Data
Do some
stuff
Do some
other stuff
Order Updates
Do more
stuff
Create
record
Create
other
record
Return
Response

15. What happened
Things that weren’t working:
- Doing too much in Logic Apps
- Rebuild & Repeat rather than Reuse
& Extend
- The implementation was getting
complex

16. Step 2 – Enter Service Bus, API Management & Functions
Loaded Railcar Message
Conversion API
Convert Weights etc
Enriched and
formatted
Message
Archive
Support User
Manually Loaded railcars
And for testing
Loaded Railcar
Data to CRM
Loaded Railcar
Data to SAP
Loaded Railcar Data
to Transport System Transport System
Loaded Railcar Receiver
Other
interfaces
Other
interfaces
Plant Data
Receiver

17. Step 3 – Add Publish data to Data Lake
Loaded Railcar Message
Conversion API
Convert Weights etc
Enriched and
formatted
Message
Synapse
Data Platform
Archive
Support User
Power App
Power BI
Manually Loaded railcars
And for testing
Loaded Railcar
Data to CRM
Loaded Railcar
Data to SAP
Loaded Railcar Data
to Transport System
Loaded Railcar Data
to Synapse
Transport System
Loaded Railcar Receiver
Other
interfaces
Other
interfaces
Plant Data
Receiver

18. Step 4 – Add Synapse to build a Data Platform
Integration
Platform
EAI Transactions to
EDMS
EDMS Platform
Event Hub
Capture Avro files
Data Lake
Synapse Analytics
Synapse Pipelines

19. Import
Step 4 – Start Building out Data Platform with Synapse
Data Platform
EAI to EDMS Functions
Landing Transform &
Orchestrate
Serve
Data Lake
Storage
Synapse Pipelines
Synapse Spark
Capture Avro
files
Synapse Analytics
Power App
EAI
Transactions
Parquet files
Bulk Imports
Enterprise Reports
Power BI
Experimentation
Power BI

20. Step 5 – Use the Data Platform to Help your EAI
Import
Data Platform
Landing Transform &
Orchestrate
Serve
Data Lake
Storage
Synapse Pipelines
Synapse Spark
Capture Avro
files
Synapse Analytics
Parquet files
Bulk Imports
Synapse Pipelines
Other interfaces

21. Required
Integration
Capabilities
Messaging
Batch
Events API
Workflow
Data

22. Where to use what?

23. What questions do people usually have?
• Logic Apps vs Power Automate
• Logic Apps vs Data Factory
• Logic Apps vs Functions
• Logic Apps Consumption vs Logic Apps Standard
• Data Factory vs Synapse
• Service Bus vs Event Hub
• Service Bus vs Event Grid
• Event Hub vs Event Grid
• API Management vs Functions

24. Logic App vs Power Automate

25. Logic Apps vs Power Automate
Logic Apps Power Automate
Key Use Cases - System to System Workflow (EAI)
- Process Automation
- Automate myself
- Human Workflow
- Application Workflows
- Dynamics / Power Platform
- SharePoint
- RPA (Power Automate Desktop)
Key Differences - Tied to Azure pricing
- Integrates with O365
- Pricing
- Scalability
- Visual Studio / VSCode / Portal
- Tied to O365 and O365 / Power Platform licensing model
- Integrates with Azure
- Web UI
Key Overlaps - They are both workflows
- API Connectors
- Runs on Logic Apps under the hood
- API Connectors

26. Trigger Get Data
Get
Template
Generate
Document
Send to
Customer
Power Automate
Truck Integration with Dataverse
and other systems
Customer Document Generation
Enterprise Integration Domain Customer Experience Platform
Domain

27. Logic App vs Data Factory

28. Logic Apps vs Data Factory
Logic Apps Data Factory
Key Use Cases - System to System Workflow (EAI)
- Process Automation
- ETL / ELT
Key Differences - Process messages
- Can do batches but not excessively large
- Complex orchestration
- Processes large volumes of data
- Simple workflow
Key Overlaps - Data transformation
- Connectors
- Data Mapping
- Connectors

29. Azure Data Factory
Power BI
Pipeline
sFTP Server

30. Send Batch of Data
Process Response Batch

31. Import Controller Logic App
Trigger
Extract
Run
Pipeline
Data Factory
Pipeline
Query
Dataset
Transform
Data
Integration Account
Load to SAP
Daily Load to SAP Logic App

32. Logic App vs Functions

33. Logic Apps vs Functions
Logic Apps Functions
Key Use Cases - System to System Workflow (EAI)
- Process Automation
- Custom code to execute small functions
- API Backend
Key Differences - Visual designer
- Long Running Processes
- Cloud Connectors
- Code based
- Short running (note does have Durable Functions for longer
running support)
- Language Support eg: C#, Powershell, Python, etc
Key Overlaps - Logic App Standard uses Functions Runtime
under the hood
- Logic App Standard Built In Connectors uses
Functions Bindings
- Functions Bindings

34. Loaded Railcar Message
Convert Weights etc
Enriched and
formatted
Message
Archive
Support User
Manually Loaded railcars
And for testing
Loaded Railcar
Data to CRM
Loaded Railcar
Data to SAP
Loaded Railcar Data
to Transport System Transport System
Loaded Railcar Receiver
Other
interfaces
Other
interfaces
Plant Data
Receiver
Conversion API

35. Logic App Standard vs
Consumption

36. Logic Apps Consumption vs Standard
Logic Apps Consumption Logic App Standard
Key Use Cases - System to System Workflow (EAI)
- Process Automation
- System to System Workflow (EAI)
- Process Automation
Key Differences - Management Unit = 1 workflow
- Per execution cost
- Network connectivity via Data Gateway
- Microsoft manage the host 100%
- Can scale to zero
- Management Unit = 1 App containing multiple workflows
- CPU time-based cost model
- Network connectivity via VNet integration
- You manage the host 50% (like App Service)
- Higher Performance
- Built in connectors
- Can be self hosted in func.exe
- BizTalk like features such as Data Mapper, custom code
support, etc
Key Overlaps - The concept of what they do is the same - How they work under the hood is different and that affects
when you would use them

37. Data Factory vs Synapse

38. Data Factory vs Synapse
Synapse Data Factory
Key Use Cases - ETL / ELT
- Data Processing – Sparkpool
- SQL on top of storage
- ETL / ELT
Key Differences - Synapse is like Data Factory plus everything else
you need to make a data platform
- Cosmos/SQL/Dataverse Link
- Data Factory is a subset of Synapse
Key Overlaps - ETL/ELT - ETL/ELT

39. Azure Data Factory
Power BI
Pipeline
sFTP Server

40. SharePoint to Data
Lake Sync
Data Platform
Synapse
Data Warehouse
Synapse Pipeline
Power BI
Landing Zone Standard Zone
Synapse Pipeline
Synapse Pipeline
Line of Business Apps

41. Service Bus vs Event Hub

42. Service Bus vs Event Hub
Service Bus Event Hub
Key Use Cases - Pub/Sub
- Durable Messaging
- Event Stream
Key Differences - Transactional Message Processing
- Peek / Lock / Delete
- Read / Delete
- Pub/Sub
- Each message is read and completed once
- Re-read stream from point in time
- Multiple concurrent readers
- Capture Feature
Key Overlaps - Sender / Receiver concept
- Fan Out via Pub/Sub
- Sender / Receiver concept
- Fan Out via Consumer Groups
https://www.mikestephenson.me/2015/03/03/azure-event-
hubs-vs-azure-messaging/

43. Service Bus vs Event Hub
Service Bus Event Hub

44. Service Bus vs Event Grid

45. Service Bus vs Event Grid
Service Bus Event Grid
Key Use Cases - Pub/Sub
- Durable Messaging
- “High value transactional messaging”
- Event driven reactive programming model
- Pub/Sub
- “The state of something has changed, just letting you know if
you want to do something about it”
Key Differences - Message based
- Transactional Message Processing
- Peek / Lock / Delete
- Read / Delete
- Pub/Sub
- Receiver pulls the message
- Throttling and load levelling
- Event based
- Subscription can push a message to an endpoint
- System Topic / Custom Topic / Event Grid Domains
Key Overlaps - Sender / Receiver concept
- Fan Out via Pub/Sub
- Sender / Receiver concept
- Fan Out via Consumer Groups

46. User uploads a file to a website
Load file Logic App
Web App writes the file to
Blob storage Blob events fire to Event Grid
Dataverse
Read File Content
Load Rows to Dataverse

47. Queue allows load levelling
User uploads a file to a website
Load file Logic App
Web App writes the file to
Blob storage Blob events fire to Event Grid
Dataverse
Read File Content
Load Rows to Dataverse

48. Event Hub vs Event Grid

49. Event Hub vs Event Grid
Event Grid Event Grid
Key Use Cases - Event Stream - Event driven reactive programming model
- Pub/Sub
- “The state of something has changed, just letting you know if
you want to do something about it”
Key Differences - Re-read stream from point in time
- Multiple concurrent readers
- Capture Feature
- Event based
- Subscription can push a message to an endpoint
- System Topic / Custom Topic / Event Grid Domains
Key Overlaps - Sender / Receiver concept
- Fan Out via Consumer Groups
- Sender / Receiver concept
- Fan Out via Consumer Groups

50. Data Platform
Event Hub
Capture Avro files
Data Lake
File create event
Triggers Pipeline
Railcar GPS API
Dedicated Data
Warehouse
When will my
railcars be back at
the plant?

51. API Management vs Functions

52. API Management vs Functions
API Management Functions
Key Use Cases - Gateway to call HTTP based API’s
- Centralized management of API use
- Add additional features on top of your API
- Security
- Policy / Transform requests
- Inbound, Outbound and Internal API’s
- Good for helper API’s with light weight code
- Light weight programming code for on demand execution
Key Differences - VNet once for all API
- Write transformation code as xml policy
- VNet for every app (note differences for ASE)
- Supports non HTTP based bindings
- C#, Python, Java, Powershell, Javascript, etc
Key Overlaps &
Similarities
- Serverless & Premium SKU’s
- Network Integration
- Serverless & Premium SKU’s
- Network Integration

53. Data Platform
Event Hub
Capture Avro files
Data Lake
File create event
Triggers Pipeline
Railcar GPS API
Dedicated Data
Warehouse
When will my
railcars be back at
the plant?

54. Loaded Railcar Message
Convert Weights etc
Enriched and
formatted
Message
Send data to 3rd
Party
Loaded Railcar Receiver
Plant Data
Receiver
Conversion API

55. Summary

56. ETL & ELT Security & Governance
Enterprise Integration Platform
Durable Messaging
Data Transformation
Helper Functions Storage Hybrid
EAI API
Enterprise Scenarios
Business Rules System Workflow Human Workflow Business Users B2B Integration
Management &
Monitoring
Power BI
API
Management
Data Factory Key Vault
Logic Apps
Logic Apps
Integration
Account
Integration
Account
Power
Automate
Azure
Functions
Service Bus
Logic Apps
Azure
Functions
Advisor
Security
Centre
Storage
Azure SQL DB
Logic App
Inline Code
Data Gateway
Power App
VNet
Integration
Enterprise
Connectors
Azure Monitor
Application
Insights
Application
Configuration Cosmos DB
Azure
Functions
Log Analytics
Synapse
Pipelines
API
Management

57. Questions