This talk, presented at tid-x 2019 https://www.tid-x.com/, summarises the evolution of integration architectures in the enterprise . It introduces basic concepts, paradigms and architectures, from messaging oriented middleware based architectures to more modern data services based integrations.

1. From MoMs to DaDS
The Evolution of Enterprise Application Integration

2. Enterprise IT evolution: Breaking Monoliths

3. Breaking Monoliths: The early years 80´s - 90´s
Mainframe
Prov
Billing
ERP
CRM
HR
1 big System
Centralized Logic&Data
System of Systems
Distributed Logic&Data

4. Breaking Monoliths: from 2012... the history repeats
Application
Backend
Monolith Microservices
What is Next?: lambda functions (?)

5. Enterprise App Integration is about making
different apps and systems collaborate for...
Business Processes Realization ⇢ Systems/Apps integration
User Interface Realization ⇢ Frontends to Backends integration
Data Processing (BI, Analytics) ⇢ Data Sharing between systems

6. O(n )
The Challenge: Avoid the point to point hell
Mainframe
Prov
Billing
ERP
CRM
HR
Mainframe
Prov
Billing
ERP
CRM
HR
n*(n-1)/2 n
2 O(n)
EAI
Tool / Suite

7. The Reality: Integration paradigms and technologies evolve, but
usually never replace completely the previous ones
Mainframe
ERP
DWH
MOM
Prov1
HR
ESB
Terminal
www1
CRM
API GW
BFF
Prov2
www2
Billing2
Data
ServicesBilling1
µ - services API GW
MobileApp1
MobileApp2
DFS
ETL
ETL
Data Platform
1
Data
Platform2
NFS
Data Query GW
www3
Prov2
Sync

8. Enterprise IT can be like urbanisation “out of control”
Aerial view of urban sprawl and smog November 6, 2013 in Mexico City, Mexico. Photograph: Richard Ellis/Alamy

9. Addressing the Challenge: Integration at a Glance
Integration Mechanisms
Basic Decoupling Strategies
Integration Architectures
clarity in the
industry
+
-

10. Integration at a Glance
Integration Mechanisms
Basic Decoupling Strategies
Integration Architectures/Suites
Interconnecting parts for Exchanging Data and Messages

11. File Transfer
System A System B
File
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

12. Shared Storage
System A System B
Database or FileSystem
System CA
B
C
read() / write ()
query()
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

13. RPC : Remote Procedure Call
System A System B
Request // invoke
Response // return
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

14. Messaging Channel
(async) Messaging
Messaging Channel
Producer Consumer
Async Message
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

15. Messaging Channel
Producer
Consumer Consumer Consumer
Pub/Sub Channel (Topic)
1:N
Producer
Consumer
Consumer
Conusmer
Point To Point Channel (Queue)
1:1
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

16. MOM: Messaging Oriented Middleware
(the software that provides async messaging services)
Topics
Queues
Producer Conusmer
MOM
API API
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

17. Streams
Producer
Consumer Consumer
(Event Logs/ infinite append only file)
0 1 2 3 end
read(from:end)
write(reccord)==append(reccord)
read(from:0)
....
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

18. Stream vs Topic Channel
Producer
Consumer1
:
client
0 1 2 3 end
read(from:end)
append(reccord)
....
Consumer2
:
client
read(from:end)
Producer
Consumer1
:
server
Consumer2
:
server
onMsgCallback(msg)
publish(msg)
subscribe(consumer1)
subscribe(consumer2)
stream:server
topic:client
read(from:offset) == video streaming
read(from:end) == live streaming
(tail-f)
subscribe(consumer) == radio station(topic) tuning
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

19. Object Storage
Producer Consumer
(Object=File+Metadata)
metadata
File
put(metadata, file) get(object id) :file
Bucket
Object
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

20. DaDS : Data as Data Services
APIAPI
Buckets Service Streams Service
Producer
Producer
Consumer
Consumer
Consumer
put(meta, file)
get(id)
append(reccord)
read(from:offset):[]reccord
Integration Mechanisms
Decoupling Strategies
Integration
Architectures/Suites

21. Integration at a Glance
Integration Mechanisms
Basic Decoupling Strategies
Integration Architectures
Avoiding point2point coupling

22. Avoiding Coupling: Adapter
System A System B
Adapter Adapter
A
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures
Decouple Systems from Integration Mechanism
Adapter ≣ Connector ≣ Extractor/Loader
B

23. Dealing with Coupling: Transformers & Routers
System A System B
Adapter Adapter
Transformer
A2B
Typical ETL ....Is it Point2Point coupling Again?
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures
A B

24. Avoiding Coupling: Canonical Model
System A
System B
AdapterA
AdapterB
Transformer
A2C
Transformer
C2B
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures
Canonical Message
System D
AdapterDTransformer
C2D
A
B
D
C

25. Integration at a Glance
Integration Mechanisms
Basic Decoupling Strategies
Integration Architectures

26. Integration Architectures : Mediators
Simplifies communication among objects in a system by
introducing a single object that manages message distribution
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

27. Integration Architectures: 2 Type of Mediators
Interaction -Centric Data -Centric
Write/Read DataSend/Receive Messages

28. Integration Architectures :
Interaction Centric Mediators
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

29. EMS: EAI Messaging Stacks
Mainframe
Prov
Billing
ERP
CRM
HR
EAI
Messaging
Stack
Async Messaging based mediation
Messages
Messages
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures
Messages
Messages Messages
Messages

30. EMS: EAI Messaging Stacks
MOM
Adapter
Automated Process Engine
Transformers / Routers
Workflow Engine
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures
Sometimes merged into the
same product

31. ESB: Enterprise Service Bus
Mainframe
Prov
Billing
ERP
CRM
HR
ESB
Enterprise
Sevice
Bus
Mediated RPCs, FileTransfers, Messages in a SOA
SOAP
Messages
FileTranfer REST
Messages
SOAP
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

32. What is an Enterprise Service Bus?
MoMWeb Services Protocols
Transformation & Routing
Traffic Management
Business Processes Orchestration
Connectors
Transaction Management
Business Application Monitoring
Business Schemas Management
“Anything” you need to mediate your systems and Services?
etc.....
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

33. BFF: Backend For Frontend
WWW
BFF
ERP
CRM
HR
Mobile
BFF
Billing
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

34. API Gateway
WWW
ERP
CRM
HR
Mobile
Billing
API GW
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

35. API Gateway and MicroServices
WWW
ERP
CRM
HR
Mobile
Billing
API GW
µSrvs API GW
Micro-services
Mediation
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

36. MicroServices and Service Mesh
µSrvs API GW
Micro-services Mediation
Service Mesh Proxy
ControlPlane&
TransversalServices
Example: Istio
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

37. Integration Architectures :
Data Centric Mediators
Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures

38. Data Query Gateway Integration Mechanisms
Basic Decoupling
Strategies
Integration
Architectures
Data Query Gateway
A
DB
Data Model A
Data Model B
Data Model C
Data Model D
Data Model ∅
µSrvs API GW
API
API
Mobile
Api == DataModel.query()
Examples: GraphQL, Falcor...
Joins,
Caching,
Data routing

39. Data lake based on DFS
Transformer
/
Processor
System A
System B
Extractor
Loader
Loader
DataPlatform
● Data and Processing Coupled (i.e HDFS)
● Governance of Folders and Files (?)
● Metadata (?)
Datalake
DFS

40. API
Data lake based on Data Services
API
Buckets Service Streams Service
Transformer
/
Processor
System A
System B
Adapter
Adapter
Adapter
DataPlatform
Transformer
/
Processor
● Decoupled Data and
Processing
● Governance and Metadata are
intrinsic
○ Service IAM
○ Policies
○ Schemas

41. A (personal) Vision - FA: Functional Architecture
Commands GWQuery GW
App
Data Model ∅
Materialized
ViewsSystem A
AdapterA
AdapterB
System B
AdapterC
System C
Commands stream
Events
stream

42. Remember...you are not likely to replace completely the
previous systems and architectures. Evolve from/with them

43. Q & A