The document discusses the convergence of integration and application development. It notes that in the past, integrations and applications were developed separately by different teams. However, nowadays application developers regularly perform integration tasks like defining and exposing APIs when building applications. Integration capabilities have become part of the standard toolkit for application developers. The document argues this trend is resulting in a new generation of powerful applications that are integration-enabled.

1. The Convergence of Integration and Application Development
Kim Clark
Integration Architect
Nick Glowacki
Integration Specialist
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
Public webinar
http://ibm.biz/agile-integration-convergence
Slides
http://ibm.biz/agile-integration-convergence-pdf

2. Convergence of Integration and Application Development
Innovative applications today are rarely self contained. They are fundamentally
dependent on the ability to bring disparate data together in new and unique
ways. This means integration is at the core of all new applications.
In the past, the creation of integrations and applications have been different
disciplines. Nowadays, application developers regularly perform integration
when defining and exposing their own APIs and events. Integration capabilities
are now simply part of application developer's toolkit.
We discuss how this is resulting in a new generation of powerful integration-
enabled applications.
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

3. synchronous(rest/http,soap/http,…)
asynchronous(messaging,events,…)
High level integration reference architecture – circa 2016
Composing invocations
Surfacing events
Data integration
Low level connectivity
SoR SoR SoR
API & Event Management
API & Event Management
SoR SoR
Application
(traditional)
Application
(microservices)
{..} {..}
{..}{..}
{..} {..} {..}
Application
(microservices)
SoR
Application
(traditional)
{..} {..} {..}
Integration
bulk(bulkAPIs,filetransfer,…)
Core
Business
Operations
Empowering
Digital teams
Systems of
Engagement
Business logic
webinar http://ibm.biz/HybridIntRefArchYouTube paper http://ibm.biz/HybridIntRefArch
{..} {..}
{..}{..}
Integration
Integration
Mobile Partners SaaS
Offerings
API
Economy IoT
XaaS
On-Premise
Cloudaffinity
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

4. On premises
A target high level architecture for enterprise integration in 2020
Systems of
Record
Systems of
Engagement
Mobile Partners
API
Economy IoT
SaaS
SaaS
SaaS
SaaS
IaaS
PaaS
PaaS PaaS
SaaS
Cloud platform
• SaaS = Software as a service
• PaaS = Platform as a service
• IaaS = Infrastructure as a service
Ployglot (e.g. microservice application)
• Language runtimes
• Specialist runtimes (e.g. integration)
Exposure points
• APIs
• Events
Key features
• Technology layering has been removed.
• Grouping is based on business
domains/functions
• Applications are self-contained, including their
integration needs.
• Events have become more a first class citizen
rather than just a transport
• Well defined application boundaries, breached
only by governed APIs and Events
• Cloud platform type (IaaS/PaaS etc.) is hidden
when looking from the outside of an
application.
• Architecture balances isolation (decoupling),
and interaction (defined interfaces).
Caveats
• Represents an idealized target integration
architecture. Most organizations do not have
this level of clarity in the separations of their
applications.
• Describes how the things within the enterprise
communicate with one another. It does not
show the enterprise in the context of its broader
ecosystem (e.g. with partners).
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

5. Decentralized Ownership
Integration
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

6. In previous presentations we have looked at how integration has changed
from an integration specialist’s point of view
Webinars http://ibm.biz/agile-integration-webcasts eBook http://ibm.biz/agile-integration-ebook
Integration Integration
Socialization/monetization Re-platforming Application autonomy
API Management
APIM APIM
APIM
APIM
APIM APIM
Eventstream
Engagement
applications
Systemsof
record
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
Socialized APIs
Integration Integration Int.
Gateway API Management
Event
stream
API Management API Management
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

7. In this webinar we take the application developer’s point of view
Integration Integration
APIM
API Management
APIM APIM
APIM
APIM
Engagement
applications
Systemsof
record
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
Socialized APIs
Integration Integration Int.
APIM
Eventstreams
API Management
Enterprise services/APIs
Enterprise services/APIs Enterprise services/APIs
Enterprise services/APIs
Event
streams
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

8. In this webinar we take the application developer’s point of view
Integration Integration
Socialization/monetization Re-platforming Application autonomy
API Management
APIM APIM
APIM
APIM
APIM APIM
Eventstream
Engagement
applications
Systemsof
record
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
Socialized APIs
Integration Integration Int.
Gateway API Management
Event
stream
API Management API Management
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

9. Traditional monolithic engagement application
Integration
Engagement
applications
Systemsof
record
Integration
Gateway Application
Boundary
Application code
Application
Database
queues
JDBC JMS
SOAP/
HTTP
HTML/
HTTP
SOAP/
HTTP
JSON/
HTTP
MQ
JSON/
HTTP
other enterprise applications
Enterprise Service Bus
ETL
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

10. In this webinar we take the application developer’s point of view
Integration Integration
Socialization/monetization Re-platforming Application autonomy
API Management
APIM APIM
APIM
APIM
APIM APIM
Eventstream
Engagement
applications
Systemsof
record
Centralized
ESB
Fine-grained
integration
deployment
Decentralized
integration
ownership
Socialized APIs
Integration Integration Int.
Gateway API Management
Event
stream
API Management API Management
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

11. Microservice
component
Microservice
component
Microservice
component
Microservice
component
Microservice
component
Microservice
component
Modern cloud-native/microservice based application
MongoDB Redis
SaaS SaaS
Blockchain
replica
ML model
enterprise applications
Internet
Of
Things
SaaS
Application
Boundary
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

12. Recurring themes in the move to cloud-native
Smaller
More
independent
Lighter
Improved
time to
market
Reduced TCO
Increased
innovation
Business
Requirements
Technical
Implementation
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

13. Existing
Applications
Costofentryoffirstfunction
Increasing abstraction from infrastructure
Bare Metal
Virtual machines
Serverless?
Containers
New
Applications
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

14. What does it mean to write a cloud-native/microservice application?
Fine-grained
components
• Self-contained well decoupled
components of an
“appropriate” size (e.g.
microservices architecture)
Strong decoupling
• Clear ownership boundaries,
formalized API and event
interfaces, component
grouping. Independent
persistence.
Minimal state
• Components are clones,
and/or can re-assign/re-
balance work. No caller
affinity, no two phase commits
Immutable deployment
• Image based deployment and
updates,
• No live configuration.
• Rollback by rolling forward to
previous image.
Infrastructure as code
• Declarative infrastructure
configuration (GitOps) at the
level of individual components.
CI/CD
• Pipeline automation (build,
test, deploy, verify) for
creation and maintenance.
Agile methods
• Short, fixed length iteration
cycles. Intrinsic business
collaboration/feedback
DevOps
• Close collaboration between
development and operations
with rapid feedback resolution
Lightweight runtimes
• Fast start up/shut down. File-
system based install,
deployment, and
configuration.
Elastic, agnostic
platform
• Platform agnostically providing
deployment, scaling, HA,
security, monitoring...
Observability
• Platform neutral logging,
tracing to streams enabling
retrospective inferred analysis
of internal state.
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

15. Language
runtime
Language
runtime
Language
runtime
Language
runtime
Language
runtime
Language
runtime
Modern microservice based application – API management focus
MongoDB Redis
Blockchain
replica
ML model
Internet
Of
Things
Application
Boundary API gateway
API Portal
API Manager
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

16. Boundaries make complex
environments manageable
Managed API gateways define and enforce
application boundaries
µService
µService
µService
µService
µService
µService
µService
µService
µService
µService
µService
µService
µService
µService µService
µService
µServiceSilo
application
Silo
application
µService
µService
API
Application
boundary
Microservice
component
API Management
https://developer.ibm.com/apiconnect/2018/10/09/apis-microservices-defining-boundaries
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

17. Microservice
component
Exposed API
Application
boundary
A service mesh provides capabilities within the application boundary
API Management
Microservice
component
Microservice
component
Service mesh
proxy
Service mesh
routed requests
Traditional
Application
Microservice
Application
Microservice
Application
API management
gateway
Security
– Authentication,
authorization, simplified
mTLS, whitelisted egress
Deployment patterns
– A/B, canary
Fault tolerance
– Retries, circuit breaker,
rate-limiting
Visibility
– Logs, Metrics, tracing
Testing
– Fault injection
The service mesh intercepts invocations, enabling policy-based interconnectivity patterns to reduce code, and simplify the
implementation of security, version management, monitoring, diagnostics, testing and more within the microservices application.
https://developer.ibm.com/apiconnect/2018/11/13/service-mesh-vs-api-managementIBM Cloud / 28th May 2020 / © 2020 IBM Corporation

18. Collaboration between API management and the service mesh
API Management
v1 v2
MicroserviceApplication
Route 1% of requests from beta and test
consumers only to v2
Beta plan
consumers
Gold plan
consumers
Consumers are known to the API
management gateway by metadata
such as their application key, and their
subscription plan.
Proxies extract the header information
making it available to policies to route
based on API consumer meta-data
Invocation header is extended during
ingress with information including the
consumers key and plan
Traditional
Application
Limit all non-gold plan consumers to 10
requests per second to the back end system
Enable end to end tracing for a particular
consumer key for specific diagnostics
Silver plan
consumers
Test plan
consumers
Inject simulated fault responses for test
consumers only.
Proxies enact policies for routing, security,
tracing, fault injection, rate limiting etc.
How does it work?
What could it do?
https://developer.ibm.com/apiconnect/2018/11/13/service-mesh-vs-api-managementIBM Cloud / 28th May 2020 / © 2020 IBM Corporation

19. Scope really matters!
Microservice EnterpriseApplication? Domain?
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

20. Microservice
component
Microservice
component
Microservice
component
Microservice
component
Microservice
component
Microservice
component
Events within and beyond the application boundary
MongoDB Redis
SaaS SaaS
Blockchain
replica
ML model
enterprise applications
Internet
Of
Things
SaaS
Application
Boundary API Management
Event
broker
Event Management?
Event broker
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

21. Event driven architecture
patterns that build on one another
Event stream data distribution: Event distribution with history
Event stream projections: Consumer specific data views
Event sourcing: Using an event log as a data master
CQRS: Command Query Responsibility Segregation
Event processing: Evaluating events over time
Saga: Combining multiple actions together
Supporting/related patterns:
change data capture, event connectors (source/sink), function as a service…
https://ibm-cloud-architecture.github.io/refarch-edaIBM Cloud / 28th May 2020 / © 2020 IBM Corporation

22. Event Log
Event streams for data distribution
External
website
application
Partner
website
application
Mobile
application
Subscribe
and
consume
Event stream
Back end
datastore
… 4 5 6 7 …
Subscribe
and
consume
Subscribe
and
consume
Publish
(change events)
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

23. Event Log (Topic A)
Event processing
Event
Event stream
Event
Propagator
… 42 43 44 45 46
Publish
… 4 5 6 7 8 … 172 173 174 175
Event Log (Topic B)
… 4 5 6 7
Event
processor
… 44 456 7 8 46
Event pattern matched
Subscribe
and
consume
Publish
4
5
Event processing could involve:
• Filtering
• Redaction
• Routing
• Event pattern matching
• Real-time analytics
• Machine learning model*
(training and/or using)
* Note that event processing can be
re-run over the event stream
history in order to gain new insights
as the learning improves
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

24. Event Log
Event stream projections
Local read-optimized datastore
External
website
application
Partner
website
application
Mobile
application
Local datastore Local datastore Local datastore
Subscribe
and
consume
Direct invocation
Event stream
Back end
datastore
… 4 5 6 7 …
Subscribe
and
consume
Subscribe
and
consume
Publish
(change events)
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

25. Event sourcing
Local “event sourced”
read-optimized datastore
Read
implementation
A
Read
implementation
B
Local datastore Local datastore
Subscribe
and
consume
Direct invocation
Event streamSubscribe
and
consume
“Event Sourced” implies the use of an
event log as the primary datastore
Application/Context
boundary
Event Log
… 4 5 6 7 …
Command
implementation
COMMAND
QUERY
Exposed API
, and CQRS
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

26. Event sourcing/CQRS within a bounded context
vs (inter) domain events
CQRS and Event Sourcing patterns are typically scoped to
within a bounded context. From outside the context, the event
sourced nature of the implementation should be hidden.
Bounded Context
Event Log
COMMAND
QUERY
Bounded Context
Only selected events (and for that matter APIs) should
be exposed across bounded contexts. Their data models
should be decoupled from the internal model.
Internal
events
(inter) domain events
API & Event Management
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

27. CQRS at different scopes
These two diagrams represent the same logical pattern,
but have radically different implementations
Logical component
boundary
Event
Log
Event
Processing
Command
Store
Command
Processing
API Management
API
In an enterprise scope, the same pattern often exists
but is broken up across multiple different sub-systems,
integration specialists would recognize these as
enterprise integration patterns. Is this still CQRS?
Search
& Read
Event
Log
Read
Datastore
Event
Processing
Command
Processing
APIM
API
Datastore
Events
Command
Store
APIM
API
Command
Forwarding
API
In an application scope, application developers would
consider this pattern part of their application design
(and perhaps call it CQRS).
APIM
Search
& Read
Read
Datastore
Consumer
specific
projection
(event-fed read-
optimized view)
Consumer specific
store-forward
pattern to improve
availability and
performance of
updates
Existing back end
system exposed as
managed APIs.
Fire-forget updates
Enabled to emit
data change
events
Internal
events Cross
boundary
events
Cross
boundary
API
Event
Processing
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

28. Saga Pattern: Orchestration vs Choreography
Orchestrated interactions (orchestration)
Pros
Mature products available (e.g. BPM)
First class notion of “process”
Modelled process visible in implementation
Monitoring/reporting at process level
Process version migration support
Clearer modelling of exception paths
Easier to introduce human interaction
Cons
Higher CPU/network due to persistence
Solution will be split between business process
integration layers
Chained interactions (choreography)
Pros
Higher performance due to minimal persistence
Can be implemented entirely in the integration layer
Cons
Largely a custom solution
No notion of “process”, only the individual steps
Process “model” not visible in implementation
Custom work for effective monitoring/reporting
Exception paths harder to understand
Custom work to migrate to new processes
Custom work to introduce human interaction
Queued events holding stateOrchestration state persistence
https://microservices.io/patterns/data/saga.html
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

29. Language
runtime
Language
runtime
Language
runtime
Language
runtime
Language
runtime
Language
runtime
Modern microservice based application – integration perspective
MongoDB Redis
SaaS SaaS
Blockchain
replica
ML model
API management
enterprise applications
Internet
Of
Things
SaaS
Queues
Application
Boundary
Integration?
Integration?
Event
broker
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

30. Myth
Busting
Myth Reality
The need for integration
has reduced
More so than ever before new innovations are based on unique use of
data and function from multiple sources. e.g. IoT, SaaS Applications, AI,
social media, and a vast number of existing systems etc.
Everything is accessible
via an API
Most datastores have propriety libraries. APIs, if available, are often not
the preferred mechanism. Many systems now also expose data via events
using a variety of protocols (e.g. Kafka, MQTT).
It is easy to integrate
with APIs
APIs are only as simple as the data model behind them, which is typically
pretty complex. Furthermore, once past the simpler agreed conventions,
there are many ways to represent the same data with REST. This is
compounded by the variety of security and access control models in play.
APIs are all REST based
API is a broad term, and REST is just one implementation, although the
one typically assumed. There is an increasing use of alternatives to REST
such as GraphQL. Many enterprise systems have been exposed as web
services, and there may be little appetite to refactor to REST. Many
organizations consider IBM MQ to be one of their standard protocols for
application interactions. And there’s more: gRPC, Apache Thrift…
Microservices talk to
each other over APIs
Many do, but good microservice design means decoupled components,
which in turn encourages event sourced patterns rather than real time
API integration.
In short, modern
applications contain
more “integration”
than ever before.
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

31. Common (microservice) integration patterns
… 4 5 … 4 5
Engagement
applications
Systems of
record
Synchronous
cross-component
composition
Synchronous
local
composition
Synchronous
local access
Synchronous
remote
access
Event stream
projection
Event driven
composition
… 4 5
Ownership
boundary
Microservice component
(dark blue for proportion
of “integration”)
System of
Record
Local
datastore
Exposed
API
Exposed API
(in another
boundary)
Topic on Event
Stream/Queue
Key
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

32. On premises
Systems of
Record
Systems of
Engagement
Mobile Partners
API
Economy IoT
SaaS
SaaS
SaaS
SaaS
IaaS
PaaS
PaaS PaaS
SaaS
Cloud platform
• SaaS = Software as a service
• PaaS = Platform as a service
• IaaS = Infrastructure as a service
Ployglot (e.g. microservice application)
• Language runtimes
• Specialist runtimes (e.g. integration)
Exposure points
• APIs
• Events
Meeting the broader needs through cloud-native approach to application and integration
Technical Implementation
Smaller
Lighter
More independent
Business Requirements
Improve timed to market
Increased innovation
Reduced TCO
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

33. IBM can help you modernize
with an Agile integration
strategy.
Agile
Integration….
…to achieve
development,
deployment, and
operational
agility
People & Process
• Decentralized
ownership
• Empowering teams
• Agile methods
Architecture
• Fine-grained
deployment
• API led
• Event-driven
• Microservices aligned
• Highly scalable
Technology
• Cloud-native
infrastructure
• Essential integration
capabilities
• Unified security,
governance, and
operations
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation

34. Learn more
IBM Cloud / © 2020 IBM Corporation
Agile Integration
ibm.com/cloud/integration/agile-integration

35. Thank you!
IBM Cloud / 28th May 2020 / © 2020 IBM Corporation