Enterprise Integration Patterns Revisited (again) for the Era of Big Data, Internet of Things and Microservices

Enterprise Integration Patterns
Revisited for an Era of Big Data and Internet of Things
Kai Wähner
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn / Xing à Please connect!

Key Take-Aways
–  Integration is the key factor for success of business in the future!
–  Enterprise Integration Patterns are used everywhere!
–  Real time integration is the game changer in a Big Data and IoT world!

Agenda
Ø  Application Integration
Ø  Enterprise Integration Patterns
Ø  Modeling
Ø  Frameworks and Tools
Ø  Big Data, Internet of Things, Microservices
Ø  Future

Growing IT Infrastructure Everywhere
•  Applications
•  Interfaces
•  Technologies
•  Products
•  Cloud
•  Mobile
•  Internet of Things

Heterogeneity
Host à Client / Server à SOA à Cloud à Big Data à IoT à ...

Spaghetti Solutions
„point-to-point hell“

The Solution: Application Integration
All roads lead
to Rome ...

AmazonS3
s3
=
new
AmazonS3Client(new
Proper6esCreden6als(

S3Sample.class.getResourceAsStream("AwsCreden6als.proper6es")));

String
bucketName
=
"my-‐ﬁrst-‐s3-‐bucket-‐"
+
UUID.randomUUID();

String
key
=
"MyObjectKey";

try
{

s3.createBucket(bucketName);

s3.putObject(new
PutObjectRequest(bucketName,
key,
createSampleFile()));

S3Object
object
=
s3.getObject(new
GetObjectRequest(bucketName,
key));

ObjectLis6ng
objectLis6ng
=
s3.listObjects(new
ListObjectsRequest()

.withBucketName(bucketName)

.withPreﬁx("My"));

s3.deleteObject(bucketName,
key);

s3.deleteBucket(bucketName);

}
catch
(AmazonServiceExcep6on
ase)
{

//
error
handling...

}
catch
(AmazonClientExcep6on
ace)
{

//
error
handling...

}

Writing a lot of Glue Code...

... vs. using Patterns, Frameworks, Tools
•  Standardized Modeling
•  Efficient Realization
•  Automatic Testing

“A software design pattern, in the sense of a template,
is a general solution to a problem in programming. A
design pattern provides a reusable architectural outline
that may speed the development of many computer
programs.”
http://en.wikipedia.org/wiki/Design_pattern
What is a Design Pattern?

Enterprise Integration Patterns (EIP)

“The goal of EIPs is to document
technology-independent design
guidance that helps developers and
architects describe and develop
robust INTEGRATION solutions.”
http://www.eaipatterns.com
Enterprise Integration Patterns (EIP)

History of EIPs
•  Talk from Gregor Hohpe
–  WSO2 Con 2011
–  „History, present, future of EIP“
–  A lot about past (ideas, creation of
book, ...)
–  http://www.youtube.com/watch?
v=Xwi1DU6KoQ4
•  Today‘s Talk
–  Not too much about history
–  Just about present and future

h"p://www.eaipa"erns.com

EIP Example: Splitter


EIP Example: Aggregator

Combining EIPs shows their true power!
A + B + C = ABC


EIP Example: Composed Message Processor

Status Quo
•  Enterprise Integration Patterns
do not change
•  Just technologies change
•  Implementations for EIPs are
already available

Modeling Tool: Microsoft Visio (Windows)

Modeling Tool: Omnigrafﬂe (Mac OS X)

Complexity
of Integration
Integra6on

Products

Integra6on
Suite

Low High
Integration
Framework
Alternatives for Application Integration

Kai Wähner - Enterprise Integration Patterns Revisited
Complexity
of Integration
Integra6on

Products

Integra6on
Suite

Low High
Integration
Framework
INTEGRATION
Connectivity
Routing
Transformation
When to use an Integration Framework?

Connectivity
Integration of
different
Technologies

Many further connectors
Easy to create own components
HTTP

FTP

File

XSLT

EJB

JDBC
Akka

TCP

SMTP

RSS

Quartz

jclouds

LDAP

JMS

MQTT

AMQP

Atom

AWS

Bean-Validation

CXF

IRC

Jetty

JMX

Lucene

Netty

RMI

SQL

Twitter
MongoDB

Connectivity

Integration
implement
Integration Frameworks

Code Example: Apache Camel (Java DSL)

Code Example: Apache Camel (XML DSL)

Kai Wähner - Enterprise Integration Patterns Revisited
Connectivity
Routing
Transformation
INTEGRATION
Tooling
Monitoring
Support
Complexity
of Integration
Integra6on

Products

Integra6on
Suite

Low High
Integration
Framework
+
When to use an Enterprise Service Bus?

35
“Citizen Integrator / Hybrid Integration” (Gartner, Forrester)
© Copyright 2000-2015 TIBCO Software Inc.

36
Integration Products
One
Integra*on

Product
does
not
ﬁt

all
problems!

Adaptive Integration Approaches
Self-Service Integration for
DIY Integrators (iSaaS)
Business Integration
Cloud Integration
Integration Platform as a
Service (iPaaS)
Enterprise
Business
Users
Department
Audience
Enterprise Integration
Platform
(on-premise and cloud)
Enterprise Service Bus

Integration Concepts
Decision
Data
Conversion
Transitions
Process
Call
Process Design Tools
Integration Processes
Connectors to vendor
applications
Connectors to technologies
Connectors
(e.g. JMS, java, FTP) (e.g. SAP, SF.com)

Enterprise Service Bus (ESB)
Intuitive graphical design environment
Streamlines time and cost of development and training
Graphical
Process
Modeling
Native
Standards based
XSLT Mapper
Drag-n-Drop
Access to
Resources
Fully Integrated
Test Environment
Adapters, Services, Processes,
Deployment, Management

41
iPaaS for the Citizen Integrator

iSaaS: Integration for the “Prosumer”
Subscription cloud service that
makes it easy to automate moving
of data between apps on the Web

iSaaS: Integration for the “Prosumer”
•  Connectors
–  SaaS APIs baked in a common
user interface
•  Recipes
–  Combinations of connectors
preconfigured as “apps”
–  Just configure to your accounts
–  Customize as needed
•  Drag and Drop ease

Evolution of Integration
Microservices

Web
APIs

Real-‐6me

Web

Oriented

Architecture

Enabling
Technologies

In-‐Memory,
Mul6core

REST,
JSON

Demand
Drivers

Mobile,
Cloud

Fast
Data,
IoT

Services

Web
Services

Real-‐6me

Service

Oriented

Architecture

Enabling
Technologies

XML,
SOAP,
WS-‐*

Process
Modeling

Demand
Drivers

E-‐Commerce

BPM

Interfaces

Adapters

Real-‐6me

Enterprise

Applica6on

Integra6on

Enabling
Technologies

Client-‐Server

Messaging
Middleware

Demand
Drivers

ERP

Analy6cs

Records

Batch
Jobs

Non-‐real6me

Enabling
Technologies

Mainframe

ETL,
Databases

Data

Integra6on

Demand
Drivers

Data
Processing

MIS

Are
we

there
yet?

Level-‐Up
by

u=lizing
the

lessons,

assets
and

prac=ces
of

the
previous

Level

Accelera=ng
Produc=vity
&

Agility

Enterprise Integration Patterns (EIPs) …
•  … are needed everywhere!
•  Not just in an Integration Framework
or Integration Product!

Connectivity
Routing
Transformation
Complexity
of Integration
Integra6on

Products

Integra6on
Suite

Low High
Integration
Framework
INTEGRATION
Tooling
Monitoring
Support
+
API MANAGEMENT
BIG DATA
INTERNET OF THINGS
STREAMING ANALYTICS
„YOU NAME IT“
+
When to use an Integration Suite?

EIPs for Streaming Analytics
Stream Processing for Real Time Processing of Big Masses of Data
à Fast Data

EIPs for Open API and API Management
Your

Enterprise

Closed
APIs

EDI
Web

B2B
SOA
FTP

Pre-‐deﬁned
integra6on
points

Limited,
trusted
partners

Strictly
constrained
interac6ons

Enterprise
friendly

Open
APIs

Opportunis6c
access
points

Many
partners,
untrusted

Encourage
new
ideas

Developer
friendly

Your

Enterprise

Partners

Innovators

Known

Par6es

Consumers

Unknown

Par6es

Suppliers

Coope66on

Employees

API
API

API

API

API

API

API

API

API
API

API

API

API

API

API

API

API

API

API

API

EIPs for API Management (e.g. Transformation or Routing)
CompositeConsumer
Domain Target Service
Domain
Target
Facade
Consumer
Target
Provider
Facade
Target
Target
Endpoint
Facade
Operation
Target
Provider
Facade
Policies
Target
Policies
Router
Routing
Policies
Partner
Policies
Facade
Endpoint
Target
Operation

EIPs for Log Analytics and Operational Intelligence
Collect
Store

Index
&

Search

Parse
&

Normalize

Visually

Analyze

Format
&

Forward


EIPs for Business Logic
E.g. with Microservices:
•  Services implementing a limited set of business functions
•  Services developed, deployed and scaled independently
Integration
Service
Monolith application SOA
Integration
Service
Business
Service
Business
Service
Business
Service
Micro
Service
API Gateway
SaaS Business
Service

EIPs for Integration of “Internet of Everything”
à Real Time Integration with Things, APIs, Business Processes, Cloud, Social, etc.
hep://blogs.cisco.com/ioe/how-‐the-‐internet-‐of-‐everything-‐will-‐change-‐the-‐worldfor-‐the-‐beeer-‐infographic

55
Integration Suite
Do
I
have
to
use
one

Integra*on
Suite
for

all
these
use
cases?

One or more frameworks / vendors?
Frameworks or Products from different Vendors
•  No license costs (for open source frameworks)
•  A lot of glue code
•  Testing / Bugfixing
•  No (combined) support
Frameworks or Products from a single Vendor
•  All integrated within one framework or product? à Vision!
•  Not integrated? à Often reality!
•  Loosely coupled, highly integrated frameworks or products? à Best choice!

Future of EIPs: The BAD News
•  New Patterns (since release of the book)? ß NOT YET
–  Conversations
–  Complex Event Processing
–  Error Handling
•  More Details: „EIP Visions“ ß TO BE DETERMINED
–  EIP I: Messaging à done
–  EIP II: Conversations à 50 page paper
–  EIP III: Process à tbd
–  EIP IV: Event Processing à tbd
http://www.eaipatterns.com/ramblings/72_eipvolumes.html

Future of EIPs: The GOOD News
•  New Patterns not necessary! ß OFTEN
–  EIPs were based on Messaging already, which is a good thing (usually)
–  Conversations == State == Anti Pattern (often)
–  Integration scenarios can be solved with existing patterns (usually)
•  Better Tooling ß YES
–  More frameworks and products (e.g. Stream Processing for Big Data / Fast
Data, Internet of Things)
–  Stable, mature tools
–  New standards (MQTT, OMG's DDS, Eclipse Open IoT Stack, etc.)
–  Support for new hardware (iBeacons, OSIsoft Pi, Google Glass, etc.)
–  Ease of use (no coding)
–  Better scalability

Future of Integration and EIPs
Buy
this
book

if
you
do
not
own
it
already!

Integration will get even more important in the future
than it is today!
The number of different data sources and technologies increases even more than in the past:
•  CRM, ERP, Mainframe, B2B (EDIFACT), etc. will not disappear
•  DWH, Hadoop cluster, event / streaming server, In-Memory DB – all of them have to
communicate
•  Cloud, Mobile, APIs, Big Data, Internet of Things are no option, but our future!
EVERYTHING HAS TO BE INTEGRATED!

Did you get the Key Take-Aways?

Integration is the key factor for success of business in the future!
Enterprise Integration Patterns are used everywhere!
Real time integration is the game changer in a Big Data and IoT world!
Recap: Key Take-Aways

Questions?
Kai Wähner
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
LinkedIn / Xing à Please connect!

Enterprise Integration Patterns Revisited (again) for the Era of Big Data, Internet of Things and Microservices

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Enterprise Integration Patterns Revisited (again) for the Era of Big Data, Internet of Things and Microservices

Similar to Enterprise Integration Patterns Revisited (again) for the Era of Big Data, Internet of Things and Microservices (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Enterprise Integration Patterns Revisited (again) for the Era of Big Data, Internet of Things and Microservices