Enterprise Integration Patterns Revisited (EIP) for the Era of Big Data, Internet of Things (IoT) and Cloud

1. Enterprise Integration Patterns
Revisited for an Era of
Big Data and Internet of Things
Kai Wähner
kwaehner@tibco.com
@KaiWaehner
www.kai-waehner.de
Xing / LinkedIn  Please connect!

2. Consulting
Developing
Coaching
Speaking
Writing
Selling
Kai Wähner
Main Tasks
– Requirements Engineering
– Enterprise Architecture Management
– Business Process Management
– Architecture and Development of
Applications
– Service-oriented Architecture
– Integration of Legacy Applications
– Cloud Computing
– Big Data
Contact
– Email: kontakt@kai-waehner.de
– Blog: www.kai-waehner.de/blog
– Twitter: @KaiWaehner
– Social Networks: Xing, LinkedIn

3. Key Messages
– Integration is the key factor for success in the future of IT!
– Enterprise Integration Patterns are used everywhere!
– Real time integration is the game changer in a Big Data and IoT world!

4. Agenda
– Application Integration
– Enterprise Integration Patterns
– Modeling
– Frameworks and Tools
– Future

5. Agenda
– Application Integration
– Enterprise Integration Patterns
– Modeling
– Frameworks and Tools
– Future

6. Growing IT Infrastructure Everywhere
• Applications
• Interfaces
• Technologies
• Products
• Cloud
• Mobile
• Internet of Things

7. Heterogeneity
Host  Client / Server  SOA  Cloud  Big Data  IoT  ...

8. The Solution: Application Integration
All roads lead
to Rome ...

9. AmazonS3 s3 = new AmazonS3Client(new PropertiesCredentials(
S3Sample.class.getResourceAsStream("AwsCredentials.properties")));
String bucketName = "my-first-s3-bucket-" + UUID.randomUUID();
String key = "MyObjectKey";
try {
s3.createBucket(bucketName);
s3.putObject(new PutObjectRequest(bucketName, key, createSampleFile()));
S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
ObjectListing objectListing = s3.listObjects(new ListObjectsRequest()
.withBucketName(bucketName)
.withPrefix("My"));
s3.deleteObject(bucketName, key);
s3.deleteBucket(bucketName);
} catch (AmazonServiceException ase) {
// error handling...
Writing a lot of Glue Code?

10. Custom Glue Code is no Option!
• Standardized Modeling
• Efficient Realization
• Automatic Testing

11. Agenda
– Application Integration
– Enterprise Integration Patterns
– Modeling
– Frameworks and Tools
– Future

12. “A software design pattern, in the sense of a template,
is a general solution to a problem in programming. A
design pattern provides a reusable architectural outline
that may speed the development of many computer
programs.”
http://en.wikipedia.org/wiki/Design_pattern
What is a Design Pattern?

13. Enterprise Integration Patterns (EIP)

14. “The goal of EIPs is to document
technology-independent design
guidance that helps developers and
architects describe and develop
robust integration solutions.”
http://www.eaipatterns.com
Enterprise Integration Patterns (EIP)

15. History of EIPs
• Talk from Gregor Hohpe
– WSO2 Con 2011
– „History, present, future of EIP“
– A lot about past (ideas, creation of
book, ...)
– http://www.youtube.com/watch?v=
Xwi1DU6KoQ4
• Today‘s Talk
– Not too much about history
– Just about present and future

16. http://www.eaipatterns.com
EIP Example: Splitter

17. http://www.eaipatterns.com
EIP Example: Aggregator

18. Combining EIPs shows their true power!
A + B + C = ABC

19. http://www.eaipatterns.com
EIP Example: Composed Message Processor

20. Status Quo
• Enterprise Integration Patterns do
not change
• Just technologies change
• Implementations for EIPs are
already available

21. Agenda
– Application Integration
– Enterprise Integration Patterns
– Modeling
– Frameworks and Tools
– Future

22. Enterprise Integration Patterns

23. Use Case Example

24. Modeling Tool: Microsoft Visio (Windows)

25. Modeling Tool: Omnigraffle (Mac)

26. Agenda
– Application Integration
– Enterprise Integration Patterns
– Modeling
– Frameworks and Tools
– Future

27. Complexity
of Integration
Enterprise Service
Bus
Integration Suite
Low High
Integration
Framework
Alternatives for Application Integration

28. Kai Wähner - Enterprise Integration Patterns Revisited
Complexity
of Integration
Enterprise Service
Bus
Integration Suite
Low High
Integration
Framework
INTEGRATION
Connectivity
Routing
Transformation
When to use an Integration Framework?

29. Connectivity
Integration of
different
Technologies

30. Many further connectors
Easy to create own components
HTTP
FTP
File
XSLT
MQ
JDBC
Akka
TCP
SMTP
RSS
Quartz
jclouds
LDAP
JMS
EJB
AMQP
Atom
AWS
Bean-Validation
CXF
IRC
Jetty
JMX
Lucene
Netty
RMI
SQL
Twitter
MongoDB
Connectivity

31. Integration
implement
Integration Frameworks

32. Code Example: Apache Camel (Java DSL)

33. Code Example: Apache Camel (XML DSL)

34. Kai Wähner - Enterprise Integration Patterns Revisited
Connectivity
Routing
Transformation
INTEGRATION
Tooling
Monitoring
Support
Complexity
of Integration
Enterprise Service
Bus
Integration Suite
Low High
Integration
Framework
+
When to use an Enterprise Service Bus?

35. Enterprise Service Bus (ESB)

36. Enterprise Service Bus (ESB)
Decision
Data
Conversion
Transitions
Process
Call
Process Design Tools
Integration Processes
Connectors to vendor
applications
Connectors to technologies
Connectors
(e.g. JMS, java, FTP) (e.g. SAP, SF.com)
Enterprise Integration
Patterns

37. Example: TIBCO BusinessWorks
Graphical
Process
Modeling
Native
Standards
based XSLT
Mapper
Adapters,
Services,
Processes,
Deployment,
Management
Drag-n-Drop
Access to
Resources
Fully Integrated
Test Environment
Intuitive graphical design environment streamlines
time and cost of development and training

38. ESB Example: TIBCO BusinessWorks

39. Adaptive Integration Approaches
• One size does not fit all
– Empower the business with a common set of tools to innovate and integrate
– Bring new business solutions to the ICC once they have matured
iSaaS:
Self-Service Integration
for DIY Integrators
TIBCO™ Simplr™
Cloud Bus
Comprehensive PaaS:
Integration Platform as
a Service
Enterprise
Business
Users
Department
Audience
Leading on-premise
Enterprise Integration
Platform
BusinessWorks

40. Simple Cloud Integration with TIBCO Simplr
TIBCO Simplr™ is a subscription
cloud service that makes it easy to
automate moving of data between
apps on the Web
The images on this slide illustrate product features or functionality under consideration by TIBCO for inclusion in future releases or future
products. If implemented, the actual functionality/design pattern may bear no resemblance to these images.

41. Simple Cloud Integration with TIBCO Simplr
• Connectors
– SaaS APIs baked in a common
user interface built by TIBCO
(for now)
• Recipes
– Combinations of connectors
preconfigured as “apps”
– Just configure to your accounts
– Customize as needed
• Drag and Drop ease
The images on this slide illustrate product features or
functionality under consideration by TIBCO for inclusion
in future releases or future products. If implemented, the
actual functionality/design pattern may bear no
resemblance to these images.

42. Agenda
– Application Integration
– Enterprise Integration Patterns
– Modeling
– Frameworks and Tools
– Future

43. Kai Wähner - Enterprise Integration Patterns Revisited
Social Networks
Location
Weather
Markets
Marketing
campaigns
Sales results
Textile manufacturing
processes
Shipping Manufacturing
Delivery time
© Copyright 2000-2014 TIBCO Software Inc.
Everything is Connected

44. Everything Generates More Data

45. Evolution of Integration
Microservices
Web APIs
Real-time
Web
Oriented
Architecture
Enabling Technologies
In-Memory, Multicore
REST, JSON
Demand Drivers
Mobile, Cloud
Fast Data, IoT
Services
Web Services
Real-time
Service
Oriented
Architecture
Enabling Technologies
XML, SOAP, WS-*
Process Modeling
Demand Drivers
E-Commerce
BPM
Interfaces
Adapters
Real-time
Enterprise
Application
Integration
Enabling Technologies
Client-Server
Messaging Middleware
Demand Drivers
ERP
Analytics
Records
Batch Jobs
Non-realtime
Enabling Technologies
Mainframe
ETL, Databases
Data
Integration
Demand Drivers
Data Processing
MIS
Are we
there
yet?
Level-Up by
utilizing the
lessons,
assets and
practices of
the previous
Level
AcceleratingProductivity&Agility

46. Enterprise Integration Patterns (EIPs) …
• … are needed everywhere!
• Not just in an Integration Framework or Enterprise
Service Bus (ESB)!

47. Acting in Real Time required!
Time
Business
Value
Business Event
Data Ready for Analysis
Analysis Completed
Decision Made
$$$$
$$$
$$
$ Action Taken
Event Processing
speeds action and
increases business
value by seizing
opportunities while
they matter

48. Connectivity
Routing
Transformation
Complexity
of Integration
Enterprise Service
Bus
Integration Suite
Low High
Integration
Framework
INTEGRATION
Tooling
Monitoring
Support
+
API MANAGEMENT
BIG DATA
INTERNET OF THINGS
STREAMING ANALYTICS
„YOU NAME IT“
+
When to use an Integration Suite?

49. EIPs for Streaming Analytics
Stream Processing for Real Time Processing of Big Masses of Data ( Fast Data)
Example: TIBCO StreamBase

50. EIPs for Open API and API Management
Your
Enterprise
Closed APIs
EDI Web
B2B SOA FTP
Pre-defined integration points
Limited, trusted partners
Strictly constrained interactions
Enterprise friendly
Open APIs
Opportunistic access points
Many partners, untrusted
Encourage new ideas
Developer friendly
Your
Enterprise
Partners
Innovators
Known
Parties
Consumers
Unknown
Parties
Suppliers
Coopetition
Employees
APIAPI
API
API
API
API
API
API
API API
API
API
API
API
API
API
API
API
API
API

51. EIPs for API Management (e.g. Transformation or Routing)
© Copyright 2000-2014 TIBCO Software Inc.

52. EIPs for Business Logic (e.g. within Microservices)
• Services implementing a limited set of business functions
• Services developed, deployed and scaled independently
Benefits
• Shorter time to results
• Increased flexibility
Challenges
• Duplication of efforts
• Sustain Quality
Back end
Interfaces
micro service
Integration
micro service
Integration
Integration
API Gateway

53. EIPs for Integration of “Internet of Everything”
 Real Time Integration with Things, APIs, Business Processes, Cloud, Social, etc.
http://blogs.cisco.com/ioe/how-the-internet-of-everything-will-change-the-worldfor-the-better-infographic

54. Open Source
Proprietary
ESB Vendor == Integration Suite Vendor?
Watch out!
• All integrated within one product?  Vision!
• Not integrated?  Often reality!
• Loosely coupled, highly integrated?  Best
choice!

55. ESB Vendor == Integration Suite Vendor?
ESB BPM MDM

56. Custom combination of ESB, BPM, MDM, etc.?
• A lot of glue code
• Testing
• Bugfixing
• No support
•„Full-stack vendors“ already had
•the problems you would have!

57. Future of EIPs: The BAD News
• New Patterns (since release of the book)?  NOT YET
– Conversations
– Complex Event Processing
– Error Handling
• More Details: „EIP Visions“  TO BE DETERMINED
– EIP I: Messaging  done
– EIP II: Conversations  50 page paper
– EIP III: Process  tbd
– EIP IV: Event Processing  tbd
http://www.eaipatterns.com/ramblings/72_eipvolumes.html

58. Future of EIPs: The GOOD News
• New Patterns not necessary!  OFTEN
– EIPs were based on Messaging already, which is a good thing (usually)
– Conversations == State == Anti Pattern (often)
– Integration scenarios can be solved with existing patterns (usually)
• Better Tooling  YES
– More frameworks and products (e.g. Stream Processing for Big Data / Fast
Data, Internet of Things)
– Stable, mature tools
– New standards (MQTT, OMG's DDS, Eclipse Open IoT Stack, etc.)
– Support for new hardware (iBeacons, OSIsoft Pi, Google Glass, etc.)
– Ease of use (no coding)
– Better scalability

59. Complementary: SOA Design Patterns
• Patterns for performance,
scalability, and availability
• Security and manageability
patterns
• Message exchange patterns
• Service consumer patterns
• Service integration patterns

60. Future of Integration and EIPs
Buy this book
if you do not own it already!
Integration will get even more important in the future
than it is today!
The number of different data sources and technologies increases even more than in the past:
• CRM, ERP, Host, B2B, etc. will not disappear
• DWH, Hadoop cluster, event / streaming server, In-Memory DB – all of them have to
communicate
• Cloud, Mobile, APIs, Big Data, Internet of Things are no option, but our future!
EVERYTHING HAS TO BE INTEGRATED!

61. Did you get the Key Message?

62. Integration is the key factor for success in the future of IT!
Enterprise Integration Patterns are used everywhere!
Real time integration is the game changer in a Big Data and IoT world!
Recap: Key Messages

63. Questions?
Kai Wähner
kwaehner@tibco.com
@KaiWaehner
www.kai-waehner.de
Xing / LinkedIn  Please connect!