Paul Fremantle's talk on the evolution of integration from QCon London

1. The evolution of integration
QCon London 2012
Paul Fremantle
CTO and Co-Founder, WSO2
paul@wso2.com
© WSO2 2012 @pzfreo #wso2

2. © WSO2 2012 @pzfreo #wso2

3. Batch file transfer -
the cockroach of integration
© WSO2 2012 @pzfreo #wso2

4. File transfer lives on
Exis%ng
System
XML/JMS
legacy
flat file Exis%ng
System
WSO2 ESB
Split/Iterate
DBLookup/Filter
Transform to MQ
Send
WSO2 ESB Database
Poll
Record->XML
XML->XML
Send
© WSO2 2012 @pzfreo #wso2
NEW YORK LONDON

5. EAI
Hub
© WSO2 2012 @pzfreo #wso2

6. EAI hub
• Many integration models still uses the “hub
model” today (even with an ESB)
– Most vendors renamed their hub to ESB
• Why?
– Well understood pattern
– Easy to manage
• Why not?
– Too many meetings with the “EAI Hub Team”
– MQSI experiences
© WSO2 2012 @pzfreo #wso2

7. Hub approach with an ESB
© WSO2 2012 @pzfreo #wso2

8. “Message Oriented Middleware”
http://www.flickr.com/photos/robeast/
© WSO2 2012 @pzfreo #wso2

9. MOM model
• Decouple message producers from
consumers
– Decoupled in addressing and in time
• Not inherently decoupled in message format
– Though in many cases that too
• One-way asynchronous messages
– But request-reply possible using “Reply Queues”
• Usually used with reliable delivery
© WSO2 2012 @pzfreo #wso2

10. Queued Transaction Processing
System
Queue
A
Transac%on
1:
Transac%on
2:
Add entry to DB1 Add entry to DB2
Enqueue Message Dequeue Message
DB1
System
DB2 B
© WSO2 2012 @pzfreo #wso2

11. AMQP
© WSO2 2012 @pzfreo #wso2

12. Event Driven Architecture
Event
Consumer
Event
Event
Producer
Event
Consumer
© WSO2 2012 @pzfreo #wso2

13. “Event Driven Architecture”
• Actually how Apache (and WSO2) work(s)
– Mailing lists = topics
• Can be layered with reliable delivery
• Used a lot in high-volume logging, trading
environments, fraud detection, etc
• Requires a very different mindset
© WSO2 2012 @pzfreo #wso2

14. EDA
© WSO2 2012 @pzfreo #wso2

15. Feedback loops
Event Broker
Adapter
Adapter
Black Black
Box Box
system system
hEp://pzf.fremantle.org/2008/09/interesOng-­‐problem-­‐in-­‐event-­‐driven.html
© WSO2 2012 @pzfreo #wso2

16. Solution
© WSO2 2012 @pzfreo #wso2

17. Why did SOA evolve?
• Directly came out of XML
– Understanding the schema and structure of
messages
– Especially within the “fabric” not just at the
endpoints
• What’s different?
– Metadata
– Policies
– Security
© WSO2 2012 @pzfreo #wso2

18. Service
http://www.flickr.com/photos/yjv/
© WSO2 2012 @pzfreo #wso2

19. SOA failures
One consumer per service
© WSO2 2012 @pzfreo #wso2

20. SOA failures
“Just buy an ESB from me”
© WSO2 2012 @pzfreo #wso2

21. SOA failures
Vendor Driven Architecture
© WSO2 2012 @pzfreo #wso2

22. “soa” successes
© WSO2 2012 @pzfreo #wso2

23. © WSO2 2012 @pzfreo #wso2

24. Why ESB/SOA model isn’t just EAI
• Policy based
– XACML, Throttling Policy, etc
– eBay’s Internal Service Router
• Independent management
– Loose coupling of configuration
– Hot deploy / re-deploy / continuous delivery
• Governance
– Lifecycle and Dependency management
– Analysis and reporting on the meta-model
• Non-blocking asynchronous routing
• Distributed architecture
© WSO2 2012 @pzfreo #wso2

25. Running your SOA like a Web startup
© WSO2 2012 @pzfreo #wso2
http://www.zdnet.com/blog/hinchcliffe/running-your-soa-like-a-web-startup/525

26. API
• An API is a business capability delivered
over the Internet to internal or external
consumers
– Network accessible function
– Available using standard web protocols
– With well-defined interfaces
– Designed for access by third-parties
© WSO2 2012 @pzfreo #wso2

27. What is different from an API and a
Service?
• Publishing your API in a Portal
• Expecting people to use it without them
having to meet with you
• Making it easy to consume (JSON? Ready
built clients in Github?)
• Governance
– Caring who uses it
– Letting them know when you version it
– Meeting an SLA
© WSO2 2012 @pzfreo #wso2

28. Key API technologies
• json / rest
• OAuth / OAuth2 keys
• SLA management
• API portal / API Store
– Catalogue, subscription/purchase
– Monetization
– Forum, Ratings, Social
• Analytics
© WSO2 2012 @pzfreo #wso2

29. © WSO2 2012 @pzfreo #wso2

30. REST description
© WSO2 2012 @pzfreo #wso2

31. High volume integration @ eBay
© WSO2 2012 @pzfreo #wso2

32. Change in focus
• Security, tokens, access control/entitlement
• Throttling, caching
• Latency and CPU usage
• Monitoring, BAM and CEP
© WSO2 2012 @pzfreo #wso2

33. © WSO2 2012 Intel Xeon CPU E5520 @ 2.27GHz, 4 Cores,
@pzfreo #wso2
8192 KB Cache. 2 CPU system.

34. API management
© WSO2 2012 @pzfreo #wso2

35. Scalable analytics
© WSO2 2012 @pzfreo #wso2

36. Cloud integration
Database
Cloud Service Gateway
ESB-­‐as-­‐a-­‐Service
SAP
© WSO2 2012 @pzfreo #wso2

37. Cloud integration
• APIs are the right approach
– Use a “cloud gateway” to bridge into internal
systems
• “Push-me pull-you” pattern
– Use an active ESB in the cloud
• Analytics
– See what is happening
© WSO2 2012 @pzfreo #wso2

38. What’s next?
• Still a long way from canonical models
• Successful systems are using “soa” and
“rest” at scale
– Architecture is more important than dogma
• Governance sounds boring but is key
• Applying monetization approaches and
“API Store” models
• Analytics and feedback loops
© WSO2 2012 @pzfreo #wso2

39. Summary
• Integration has evolved in some interesting
ways
– Async messaging, EDA, APIs, High Volume
• Evolution isn’t monotonic
• Doing APIs right is about the mindset as
much as the technology
© WSO2 2012 @pzfreo #wso2

40. © WSO2 2012
? @pzfreo #wso2