Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SoC Keynote:The State of the Art in Integration Technology


Published on

This talk discusses Outline of the state of the art of Enterprise Software and how we get there, as I see it. Also second part describes Ballerina, a new programming language WSO2 has built for Enterprise Computing.

It is presented as a Keynote at 11th Symposium and Summer School On Service-Oriented Computing.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

SoC Keynote:The State of the Art in Integration Technology

  1. 1. Srinath Perera VP Research, WSO2 Member, Apache Software Foundation, @srinath_perera The State of the Art in Integration Technology: An Industry Perspective
  2. 2. Part 1: Walk down the Integration Memory Lane
  3. 3. My Memory Lane (15 years) 3 Worked in a J2EE Server Wrote two SOAP Engines and worked in one more ESB ( Apache Synapse) WSO2 ( Opensoure middleware company 500+ employees, 300+ customers including BNY, TFL, Boeing, Fidelity, Ebay ..) Application Server Message Broker Identity ServerWorkflow Engine (Apache Ode) 5-10 release each @WSO2: Has helped design, worked, debugged Stream Processor
  4. 4. Enterprise Systems (e.g. Book a flight that involve two airlines) 4
  5. 5. Interoperability 5 1st try: Proprietary protocols ( RMI, COM) Agreeing on the wire via XML worked 2nd Try: CORBA 3rd Try: Web Service Back to binary protocols? Thrift, Protobuf, Avro
  6. 6. Lessons 6 Simplicity Wins ( as you need developers to understand it) UX can decide technology fate
  7. 7. Middleware: SOAP Engines 7 e.g. GSoap, Apache Axis, Apache Axis2, Apache CXF, .NET Service authoring and Hosting Code first vs. contract (WSDL) first Later JAX-WS Specification for Java REST/ Mircoservices Frameworks (DropWizard, SpringBoot, WSO2 MSS4J
  8. 8. SOA vs. ROA After lot of battles we kind of agreed to use one that is natural to the use case Build the architecture by recomposing loosely coupled Units
  9. 9. Enterprise Integration Integration is mediation
  10. 10. Enterprise Integration (See EIP patterns ( for details) Gregor Hohpe and Bobby Woolf
  11. 11. Middleware: ESB 11 Keep backend simple by terminating security etc Simple script language to code mediation logic Abstract service via proxies One place to all integration Logic ( Avoid Spaghetti) e.g. WSO2 ESB, Mule, Boomi, Apache Camel
  12. 12. Async Persistent Messaging 12 ESBs learnt to work with Message brokers ( mainly JMS) and added operators like message stores and processors Decoupling: Time, Space, Synchronization
  13. 13. Composition We use workflows for long running Others done via Code Recomposition part of SOA/ROA Most done via UI Code Think a home loan, which will last longer than servers it will run on ESB can do it also Short running
  14. 14. APIs/ API Management 14 Crossing Trust boundaries adds complexity • Secure the endpoint • User provisioningAlso it acts as the API Registry API management provided a packaged solutions
  15. 15. Enterprise Architecture in 2014 15 Most service has more than one client Had lot of services Services are recomposed at higher levels They shared data bases Deployment was complicated. Services couldn’t evolve independently ( although SOA always thought about them that way). Services are developed as one big system, or several big systems
  16. 16. Microservices Services are versioned and backward simple, lightweight, loosely coupled services that can be developed and released independently of each other. No shared database Avoid dependency hell • Composition • Security It is mostly set of good practices to build large systems I have written about this topic in details in walking-the-microservices-path-towards-loose-coupling-few-pitfalls-4067bf5e497a
  17. 17. No Shared Databases Merge two Microservices To remove coupling between service implementations Use asynchronous messaging Use two and deal with Consistency • Use distributed transactions • Use compensation or lesser guarantees What if two microservices share data?
  18. 18. Service Evolution Backward and forward compatible ( time bounded) Point of microservices is new release of a service must not break other services Services should be versioned In most cases this is a matter of adding optional parameters and never renaming or removing existing parameters.
  19. 19. Avoid Dependency Hell Don’t do this e.g. RXJava, node.js model Hard to get it right Not always possible Perf and security concerns
  20. 20. No ESB allowed?? 20 The Anger of Achilles, by Giovanni Battista Tiepolo Mico Integration (One integration per server) “A rallying cry” in Microservices movement Client driven Model Not sure what is the solution
  21. 21. May you get to build a Distributed System and manage it As in “May you live in interesting times”
  22. 22. Now you have truly built a distributed System • You can only justify this if • Team > Two Pizza team • If you do > 1000s TPS • If not may be you should go back to small web app you had not stop making your life hard 22 If yes, now you have new problems • Keep it running • Find and debug problems
  23. 23. Tracing • Need a distributed tracing infrastructure to find out when things go wrong • One option is Elasticsearch, that let you collect logs, search them via text indexing, and visualize them • Can use analytics infrastructure as we will discuss next 23
  24. 24. APIs are Great Data Collection Points • In a architecture done around APIs, all key functionalities captured by APIs • Data collected at these points can be used to understand business 24
  25. 25. Part 2: Can we rethink middleware for Integration?
  26. 26. We are entering the age of APIs • Future Apps are built (mostly) with API composition • We must make it easier to produce and consume networked services
  27. 27. Integration 27 Inherently distributed • Built with messages over the network • Needs security XML/ JSON based ( as that only format that work across systems) Parallel Program by people with limited programming knowledge Glued with data manipulations and mapping
  28. 28. Integration is often done in a Second Class Way 28 Parallelism need to be explicitly handled XML/ JSON conversions manually done Concurrency is poorly handled Users need to think through the security model Network communications need to be explicitly handled Need lot of plumbing for testing
  29. 29. What is Wrong with DSLs? 29 Type system is weakPoor performance in non obvious scenarios Concurrency is poorly handled Editors and tooling are not native Debugging is painful Painful when actual complex logic is needed
  30. 30. Ballerina 30 Natively parallel First class programming language with tools, debugging etc Textual and graphical parity on sequence diagram metaphor Strongly and statically typed with powerful type system Designed for network with JSON, XML, SQL, MIME with HTTP, JMS
  31. 31. Key Concepts
  32. 32. Visual and Textual Parity • Make it easier to produce and consume networked services 32
  33. 33. Functions and Annotation • Let you compose the program in term of decomposable units which is the key to scale the visual representation • Annotations let you • Separate configurations from logic • Externalize configurations to files without code 33
  34. 34. Code vs. Biz 34 Visual Parity and functions should let the Geeks implements detail functions which can be recomposed by Business Analysts
  35. 35. Ballerina knows XML and JSON • Include XML and JSON as native types that can be casted to and from ballerina structs • Language include native type mapping support, with type mapping drag and drop editor 35
  36. 36. Connectors 36 receive events via Resources Support HTTP, Web Socket, JMS etc built into the language Tool: Swagger - > connector make recomposing ballerina services Web APIs: Twitter, GMail, LinkedIn, Facebook, Lambda Functions Security: BasicAuth, OAuth, AmazonAuth Call external APIs via Connectors
  37. 37. When to use Ballerina? • Write micro ( integration) services:80-20 rule • 80%. - work with other services • 20% - your logic • Recompose existing services to an API backend ( very similar to central ESB • Integration scripts, scripts for Web API programming ( main)
  38. 38. It is open source under Apache License Find us through • Users: ballerina- • Slack: #ballerinalang • Twitter: @ballerinalang • StackOverflow: #ballerinalang • Developers: ballerina-
  39. 39. Questions?