Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From EAI to Serverless

52 views

Published on

1. What is integration? 2. Integration challenges 3. “Traditional” integration approach 4. Recents insights and developments 5. Integration Patterns 6. Out of the Box 7. Conclusion

Vortragsfolien vom Digital Xchange 2019, weitere Informationen auf digital-xchange.de

Published in: Technology
  • Be the first to comment

From EAI to Serverless

  1. 1. From EAI to Serverless https://www.redexpertalliance.com From EAI to Serverless Evolution of Integration patterns and architectures Digital Exchange 2019, Gummersbach 2019/05/25
  2. 2. From EAI to Serverless https://www.redexpertalliance.com Red Expert Alliance - Who we are 2
  3. 3. From EAI to Serverless https://www.redexpertalliance.com Who are we? 3 Lucas Jellema (AMIS, NL) Arturo Viveros (Sysco, NO) Sven Bernhardt (OPITZ, DE)
  4. 4. From EAI to Serverless https://www.redexpertalliance.com Agenda 1. What is integration? 2. Integration challenges 3. “Traditional” integration approach 4. Recents insights and developments 5. Integration Patterns 6. Out of the Box 7. Conclusion 4
  5. 5. From EAI to Serverless https://www.redexpertalliance.com What is integration? 5
  6. 6. From EAI to Serverless https://www.redexpertalliance.com What is integration? System A System B XX Simple definition: enabling systems to talk to each other 6
  7. 7. From EAI to Serverless https://www.redexpertalliance.com What is integration? System A System B Ensure that data is where it is needed when it is needed in the form in which it is needed 7
  8. 8. From EAI to Serverless https://www.redexpertalliance.com Integration challenges 8
  9. 9. From EAI to Serverless https://www.redexpertalliance.com Integration Challenges ● Monolithic / aging systems ● Disparate data models ● Architectural complexity ● Heterogeneous ecosystem ● Technical debt ● Bloated projects / overhead ● Visibility / error handling ● Production deployment 9
  10. 10. From EAI to Serverless https://www.redexpertalliance.com IT organizations are drowning 10
  11. 11. From EAI to Serverless https://www.redexpertalliance.com “Traditional” integration approach 11
  12. 12. From EAI to Serverless https://www.redexpertalliance.com 12
  13. 13. From EAI to Serverless https://www.redexpertalliance.com ESB as basic architecture element 13 global internal specific External Interfaces – Proxy Servics Integration Logic Internal Interfaces – Business Interfaces
  14. 14. From EAI to Serverless https://www.redexpertalliance.com Limited and expensive Scalability Loadbalancer ESB 1 ESB 2 ESB 3 14
  15. 15. From EAI to Serverless https://www.redexpertalliance.com Caught in a technology Trap Oracle SOA Cloud Service Fixed Technologies Set <XML /> { „dialect“:“JSON“ } 15
  16. 16. From EAI to Serverless https://www.redexpertalliance.com Limited agility ● Hard to implement Continuous Integration / Continuous Delivery ○ Complex infrastructure ○ Missing automated testing capabilities ● Monolithic deployment approach ● Expert knowledge needed ○ Implementation ○ Deployment ○ Error analysis http://geek-and-poke.com/ 16
  17. 17. From EAI to Serverless https://www.redexpertalliance.com Recent insights and developments 17
  18. 18. From EAI to Serverless https://www.redexpertalliance.com New insights, new context 18 increased scale – because of real -time and web scale demands introduced by IoT and a further growth of the use of Apps Need for speed - real time insights from fast data streams No/less strict delineation between Enterprise (message based) Integration and ETL/Data Integration tighter security requirements, faster change cycle (Agile, DevOps), 24/7 Operations Scarcity of talent/developers Is a canonical model/language such a good idea? What about ‘domain/context mapping’ and ‘anti corruption layer’ from domain driven design We have learned to ask: How fresh should data be? How crucial is ACID and instant consistency? increased usage of standard applications (primarily SaaS) the advent of cloud (necessity to support hybrid scenarios - cross on premises and cloud as well as multi vendor cloud, new architecture patterns such as domain driven design, microservices, CQRS, event sourcing, “dumb pipes, smart end points” advances in technology, containers, serverless, REST, API technology – JSON (vs XML), REST (vs SOAP), containers, serverless, “embedded HTTP server”, industry standards like Kafka TCO of Middleware Platforms Strategic roadmap of middleware vendors
  19. 19. From EAI to Serverless https://www.redexpertalliance.com The world is getting more complex! 19
  20. 20. From EAI to Serverless https://www.redexpertalliance.com Shift to hybrid integration solutions 20
  21. 21. From EAI to Serverless https://www.redexpertalliance.com Integration is not just ESB anymore Modern Integration platforms consists of diverse components ● Integration (µ)Services ● API Management ● Event Hub 21 https://www.linkedin.com/pulse/when-creating-apis-focus-your-api-gateway-what-does-best-wunderlich
  22. 22. From EAI to Serverless https://www.redexpertalliance.com Integration, Big Data and Analytics 22 Analytics App 1 Analytics App 2 Analytics App 3 Analytics App 4 Landing Zone Data Pipeline / Data Ingestion Metadata Store Content Store Rawdata Store Big Data Platform DataLake Integration Services APIs
  23. 23. From EAI to Serverless https://www.redexpertalliance.com Integration pattern 23
  24. 24. From EAI to Serverless https://www.redexpertalliance.com Two dimensions of integration 24
  25. 25. From EAI to Serverless https://www.redexpertalliance.com ● End User ”integration” ● API Gateway as runtime component for ○ API policy enforcement ○ Request routing ● (Near) Real-time, synchronous interactions ● APIs born from business requirements ○ Who is using the API? ○ How is the API being used? ● Use of lightweight, standard protocols Vertical integration 25https://microservices.io/patterns/apigateway.html API Gateway Backend for Frontend
  26. 26. From EAI to Serverless https://www.redexpertalliance.com API first design approach 26
  27. 27. From EAI to Serverless https://www.redexpertalliance.com Demo Use Case 27 Customer API CRM Database Client Application● Integration of a DB-based CRM system ● Client application uses API to ○ Find/Read Customer data ○ Create Customer data ○ Update Customer data ○ Delete Customer data ● API ensures secured access of Customer data ● Integration Service takes care of providing the data in the correct format according to the API description VerticalIntegration
  28. 28. From EAI to Serverless https://www.redexpertalliance.com Evolution of ‘vertical integration flow’ implementation 28
  29. 29. From EAI to Serverless https://www.redexpertalliance.com Horizontal integration 29 ● Classic System2System integration ● Heavy-weight, proprietary protocols ● Technical interfaces ● Batch processing ● Asynchronous interaction patterns ● New insights: relaxed freshness and consistency demands, data replication ● New context: cloud, hybrid, “dumb pipes, smart endpoints”, Standard Apps & SaaS evolution
  30. 30. From EAI to Serverless https://www.redexpertalliance.com Horizontal integration 30 Technology & Application Adapters & Connectors, Workflow coordination, Event Bus. Exception Handling, Job Scheduling & Mgt, Transformation, Routing. Monitoring. Streaming Analysis. Authentication.
  31. 31. From EAI to Serverless https://www.redexpertalliance.com (Cloud) Adapters & iPaaS platforms 31
  32. 32. From EAI to Serverless https://www.redexpertalliance.com Challenges with Horizontal Integration ● Handle requirements for speed, consistency, volume & [global] distribution ● Detect Change ● Extract Data ● Publish Change ● Transport ○ deal with limited connectivity, temporary unavailability ● Convert/Transform ● Apply ● Monitor ● Handle exceptions ● Enforce constraints & authorization 32
  33. 33. From EAI to Serverless https://www.redexpertalliance.com Event-Driven Integration 33 “Life is a Stream of Events” https://www.slideshare.net/ConfluentInc/life-is-a-stream-of-events
  34. 34. From EAI to Serverless https://www.redexpertalliance.com Event-Driven Integration 34 ● Systems act and react to internal & external events ● Streams of events represent information and behavior ● Data should end up in the right places… eventually ● Popular design patterns: ○ Event Sourcing ○ CQRS ○ Materialized view
  35. 35. From EAI to Serverless https://www.redexpertalliance.com Event-Driven Integration 35
  36. 36. From EAI to Serverless https://www.redexpertalliance.com Event-Driven Integration 36
  37. 37. From EAI to Serverless https://www.redexpertalliance.com “Serverless functions provide a synergistic relationship with event streaming applications; they behave differently with respect to streaming workloads but are both event driven” Event-Driven Serverless Architectures https://confluent.i o
  38. 38. From EAI to Serverless https://www.redexpertalliance.com Out of the box Ahead of Time Integration Collective Data Set Event Sourcing 38
  39. 39. From EAI to Serverless https://www.redexpertalliance.com JIT vs AOT 39
  40. 40. From EAI to Serverless https://www.redexpertalliance.com JIT vs AOT Data Integration ● JIT ○ Process [synchronous] request for data ‘just in in time’ ○ Performance overhead suffered at (every) request time ○ Only data that is requested ○ Fresh as can be ○ All authorization rules applied ○ Single Source of Truth ○ Transformation & Aggregation at request time ○ Hard (runtime) dependencies from consumer on producer ● AOT ○ Pre-process / pre-share data ‘ahead of time’ ○ Potentially includes data that will never be requested ○ Data can be outdated 40
  41. 41. From EAI to Serverless https://www.redexpertalliance.com CQRS => Specialized Query Stores Cater for different (read) Data Usage Scenarios Different in: ● Why ● What – grain, aggregation, breadth, type of and variation in queries ● How – format, language, accuracy and freshness, consistency ● When – frequency, time of day | week | month, 24/7 ● Where – latency, band width, off line ● Who – scale, different roles & user groups 41
  42. 42. From EAI to Serverless https://www.redexpertalliance.com CQRS => Optimized Data Set for Special Use Case ● Shape & Format ● Right Grained (aggregation, consolidation) ● Filtered ● Enriched (from multiple sources) ● Location ● Time (refreshed at the right time) ● Search Options ● BASE (eventually – when? – consistent) ● Authorization ● GDPR 42 C Q BFF API Frontend
  43. 43. From EAI to Serverless https://www.redexpertalliance.com Collective Data Set 43
  44. 44. From EAI to Serverless https://www.redexpertalliance.com Data Sharing Ecosystems ● Multiple independent organizations ● Mutually benefit from having access to the same Data at the same time ● Share data & Have access to data with minimum of overhead and with equality among the ecosystem partners ● Partners are on different network domains, physical locations, technology stacks, … ● New Partners can join ● Partners can leave ● Consensus on what constitutes correct data 44
  45. 45. From EAI to Serverless https://www.redexpertalliance.com Collective Data Set 45
  46. 46. From EAI to Serverless https://www.redexpertalliance.com Collective Data Set 46
  47. 47. From EAI to Serverless https://www.redexpertalliance.com Collective Data Set == Collective Data Change Events 47
  48. 48. From EAI to Serverless https://www.redexpertalliance.com Collective Data Set => Chained Distributed Data Store 48
  49. 49. From EAI to Serverless https://www.redexpertalliance.com Conclusion 49
  50. 50. From EAI to Serverless https://www.redexpertalliance.com Key Takeaways 50
  51. 51. From EAI to Serverless https://www.redexpertalliance.com Questions & Answers 51 Thanks for your attention!

×