Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Integration strategies best practices- Mulesoft meetup April 2018
Similar to Integration strategies best practices- Mulesoft meetup April 2018 (20)
Recently uploaded
Recently uploaded (20)
Integration strategies best practices- Mulesoft meetup April 2018
- 1. Integration Strategies & Best Practices Kick starting on Integrations, here is what you need to know
- 2. Personal Questions • How many $ did you spend today • How many APIs did you use
- 3. About meRohan Rasane Integrations Architect - ServiceNow Integrations Architect - ServiceNow
- 4. Agenda • About a Integration platform • Buying/building guide • Objections of an integration platform • How to kick start • Practices • Bulkhead and Circuit-breaker in depths
- 5. About your Integration Platform Don’t have an integration platform yet You own one, but it is unorganized and there is no control Third party apps are very unresponsive, reliability is at stake Dev to Deploy time is huge Where do you fit ?
- 6. Batch Processing Loose coupling Micro services & Reuse Governance Analytics Security Integrations Platform – Features What features should the platform provide Points to consider while buying or building an integration platform
- 7. Integrations Platform – Key objectives Reliable Available Intuitive APIs Least manual interventions Consumer Friendly errors & exceptions Ease of Monitoring & Reporting Enable Self Service Low maintenance costsWhat are the customers confidence factors that the platform should abide to get a great NPS J API or integration platform users = customers
- 8. Integration - Resiliency Patterns Timeouts Circuit breakers Bulkheads HeartBeat Queues - Asynchronous Communication Retries Idempotency These are the platform’s pillars that help detect failures and recover quickly #design for failuresNetflix OSS
- 9. Design Questions • Is there an existing integration, if yes, how does it work and what are the problems • What data needs to be integrated • Who are the users (systems or applications) • Transaction count: daily-monthly-yearly. • What kind of orchestrations are required, if any • Personas and use-cases • Manual tasks • Fallbacks • Future scaling needs • Data security or confidentiality Ask these questions when there is a need of integrations #non-technical #birds-eye view
- 11. Some more questions • Data • Data transfer format • Rest (json/xml), graph, SOAP, File, Email, Queue, manual feed • Single / multiple payload • Synchronous / Asynchronous responses • Performance • Counts of requests • Size of the data payload • What is the system max parallel request limit • System transactions per second • Response times • System resilience • Security • Service authentication • Is the data confidential/federal • Error • How to react to errors, when & how to retry • Analytics and Reporting #technical #Worm’s-eye view Ask these questions to the 3rd party application technical team and analysts
- 12. Object level considerations - decisions • IDs or GUIDs • Rule of thumb - In Hub-Spoke integration model, the spoke systems should use the id/ GUID created by the master. This reduces a lot of maintenance problems that occur otherwise. • Date-Time • These should be in GMT always and the consuming systems or the client should convert them as needed. Use standard date time format- ISO 8601 • APIs • Export self-explanatory API objects • Huge flat structures should be always categorized in groups or nested objects, This makes the object look like a DOM/tree. • Version APIs • Paginate bulk web service calls. This solves performance issues. • Data Mapping • Define the role of the middleware. There are many misconceptions about what a middleware should do, correct those. • Outline roles of each system, so as to make sure there is a unified mapping strategy. • Avoid patch work mappings like using contains/ substring based decision making on strings. #technical #discussions #before the design
- 13. • Hardcode only if there is a technical limitation. • Always remember that complex data mapping often leads to a lot of serialization and de-serialization, in turn causing performance bottlenecks. • Orchestration • Answer the following questions. These help the consumers to know what is happening under the water • When to orchestrate • How to report the results back to the consuming systems. These updates could be in a form of status changes which should be reported • What happens when a part of a complete transaction set fails. Is the transaction type - ACID and what is the rollback policy. • Retries • Discuss and decide if the transaction should be retried and under what conditions. Usually the retry logic should be the same for a complete integration unless there is an exception • Error Reporting • This is a very crucial aspect and a good integration is one which is designed for failures. • List all the known failures based on integration type and decide the action to take. Some failures might need immediate attention while some can be retried for a certain period. Object level considerations - decisions #technical #discussions #before the design
- 14. Analysis Design Develop Test & Performance Manage Checkups Important Phases in Integration Building an integration is same as building a car - invest more on the design, so as worry less on the maintenance
- 15. • Analysis o Understand the roadmap of the integrating app o Identify Primary and secondary needs • Design o Reusable Patterns • FTP • Schedulers • PUB-SUB • Errors o Always think of performance • Throughput • Latency Analysis Design Develop Test Perform ance Manage & checkups Best Practices
- 16. o Secure o Reliable o Flexible o Error handling o Entity and relationship o Data exchange patterns • Synchronous vs asynchronous o Data in Chunks • Paginate o API data transfer should be in JSON. XML only if the consumer has a limitation. Best Practices Analysis Design Develop Test Perform ance Manage & checkups
- 17. • Develop o Handle all known cases o Mock the service first – consumers will not be surprised later o Write unit tests o Usable • Example - Document usage on the api console, avoid multiple documents o Follow Http web service guidelines: • POST - create a record or resource • PUT – update a record or resource • GET - read a record or resource(single or multiple) • DELETE - remove a record or resource(single or multiple) • A Response should be simple and meaningful, avoid duplicate responses • Http status 200 OK is more than enough for a consumer to know that is request was successful Best Practices Analysis Design Develop Test Perform ance Manage & checkups418 I'm a teapot
- 18. o Coding conventions • Java for internal code • Restful APIs § Naming conventions o Redesign if necessary o Ensure loose coupling for reusability o Code Reviews • Test & Performance o Automate o Load test with JMeter, Nodejs Simulators Best Practices Analysis Design Develop Test Perform ance Manage & checkups
- 19. • Manage oIdentify Risks – Impacting Systems oHandle changes with versioning • Regular checkups and adjustments oNode Capacity oFrequency Best Practices Analysis Design Develop Test Perform ance Manage & checkups
- 20. Circuit Breaker & BulkHead & How we use it
- 22. Attributes • IdentifierName BestPhotoCompanyCktBreaker • List of exception that would trip the circuit Exceptions java.util.concurrent.TimeoutException, java.net.SocketException • Window of time in ms to count the requestsRolling Window 100 • CountThreshold 4 • Ms time to close the circuit againTrip Timeout 7000
- 23. Response Timeout Trip Timeout 7000 ms Rolling Window 100 ms Timed out requests Circuit Open - All requests are being rejected Circuit breaker in action
- 24. Response Timeout Rolling Window 100 ms Timed out requests
- 25. Boat Hull
- 27. Thread Pool Executor 1 4 2 3 5 6 7 8 Proxy Hosting App 9 12 10 11 1. Send request to the ThreadPool if there are any free threads 2. Send request to the Queue is there no threads 3. Reject Any new requests Thread Pool Executor
- 28. Links & Resources • Netflix Open Source Software Center : https://netflix.github.io/ • How drawing can help : https://lifehacker.com/how-drawing-can-help-improve-your-memory-according-to-1772702367 • Rest : https://en.wikipedia.org/wiki/Representational_state_transfer • Rest APIs : http://www.restapitutorial.com/index.html • Http Methods : http://www.restapitutorial.com/lessons/httpmethods.html • Https Status Codes : http://www.restapitutorial.com/httpstatuscodes.html • JSON : http://www.json.org/ • Micro services : http://microservices.io/patterns/microservices.html • API Led connectivity : https://www.mulesoft.com/sites/default/files/resource-assets/API-led-connectivity-new-soa-updated.pdf • 418 – I’m a teapot : https://tools.ietf.org/html/rfc2324 • ISO 8601 - https://en.wikipedia.org/wiki/ISO_8601