Just for you: FREE 60-day trial to the world’s largest digital library.
The SlideShare family just got bigger. Enjoy access to millions of ebooks, audiobooks, magazines, and more from Scribd.
Cancel anytime.More Related Content
Related Books
Free with a 14 day trial from Scribd
Resilient Functional Service Design
- 1. Resilient Functional Service Design The usually forgotten parts of resilient software design Uwe Friedrichsen – codecentric AG – 2015-2017
- 2. @ufried Uwe Friedrichsen | uwe.friedrichsen@codecentric.de | http://slideshare.net/ufried | http://ufried.tumblr.com
- 3. What’s that “resilience” thing?
- 4. Business Production Availability
- 5. (Almost) every system is a distributed system Chas Emerick http://www.infoq.com/presentations/problems-distributed-systems
- 6. A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. Leslie Lamport
- 7. Failures in todays complex, distributed and interconnected systems are not the exception. • They are the normal case • They are not predictable
- 8. … and it’s getting “worse” • Cloud-based systems • Microservices • Zero Downtime • Mobile & IoT • Social Web à Ever-increasing complexity and connectivity
- 9. Do not try to avoid failures. Embrace them.
- 10. resilience (IT) the ability of a system to handle unexpected situations - without the user noticing it (best case) - with a graceful degradation of service (worst case)
- 11. Beware of the “100% available” trap!
- 12. Designing for resilience The pitfall
- 13. First, you learn about resilience …
- 14. Complement Core Detect Prevent Recover Mitigate Treat
- 15. Core Detect Treat Prevent Recover Mitigate Complement Supporting patterns Redundancy Stateless Idempotency Escalation Zero downtime deployment Location transparency Relaxed temporal constraints Fallback Shed load Share load Marked data Queue for resources Bounded queue Finish work in progress Fresh work before stale Deferrable work Communication paradigm Isolation Bulkhead System level Monitor Watchdog Heartbeat Acknowledgement Either level Voting Synthetic transaction Leaky bucket Routine checks Health check Fail fast Let sleeping dogs lie Small releases Hot deployments Routine maintenance Backup request Anti-fragility Diversity Jitter Error injection Spread the news Anti-entropy Backpressure Retry Limit retries Rollback Roll-forward Checkpoint Safe point Failover Read repair Error handler Reset Restart Reconnect Fail silently Default value Node level Timeout Circuit breaker Complete parameter checking Checksum Statically Dynamically Confinement
- 16. ... then, you digest the stuff just learned
- 17. Confinement Core Detect Treat Prevent Recover Mitigate Complement Supporting patterns Redundancy Stateless Idempotency Escalation Zero downtime deployment Location transparency Relaxed temporal constraints Fallback Shed load Share load Marked data Queue for resources Bounded queue Finish work in progress Fresh work before stale Deferrable work Communication paradigm Isolation Bulkhead System level Monitor Watchdog Heartbeat Acknowledgement Either level Voting Synthetic transaction Leaky bucket Routine checks Health check Fail fast Let sleeping dogs lie Small releases Hot deployments Routine maintenance Backup request Anti-fragility Diversity Jitter Error injection Spread the news Anti-entropy Backpressure Retry Limit retries Rollback Roll-forward Checkpoint Safe point Failover Read repair Error handler Reset Restart Reconnect Fail silently Default value Node level Timeout Circuit breaker Complete parameter checking Checksum Statically Dynamically Oh, my! Theoretical blah! Uncool! Know that anyway for eons. So, let’s move on to the cool parts … Ah, now we’re talkin’! Here’s the cool stuff! That‘s practical, applicable. Don‘t you have more code examples? Or even better: Can‘t we turn that all into a live hacking session? Offline activities? Hmm, let‘s focus on the other stuff. Uh, sounds like one-off, tough stuff … Better start with the easier stuff, best with library support Yeah, more cool stuff! Aren‘t there more libs like Hystrix that we can drag into our projects with a line of configuration? Well, neat … I’ll come back to that stuff whenever I really need it
- 18. Core Detect Recover Mitigate Treat Prevent Complement Developer priority Relevance for application robustness Ye be warned! If you don’t get this part right, nothing else matters Here be dragons! This is extremely hard and poorly understood
- 19. Let’s recap …
- 20. The core parts are • extremely important • poorly understood • massively underestimated
- 21. Houston, we have a problem!
- 22. Let’s have a closer look at the core parts
- 23. Complement Core Detect Prevent Recover Mitigate Treat
- 24. Core Detect Treat Prevent Recover Mitigate Complement Isolation
- 25. Isolation • System must not fail as a whole • Split system in parts and isolate parts against each other • Avoid cascading failures • Foundation of resilient software design
- 26. Core Detect Treat Prevent Recover Mitigate Complement Isolation Bulkhead
- 27. Bulkheads are not about thread pools!
- 28. Bulkheads • Core isolation pattern (a.k.a. “failure units” or “units of mitigation”) • Diverse implementation choices available, e.g., µservice, actor, scs, ... • Implementation choice impacts system and resilience design a lot • Shaping good bulkheads is extremely hard (pure design issue)
- 29. Sounds easy. Where is the problem?
- 30. Service A Service B Request Due to functional design, Service A always needs backing from Service B to be able to answer a client request, i.e. the isolation is broken by design How do we avoid this …
- 31. Service Request Due to functional design we need to call a lot of services to be able to answer a client request, i.e. availability is broken by design ... and this ... Service Service Service Service Service Service Service Service Service Service Service Service
- 32. Mothership Service (a.k.a. Monolith) Request By trying to avoid the aforementioned issues we ended up with cramming all required functionality in one big service i.e. the isolation is broken by design ... without ending up with this?
- 33. Let’s use the well-known best practices • Divide & conquer a.k.a. functional decomposition • DRY (Don’t Repeat Yourself) • Design for reusability • Layered architecture • …
- 34. Unfortunately, …
- 35. Service A Service B Request Due to functional design, Service A always needs backing from Service B to be able to answer a client request, i.e. the isolation is broken by design ... this usually leads to this ...
- 36. Service Request Due to functional design we need to call a lot of services to be able to answer a client request, i.e. availability is broken by design ... and this ... Service Service Service Service Service Service Service Service Service Service Service Service
- 37. Mothership Service (a.k.a. Monolith) Request By trying to avoid the aforementioned issues we ended up with cramming all required functionality in one big service i.e. the isolation is broken by design ... and in the end also often to this.
- 38. Welcome to distributed hell!
- 39. Caches to the rescue!
- 40. Service A Service B Request Due to functional design, Service A always needs backing from Service B to be able to answer a client request, i.e. the isolation is broken by design CacheofB Break tight service coupling by caching data/responses of downstream service
- 41. Caches to the rescue?
- 42. Do you really think that copying stale data all over your system is a suitable measure to fix an inherently broken design?
- 43. We have to re-learn design for distributed systems!
- 44. A works-out-of-the-box-in-all-contexts, just-add-water-and-stir, three-bullet-point panacea for designing perfect bulkheads
- 45. You need lots of those …
- 46. ... maybe some of those
- 47. Then it is a lot of hard work …
- 48. ... and there is no silver bullet
- 49. Yet, a few guiding thoughts about bulkhead design …
- 50. Foundations of design • High cohesion, low coupling • Separation of concerns • Crucial across process boundaries • Still poorly understood issue • Start with • Understanding organizational boundaries • Understanding use cases and flows • Identifying functional domains (à DDD) • Finding areas that change independently • Do not start with a data model!
- 51. Short activation paths • Long activation paths affect availability • Increase latency and likelihood of failures • Minimize remote calls per request • Need to balance opposing forces • Avoid monolith à clear separation of concerns • Minimize requests à cluster functionality & data • Caches sometimes help, but stale data as trade-off
- 52. Dismiss reusability • Reusability increases coupling • Reusability leads to bad service design • Reusability compromises availability • Reusability rarely pays • Do not strive for reuse • Strive for replaceability instead
- 53. Broadening the options ...
- 54. Core Detect Treat Prevent Recover Mitigate Complement Isolation Communication paradigm Bulkhead
- 55. Communication paradigm • Request-response <-> messaging <-> events <-> … • Heavily influences resilience patterns to be used • Also heavily influences functional bulkhead design • Very fundamental decision which is often underestimated
- 56. µS Request/Response : Horizontal slicing Flow / Process µS µS µS µS µS µS Event-driven : Vertical slicing µS µS µS µS µS Flow / Process
- 57. Synchronous R/R vs. asynchronous events • Decomposition • Vertically divide-and-conquer vs. horizontally go-with-the-flow • Coordination • Coordination logic/services and orchestration vs. event chains and choreography • Transactions • Built-in transaction handling vs. external supervision • Error handling • Built into service vs. escalation/supervision strategy • Separation of concerns • Multiple responsibilities service vs. single responsibility services • Encapsulation • Domain logic distributed across services vs. domain logic in one place • Reusability vs. Replaceability • Complexity • A draw …
- 58. The communication paradigm influences the functional service design a lot and also the resilience patterns to be used
- 59. Example: order fulfillment • Simple order, credit card, non-digital items • Add coupons incl. validation • Add promotions incl. usage notification • Add bonus card incl. purchase notification • Customer accounts as payment type • PayPal as payment type • Integrate digital music library • Integrate digital video library • Integrate e-book library
- 60. Design exercise – Part 1 Create a bulkhead design for the case study • Use one communication paradigm • Synchronous request/response (e.g., REST) • Asynchronous messaging (e.g., Akka) • Asynchronous events (e.g., Pub/Sub) • Assume incremental requirements • How many services do you need to touch • What about the functional isolation of the services • How big/maintainable are the resulting services • Take a few notes
- 61. Online Shop Checkout Credit Card Provider Warehouse System Coupon Management Campaign Management PayPal Loyalty Management Accounts Receivables Music Library E-Book Library Video Library E-Mail Server Customer pressed “Buy now” ?
- 62. Order Fulfillment Service Online Shop Payment Service Credit Card Provider Shipment Service Warehouse System <Foreign Service> <Own Service> Coupon Management Promotion Campaign Management Loyalty Account Service Payment Provider PayPal Loyalty Management Accounts Receivables Music Library E-Book Library Video Library E-Mail Server Coupon Credit Card Coordinate Warehouse Coordinate Assets Notify Cust. PayPal Coordinate
- 63. Order confirmed Online Shop Credit Card Provider Warehouse System <Foreign Service> <Own Service> Coupon Management Campaign Management Account service Credit Card Service Loyalty Management Accounts Receivables Music Library E-Book Library Video Library E-Mail Server PayPal PayPal Service Warehouse Service Promotion Service Bonus Card Service Coupon Service Music Library Service Video Library Service E-Book Library Service Notification Service Payment authorized Digital asset provisioned Payment failed <Event> Order fulfillment supervisor Track flow of events Reschedule events in case of failure Services are responsible to eventually succeed or fail for good, usually incorporating a supervision/escalation hierarchy for that
- 64. Do not limit your design options upfront without an important reason
- 65. Wrap-up • Today’s systems are distributed • Failures are not avoidable, nor predictable • Resilient software design needed • Bulkhead design is • crucial for application robustness • poorly understood • massively underrated • different from traditional design best practices • Communication paradigms broaden your bulkhead design options
- 66. We have to re-learn design for distributed systems
- 67. @ufried Uwe Friedrichsen | uwe.friedrichsen@codecentric.de | http://slideshare.net/ufried | http://ufried.tumblr.com