3. PAGE3
NaradaCode
Thomas Ivantoro Prasetyo
Solution Architect, Head of SME
IT Solution Architect Channel & Payment
IT Application Architect,
IT Development Supervisor
Squad Leader Sales Management System
Technology Innovation Manager, NaradaCode
thomas.ivantoro@naradacode.com
https://www.linkedin.com/in/thomasivantoro/
Previous Experience
4. PAGE4
NaradaCode
We provide end-to-end service to improve your digital capability
Lean Digital Transformation Company
Enterprise DevOps Lean Digital Governance Competency Development
Enterprise DevOps focuses on how
enterprise can leverage modern IT
and technology to drive business
faster and achieve the maximum
outcome
Lean Digital Governance ensure
enterprise has enough controls and
still ensure the speed of digital to
stay ahead of competition
Complete competency development
and mindset changing program to
upskill your people in order to
accelerate digital transformation
5. PAGE5
NaradaCode
DevOps Indonesia Meetup, May 05th 2020
Microservices Benefits and Complexities
Cascading Failure in Microservices
Strategy to Avoid Cascading Failures
Demo – Handling Cascading Failures
ThingstoDiscuss
7. PAGE7
NaradaCode
How to avoid cascading failure in Microservices
DevOps Indonesia Meetup, May 5th 2020
Microservices
Benefit &
Complexities
8. PAGE8
NaradaCode
Microservices–BenefitsandComplexities
DevOps Indonesia Meetup, May 05th 2020
Microservices provide benefits …but come with costs
Strong Module Boundaries
Microservices reinforce modular structure, which is
particularly important for larger teams.
⚠ Distribution
Distributed systems are harder to program, since
remote calls are slow and are always at risk of failure.
Independent Deployment
Simple services are easier to deploy, and since they are
autonomous, are less likely to cause system failures
when they go wrong. Especially if you have automation
pipeline.
⚠ Eventual Consistency
Maintaining strong consistency is extremely difficult for
a distributed system, which means everyone has to
manage eventual consistency.
Technology Diversity
With microservices you can mix multiple languages,
development frameworks and data-storage
technologies.
⚠ Operational Complexity
You need a mature operations team to manage lots of
services, which are being redeployed regularly.
Source: https://martinfowler.com/articles/microservice-trade-offs.html#boundaries
14. PAGE14
NaradaCode
How to avoid cascading failure in Microservices
DevOps Indonesia Meetup, May 5th 2020
Strategy to avoid
Cascading Failures
15. PAGE15
NaradaCode
HowtoAvoidCascadingFailuresinMicroservices
Circuit Breaker
A circuit breaker is an automatically
operated electrical switch designed to protect
an electrical circuit from damage caused by
excess current from an overload or short
circuit. Its basic function is to interrupt
current flow after a fault is detected. Unlike
a fuse, which operates once and then must
be replaced, a circuit breaker can be reset
(either manually or automatically) to resume
normal operation. (Wikipedia)
DevOps Indonesia Meetup, May 05th 2020
20. PAGE21
NaradaCode
HowtoAvoidCascadingFailuresinMicroservices
As mentioned in Design for Failure Principle, the Purpose is to:
• Isolate failure & isolate resources, to avoid failure to cascade
• Fail fast, degrade system gracefully with good fallback without sacrificing user
experiences
• Fail Fast, to save resources and network (save cost)
DevOps Indonesia Meetup, May 05th 2020
25. PAGE26
NaradaCode
HowtoAvoidCascadingFailuresinMicroservices
Benefits of using Sidecar:
– Extends functionalities without changing primary application
– Reduce complexity code on primary application
– Use of Polyglot language in microservices without code duplication
– Isolate resources from primary application
Microservice
A
Sidecar
Microservice
B
Sidecar
Microservice
C
Sidecar
Control Plane
- Circuit Breaker
- TCP & HTTP Pool
- Timeout
- Retries
- etc
DevOps Indonesia Meetup, May 05th 2020
26. PAGE27
NaradaCode
ManyOtherWays
There are many other ways to avoid cascading failures and to achieve
application resiliency or might be self healing:
Request Timeout
Retries
Fallback
Rate Limiter
Event-driven (CQRS, Event Sourcing)
…
DevOps Indonesia Meetup, May 05th 2020
27. PAGE28
NaradaCode
How to avoid cascading failure in Microservices
DevOps Indonesia Meetup, May 5th 2020
Demo on
Avoiding Cascading
Failures
29. PAGE30
NaradaCode
• Enhancement Book Reviews to add Review Ratings
• In this sample they do Canary Release with A/B Testing
V.1
V.2 V.3
V.1, Reviews without ratings.
User: All Users (except A, B, chaos)
V.2 , Ratings color is BLACK
User: A
V.3 , Ratings color is RED
User: B and chaos
Demo
31. PAGE32
NaradaCode
Scenario:
1. User (with name chaos) will trigger Fault Injection to simulate timeout (hanging) on
Reviews Service V.3
2. User chaos will make consecutive failures at least 3 times. Review Service V.3 have Circuit
Breaker with this configuration:
If there are three consecutive errors
(502, 503, 504) within 20 seconds,
Circuit Breaker will be open and
Review Service V.3 will be ejected
for 3 minutes.
Demo
32. PAGE33
NaradaCode
Expected Result
1. Circuit Breaker on Reviews Service V.3 will be opened automatically, stopping access to this
service.
2. Book Info page of user B, Book Reviews section will not be shown for 3 minutes.
3. Book Info page of user A, Book Reviews section will be shown same as before.
4. Book Info page of other users (or without login), Book Reviews section will be shown same
as before.
5. After 3 minutes, Circuit Breaker on Reviews Service V.3 will be closed automatically, Reviews
Service V.3 ready to serve.
6. Book Info page of user B, Book Reviews section will be shown same as before.
Demo
33. PAGE34
NaradaCode
How to avoid cascading failure in Microservices
DevOps Indonesia Meetup, May 5th 2020
Demo on
Avoiding Cascading
Failures
34. PAGE35
NaradaCode
PatternofResilience
Do not try to use all of them, use only if “Value > Cost + Risk“
Pattern of resilience by Uwe
Friedrichsen (CTO Codecentric)
DevOps Indonesia Meetup, May 05th 2020