My talk at Confoo 2016 Montreal It is well said that "The more you sweat on the field, the less you bleed in war". Failures are an inevitable part of complex systems. Accepting that failures happen, will help you design the system's reactions to specific failures. This talks on best practices for building resilient, stable and predictable services: preventing Cascading failures, Timeouts pattern, Retry pattern,Circuit breakers and many more techniques in microservices

1. Expect the unexpected:
Anticipate and prepare for failures
in micro services
Bhak&
Mehta
@bhak&_mehta

2. Introduc&on
• Senior
So7ware
Engineer
at
Blue
Jeans
Network
• Worked
at
Sun
Microsystems/Oracle
for
13
years
• CommiGer
to
numerous
open
source
projects
including
GlassFish
Applica&on
Server

3. My
recent
book

4. Previous
book

5. Blue
Jeans
Network

6. Blue
Jeans
Network
• Video
conferencing
in
the
cloud
• Customers
in
all
segments
• Millions
of
users
• Interoperable
• Video
sharing,
Content
sharing
• Mobile
friendly
• Solu&ons
for
large
scale
events

7. What
you
will
learn
• Microservices
architecture
• Challenges
at
scale
• Lessons
learned,
&ps
and
prac&ces
to
prevent
cascading
failures
• Resilience
planning
at
various
stages
• Real
world
examples

8. Customer B
Top
level
architecture
INTERNET
Customer A
SIP, H.323
HTTP / HTTPS
Media Node
Web
Server
Middleware
services
Cache
Service
discovery
Messaging
DB
Proxy
layer
Connector
Node

9. Micro
services
architecture

10. Path
to
Micro
services
• Advantages
– Simplicity
– Isola&on
of
problems
– Scale
up
and
scale
down
– Easy
deployment
– Clear
separa&on
of
concerns
– Heterogeneity
and
polyglo&sm

11. Microservices
• Disadvantages
– Not
a
free
lunch!
– Distributed
systems
prone
to
failures
– Eventual
consistency
– More
effort
in
terms
of
deployments,
release
managements
–
Challenges
in
tes&ng
the
various
services
evolving
independently,
regression
tests
etc

12. Monoliths
to
Micro
services

13. Resilient
system
• Processes
transac&ons,
even
when
there
are
transient
impulses,
persistent
stresses
• Func&ons
even
when
there
are
component
failures
disrup&ng
normal
processing
• Accepts
failures
will
happen
• Designs
for
crumple
zones

14. Kinds
of
failures
• Challenges
at
scale
• Integra&on
point
failures
–
Network
errors
– Seman&c
errors.
– Slow
responses
– Outright
hang
– GC
issues

17. An&cipate
failures
at
scale
• An&cipate
growth
• Design
for
next
order
of
magnitude
• Design
for
10x
plan
to
rewrite
for
100x

19. Resiliency
planning
Stage
1
• When
developing
code
– Avoiding
Cascading
failures
• Circuit
breaker
• Timeouts
• Retry
• Bulkhead
• Cache
op&miza&ons
– Avoid
malicious
clients
• Rate
limi&ng

20. Resiliency
planning
Stage
2
• Planning
for
dealing
with
failures
before
deploy
– load
test
– a/b
test
– longevity

21. Resiliency
planning
Stage
3
• Watching
out
for
failures
a7er
deploy
– health
check
– metrics

23. Cascading
failures
Caused
by
Chain
reac&ons
For
example
One
node
in
a
load
balance
group
fails
Others
need
to
pick
up
work
Eventually
performance
can
degenerate

24. Cascading
failures
with
aggrega&on

25. Cascading
failure
with
aggrega&on

27. Timeouts
• Clients
may
prefer
a
response
–
failure
–
success
–
job
queued
for
later
All
aggrega&on
requests
to
microservices
should
have
reasonable
&meouts
set

28. Types
of
Timeouts
• Connec&on
&meout
– Max
&me
before
connec&on
can
be
established
or
Error
• Socket
&meout
– Max
&me
of
inac&vity
between
two
packets
once
connec&on
is
established

29. Timeouts
paGern
• Timeouts
+
Retries
go
together
• Transient
failures
can
be
remedied
with
fast
retries
• However
problems
in
network
can
last
for
a
while
so
probability
of
retries
failing

30. Timeouts
in
code
In
JAX-­‐RS
Client client = ClientBuilder.newClient();
client.property(ClientProperties.CONNECT_TIMEOUT, 5000);
client.property(ClientProperties.READ_TIMEOUT, 5000)

31. Retry
paGern
• Retry
for
failures
in
case
of
network
failures,
&meouts
or
server
errors
• Helps
transient
network
errors
such
as
dropped
connec&ons
or
server
fail
over

32. Retry
paGern
• If
one
of
the
services
is
slow
or
malfunc&oning
and
other
services
keep
retrying
then
the
problem
becomes
worse
• Solu&on
– Exponen&al
backoff
– Circuit
breaker
paGern

33. Circuit
breaker
paGern
Circuit
breaker
A
circuit
breaker
is
an
electrical
device
used
in
an
electrical
panel
that
monitors
and
controls
the
amount
of
amperes
(amps)
being
sent
through

34. Circuit
breaker
paGern
• Safety
device
• If
a
power
surge
occurs
in
the
electrical
wiring,
the
breaker
will
trip.
• Flips
from
“On”
to
“Off”
and
shuts
electrical
power
from
that
breaker

35. Circuit
breaker
• Neflix
Hystrix
follows
circuit
breaker
paGern
• If
a
service’s
error
rate
exceeds
a
threshold
it
will
trip
the
circuit
breaker
and
block
the
requests
for
a
specific
period
of
&me

36. Bulkhead

37. Bulkhead
• Avoiding
chain
reac&ons
by
isola&ng
failures
• Helps
prevent
cascading
failures

38. Bulkhead
• An
example
of
bulkhead
could
be
isola&ng
the
database
dependencies
per
service
• Similarly
other
infrastructure
components
can
be
isolated
such
as
cache
infrastructure

39. Rate
Limi&ng
• Restric&ng
the
number
of
requests
that
can
be
made
by
a
client
• Client
can
be
iden&fied
based
on
the
access
token
used
• Addi&onally
clients
can
be
iden&fied
based
on
IP
address

40. Rate
Limi&ng
• With
JAX-­‐RS
Rate
limi&ng
can
be
implemented
as
a
filter
• This
filter
can
check
the
access
count
for
a
client
and
if
within
limit
accept
the
request
• Else
throw
a
429
Error
• Code
at
hGps://github.com/bhak&-­‐mehta/samples/
tree/master/ratelimi&ng

41. Cache
op&miza&ons
• Stores
response
informa&on
related
to
requests
in
a
temporary
storage
for
a
specific
period
of
&me
• Ensures
that
server
is
not
burdened
processing
those
requests
in
future
when
responses
can
be
fulfilled
from
the
cache

42. Cache
op&miza&ons
Gelng
from
first
level
cache
Gelng
from
second
level
cache
Gelng
from
the
DB

43. Dealing
with
latencies
in
response
• Have
a
&meout
for
the
aggrega&on
service
• Dispatch
requests
in
parallel
and
collect
responses
• Associate
a
priority
with
all
the
responses
collected

44. Handling
par&al
failures
best
prac&ces
• One
service
calls
another
which
can
be
slow
or
unavailable
• Never
block
indefinitely
wai&ng
for
the
service
• Try
to
return
par&al
results
• Provide
a
caching
layer
and
return
cached
data

45. Asynchronous
PaGerns
• PaGern
to
deal
with
long
running
jobs
• Some
resources
may
take
longer
&me
to
provide
results
• Not
needing
client
to
wait
for
the
response

46. Reac&ve
programming
model
• Use
reac&ve
programming
such
as
CompletableFuture
in
Java
8,
ListenableFuture
• Rx
Java

47. Asynchronous
API
• Reac&ve
paGerns
• Message
Passing
– Akka
actor
model
• Message
queues
– Communica&on
between
services
via
shared
message
queues
– Websockets

48. Logging
• Complex
distributed
systems
introduce
many
points
of
failure
• Logging
helps
link
events/transac&ons
between
various
components
that
make
an
applica&on
or
a
business
service
• ELK
stack
• Splunk,
syslog
• Loggly
• LogEntries

49. Logging
best
prac&ces
• Include
detailed,
consistent
paGern
across
service
logs
• Obfuscate
sensi&ve
data
• Iden&fy
caller
or
ini&ator
as
part
of
logs
• Do
not
log
payloads
by
default

50. Best
prac&ces
when
designing
APIs
for
mobile
clients
– Avoid
chalness
– Use
aggregator
paGern

51. Resilience
planning
Stage
2
• Before
deploy
– Load
tes&ng
– Longevity
tes&ng
– Capacity
planning

52. Load
tes&ng
• Ensure
that
you
test
for
load
on
APIs
– Jmeter
• Plan
for
longevity
tes&ng

53. Capacity
Planning
• An&cipate
growth
• Design
for
handling
exponen&al
growth

54. Resilience
planning
Stage
3
• A7er
deploy
– Health
check
– Metrics
and
Monitoring
– Phased
rollout
of
features

56. Health
Check
• Memory
• CPU
• Threads
• Error
rate
• If
any
of
the
checks
exceed
a
threshold
send
alert

58. Metrics
• Response
&mes,
throughput
– Iden&fy
slow
running
DB
queries
• GC
rate
and
pause
dura&on
– Garbage
collec&on
can
cause
slow
responses
• Monitor
unusual
ac&vity
• Third
party
library
metrics
– For
example
Couchbase
hits
– atop

59. Metrics
• Load
average
• Up&me
• Log
sizes

60. Monitoring
Monitoring
server
Produc&on
Environment
CHECKS
ALERTS
Email

61. Monitoring
Stack
• Log
Aggrega&on
framework
Applica&on
• So7ware
analy&cs
tool
that
monitors
performance
OS
/
Applica&on
Code
• Collectd
/
Graphite
Network,
Server

62. Rollout
of
new
features
• Phasing
rollout
of
new
features
• Have
a
way
to
turn
features
off
if
not
behaving
as
expected
• Alerts
and
more
alerts!

63. Real
&me
examples
• Neflix's
Simian
Army
induces
failures
of
services
and
even
datacenters
during
the
working
day
to
test
both
the
applica&on's
resilience
and
monitoring.
• Latency
Monkey
to
simulate
slow
running
requests
• Wiremock
to
mock
services
• Saboteur
to
create
deliberate
network
mayhem

64. Takeaway
• Inevitability
of
failures
– Expect
systems
will
fail
– Failure
preven&on

66. References
• hGps://commons.wikimedia.org/wiki/File:Bulkhead_PSF.png
• hGps://en.wikipedia.org/wiki/Circuit_breaker#/media/
File:Four_1_pole_circuit_breakers_fiGed_in_a_meter_box.jpg
• hGps://www.flickr.com/photos/skynoir/
Beer
in
hand:
skynoir/Flickr/Crea&ve
Commons
License

67. Ques&ons
• TwiGer:
@bhak&_mehta
• Email:
bhak&@bluejeans.com