With the rise in popularity of microservices architectures and, subsequently, the rising number of production services, it's paramount to develop resilient and observable services. This session will present some techniques that can help you test and validate services resiliency. With the help of the service mesh, we will also show how to apply common configuration across the whole mesh and explain how to read the numerous metrics collected by the mesh and detect anomalies sooner.

1. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 1
Peter Jausovec
@pjausovec
Path to Resilient and
Observable Microservices

2. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
Confidential – Oracle Internal/Restricted/Highly Restricted 2

3. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 3
About me
Peter Jausovec
• Consulting Member of Technical Staff at Oracle (Fn)
• Previously: Microsoft (Visual Studio, SharePoint, Azure, …)
• Wrote a couple of books
@pjausovec

4. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 4
Microservices

5. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 5
MONOLITH
1
3
2
4

6. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 6
1 32 4
1
1
1
2
2
2
2
3
3
3
3
4
4
4
4

7. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 7
Ability to provide and maintain an
acceptable level of service in the face of
faults and challenges to normal operation
Ability to recover from failures and
continue to function

8. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 8
After failure occurs,
return the service to a
fully functioning state

9. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 9
High Availability
🏥 Healthy
🛌 No (significant) downtime
🛌 Responsive
📈 Meeting SLAs

10. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 10
Disaster Recovery
HA design can’t handle the impact of faults
Data backup
Archiving
🛌

11. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 11
Path to Resiliency
Understand the requirements
Define service availability
Design for resiliency
Strategies for detection & recovery
Testing
Monitoring

12. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 12
Resiliency Strategies
Load balancing
Timeouts and retries
Circuit breakers and bulkhead pattern
Data replication
Graceful degradation
Rate limiting

13. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 13
How to do it?

14. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 14
A dedicated infrastructure layer for managing service-to-
service communication to make it manageable, visible
and controlled.

15. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 15
Resiliency Strategies
Load balancing
Timeouts and retries
Circuit breakers
Rate limiting

16. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 16
Testing for Resiliency
Test
Measure
Analyze (fix the issues)

17. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 17
SERVICE MESHTesting for Resiliency
💉 Inject failures:
⏱ Delays
🚨 Faults

18. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 18
Observability is the act of measuring, collecting
and analyzing metrics, traces, logs, events, … from
services
instrument  collect & store  analyze, visualize and
alert

19. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 19
Observability
Use correlation IDs
Structure logging
No private information!
Unique metric names
Categorize log entries
Use common format

20. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 20
SERVICE MESHObservability
Grafana
Jaeger
Kiali
ELK (Elasticsearch + Fluentd + Kibana)
PagerDuty

21. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 21
Demo
WEATHER
CITY
TEMPERATURE{
"city":"Redmond, WA",
"temperature":"60˚F",
"zipCode":"98052”,
“windDirection”: “NW”,
“windspeed”: “5 mph”
}
GET /weather/98052
response

22. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 22
Thank you!
@pjausovec
https://peterj.dev