Doppa2020 observable microservices make monitoring simple
•
0 likes•354 views
Had an opportunity to present on observability in Doppa2020 virtual conference. Very informative, especially in the Testing & Devops fields.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Doppa2020 observable microservices make monitoring simple
Similar to Doppa2020 observable microservices make monitoring simple (20)
Recently uploaded
Recently uploaded (20)
Doppa2020 observable microservices make monitoring simple
- 3. Observable Microservices Make Monitoring simple
- 5. Microservices Enterprise Message Bus Products Service Orders Service Inventory Service Payment Service UI Application API GW
- 6. The Transformation Comes with a cost !
- 7. An incident... Where to start with? What to search for? Enterprise Message Bus Products Service Orders Service Payment Service UI Application API GW Inventory Service
- 8. Diagnose #OrderService-dd-mm-yyyy.log 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) #InventoryService-dd-mm-yyyy.log 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151)
- 9. Log analysis #OrderService-dd-mm-yyyy.log 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) #OrderService-dd-mm-yyyy.log 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) #OrderService-dd-mm-yyyy.log 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) 05:05:08 2020-02-16 05:05:08 DEB UG 876 54 GET /orderservice/orders/987765?abc=xyz&qwe=def 5 0 0 1 7 Apache-HttpClient/4.5.2(Java/1.8.0-151) 05:05:08 2020-02-16 05:05:08 DEBUG 87654 GET /orderservice/orders/987765?abc=xyz&qwe=def 500 17 Apache-HttpClient/4.5.2(Java/1.8.0-151) Timestamp Log-Level Process-Id Requested URL HTTP Code Response Size User Agent
- 10. The Scope It is out-of-scope! “ Order Invoice Inventory Payment MailingShipping PayU PayTm GPay BHIM PayPalUPI MktPlace Webhook PUSH Gateway What I maintain What I am responsible for
- 11. Is it Possible Rely only on Logs Attach Debugger to whole Production Env
- 12. Observability
- 13. Speed Limiter
- 14. Load Balancer Request Payload Load Balancer app1 instance#1 app1 instance#2 Request Payload app1 instance#3 Health check pings
- 15. Auto Scaling Request Payload app1 instance#1 app1 instance#2 Request Payload Load Balancer Utilization metrics Threshold >65% Monitor Desired capacity app1 instance#3 app1 instance#4 app1 instance#5 Scale-out max capacity Utilisation 66%
- 16. Is it APM ? APM HOW is the application performing... about bottlenecks, access speeds, downtime, connectivity etc Telemetry OBSERVABILITY WHY is failing... how telemetry is used to diagnose quickly and fix the problem in context Not a telemetry
- 18. From data... Distributed TracingMonitor Vitals Event Driven RetryFail Fast & Fallback
- 21. BooksInventoryService Payment Service Not Event Driven API Driven OrderServiceAppln OrderService Order Item1 REST API: Get Stock Count(bookId)2 RESTAPI Pay(OrderDetails) 3 PaymentResponse 5 Order Confirmation 6 4 PayUsingGW(Amount)
- 22. BooksInventoryService REST API: Get Stock Count(bookId) RESTAPI Pay(OrderDetails) Please try later ! Order Item Payment Service OrderServiceAppln OrderService 1 2 3 4 X Intermittent Failure Not Event Driven API Driven
- 23. BooksInventoryService REST API: Get Stock Count(bookId) Order Item Payment Service Event Driven OrderServiceAppln OrderService 1 2 Order Confirmation 8 5 PayUsingGW(Amount) Place Order3 4 ORDERS PAYMENTS6 7 Payment_ Processed 3
- 24. BooksInventoryService REST API: Get Stock Count(bookId) Order Confirmation Order Item Payment Service Event Driven OrderServiceAppln OrderService 1 2 5 8 PayUsingGW(Amount) Place Order3 4 ORDERS 6 PAYMENTS 7 Payment_ Processed Storage#1 Storage#2 X Intermittent Failure? 3
- 25. Fail Fast & Fallback BooksInventoryService OrderService Get Similar Items(item) Get Similar Items(item) BooksInventoryService OrderService Get Similar Items(item) Get Similar Items(item) 4
- 26. Circuit Breaker State Metrics OrderService BooksInventoryService Fail Fast & Fallback Circuit Breaker State Metrics BooksInventoryService OrderService cache 4
- 27. Retry BooksInventoryService Retry Thrice, at Fixed Interval X Delay - 2s X X Delay - 2s X Delay - 2s OrderServiceService Load Cache for Similar Items 5 cache
- 28. Retry BooksInventoryService Load Cache for Similar Items X Delay - 2s X Delay - 4s (= 2 X 2 ^ 1) X Delay - 8s (= 2 X 2 ^ 2) Retry Thrice, with Exponential Backoff cache 5 with e^ backoff OrderServiceService
- 29. From data... Hence Simplified ! Distributed TracingMonitor Vitals RetryFail Fast & Fallback Event Driven DEMO
- 30. Observable Microservices... Effective Logging & Tracing Monitor only Vital Telemetry Establish Observability as Coding Culture Continual Investment in Observability
- 31. When an incident occurs, Software engineers need not be on call for autorecoverables issues in their services Engineers need not open 6 different tabs to look at most of the issues as they are self-healed/ redirected by the system Observable Microservices... Benefits
- 32. Software Engineers & Site Reliability Engineers need fault- tolerant observable microservices to make their Monitoring & Debugging simple Observable Microservices... Conclusion