This document discusses using ToxiProxy to test network conditions and the resiliency of distributed applications. ToxiProxy allows developers to simulate network errors like latency, bandwidth limitations, and timeouts to test how an application responds. Distributed applications are dependent on network communication, so it is important for applications to gracefully handle network failures to provide a good user experience. ToxiProxy provides an API that allows developers to easily simulate failures and automate resiliency testing without modifying production environments.

1. Testing network
conditions with
ToxiProxy
Celso Crivelaro
@celsocrivelaro

2. Tech Lead @

3. Professor @

4. Past:
Monolithic
Applications

5. Today:
Distributed
Systems

6. External
API
Service
Server
App
DB
API
Mobile
Mobile
App
Web App

7. Share Resources
Communication
over TCP/IP

8. Network-
dependent Apps

10. https://engineering.riotgames.com/news/fixing-internet-real-time-applications-part-i

15. Network fails

16. “A Distributed System is one in which the
failure of a computer you didn’t even know
existed can render your own computer
unusable”
Leslie Lamport

18. Network Model

19. "Ozzy" Model?

21. 200.124.124.1
200.0.124.5
200.5.34.12
200.3.5.51
200.3.5.52

22. 200.124.124.1
200.0.124.5
200.5.34.12
200.3.5.51
200.3.5.52
200.124.124.1:80 -> 200.3.5.52:80

23. Networking
Problems

24. No connection No connection between parts
Latency
Timeout
Bandwidth
Slow Close
Time to packages go and return
Stop responding
Bandwidth size in Mb/s
Slow close connection

25. What are we
testing?

26. How your app
responds to
network errors

27. User perception to network problem
Business rule to the problem
Technical action to the problem

28. Resiliency

29. How to simulate?

30. Hard modify a staging env
Hard to automate
Low level tools -> For sysadmins
No APIs for Devs or QA

31. http://toxiproxy.io

32. TCP TCP
APP SERVICE

33. TCP TCP
APP SERVICE
Toxies
Latency
Bandwidth
...

34. TCP
TCP
TCP TCP
APP
SERVICE
2
SERVICE
1
HTTP API

36. How to automate

37. Some Solutions

38. Low timeout
Retries Patterns - Circuit Break
Fallbacks
Caching strategies

39. Resiliency Matrix

40. App 1
Database API
App 2
ZIP Checker External API
App 3
Unavailable
Unavailable
Unavailable
Unavailable
Degraded
Available
Available
Not Aplic.
Not Aplic.
Unavailable
Degraded
Degraded
If integration fails...

41. References

43. http://toxiproxy.io

44. https://engineering.shopify.com/17489072-building-and-testing-resilient-ruby-on-rails-applications

45. Thank you!
@celsocrivelaro
Slides in: http://crivelaro.me