Failure Self Defense: How to defend your App against failures in a (micro) services world.

1. Failure Self
Defense
1
Defend your App against failures in a (micro) services world

2. Tony Fabeen

3. APP

4. APP
CRM

5. APP
CRM
Email

6. APP
CRM
Email
Billing
Payments

7. APP
CRM
Email
Billing
Payments
Database
Search
Queue

8. APP
CRM
Email
Billing
Payments
Database
Search
Queue

10. WELCOME TO
THE
DEV PARTY

12. Database

13. Database
APPAPPAPPAPP

16. APPAPPAPPAPP

19. APP
Products
Orders
Payments

20. module Api
class Products
def initialize(host = nil)
@client = HTTPClient.new(host)
end
def all
@client.get('/products')
end
end
end

21. def send_checkout
params = { email: @email, token: @token, ssl_version: :SSLv3 }
RestClient.post(checkout_url, checkout_xml,
params: params,
content_type: "application/xml"){|resp, request, result| resp }
end

22. APP Orders
Payments

23. Orders
Payments
CASCADING
FAILURE

25. MAP DEPENDENCIES
THAT IMPACTS YOUR SYSTEM

26. Products
Orders
Payments
Catalog
Checkout
Pack
Route
Authorize
Charge
Feature Service

27. Products
Orders
Payments
Catalog
Checkout
Pack
Route
Authorize
Charge
Degraded
Down
Up
Feature Service

28. Products
Orders
Payments
Catalog
Checkout
Pack
Route
Authorize
Charge
Degraded
Down
Up
Feature Service

29. Products
Orders
Payments
Catalog
Checkout
Pack
Route
Authorize
Charge
Degraded
Down
Up
Feature Service

31. https://github.com/Shopify/toxiproxy

32. Toxiproxy.populate([{ name: 'redis',
listen: '127.0.0.1:22222',
upstream: '127.0.0.1:6379' }])

33. context 'when service UP' do
before { Cache.put('key', 'value') }
it 'saves value' do
expect(Cache.get('key')).to eq('value')
end
end
context 'when service DOWN' do
it 'will raises error' do
Toxiproxy[:redis].down do
expect { Cache.put('key', 'value') }.to raise_error(Redis::CannotConnectError)
end
end
end

34. context 'when service UP' do
before { Cache.put('key', 'value') }
it 'saves value' do
expect(Cache.get('key')).to eq('value')
end
end
context 'when service DOWN' do
it 'will raises error' do
Toxiproxy[:redis].down do
expect { Cache.put('key', 'value') }.to raise_error(Redis::CannotConnectError)
end
end
end

35. Fault Tolerance

36. An application with an average Response Time of 60ms can process
1.000 Requests Per Minute (RPM) per Thread.

37. An application with an average Response Time of 60ms can process
1.000 Requests Per Minute (RPM) per Thread.
How many Threads we need to handle 100.000 RPM of Throughput ?

38. 100

39. Imagine that 1% of the traffic timeout on a Service after 30 seconds,
the Response Time will raise to 360 ms.

40. Imagine that 1% of the traffic timeout on a Service after 30 seconds,
the Response Time will raise to 360 ms.
How many Threads we need to handle 100.000 RPM of Throughput ?

41. 600

42. Service
600 RPM
0.01 s
0.01 s
0.01 s
0.01 s
0.01 s
0.01 s

43. Service
60 RPM
0.01 s
0.01 s
0.10 s
0.10 s
0.10 s
0.10 s

44. Service
60 RPM
0.01 s
0.01 s
0.10 s
0.10 s
0.10 s
0.10 s
High Response Time
Less Throughput

45. Fail Fast

46. Fail Fast
Low timeouts

47. Fail Fast
Low timeouts
Connection timeout

48. Fail Fast
Low timeouts
Connection timeout
Socket Read timeout

49. Fail Fast
Low timeouts
Connection timeout
Socket Read timeout
Resource aquisition

50. Fail Gracefully

51. class Cache
def self.put(key, value)
service.set(key, value)
end
def self.get(key)
service.get(key)
end
end
end

52. class Cache
def self.put(key, value)
service.set(key, value)
end
def self.get(key)
service.get(key)
end
end
end
Cache.put('key', 'value')
Cache.get('key')

54. def put(key, value)
service.set(key, value)
true
rescue Redis::CannotConnectError => error
AwesomeLogger.log(error)
false
end

55. def get(key, fallback_value = nil)
service.get(key)
rescue Redis::CannotConnectError => error
AwesomeLogger.log(error)
fallback_value
end
end

57. Don't try if you can't succeed

58. Circuit Breakers

59. Client
Circuit
Breaker
Service
Closed

60. Client
Circuit
Breaker
Closed

61. Client
Circuit
Breaker
Open
Error

62. Client
Circuit
Breaker
Open

63. Client
Circuit
Breaker
Open
Error

64. Client
Circuit
Breaker
Service
Closed

65. Slow services
High timeouts

66. Bulkheads

67. Feature
A
Bulkhead
2 available
Feature
C
Feature
B

68. Bulkhead
Feature
C
1 available
Feature
A
Feature
B

69. Feature
C
no requests
available
Feature
A
Feature
B
Bulkhead

70. Bulkhead
Feature
C
no requests
available
Feature
A
Feature
B
Error

72. Monitor Service Calls

73. Monitor Service Calls
Timeout rate Rejected call rate
Short circuit rate Failure/Success rate
Response Times

76. Summary
Know your dependencies
Improve your test suite
Fail Fast
Timeouts
Fail Gracefully
Fallbacks
Don't try if you can't succeed
Circuit Breakers and Bulkheads are friends
Monitor Service Calls
Notice problems

77. ?

78. https://github.com/tonyfabeen
https://twitter.com/tonyfabeen
https://linkedin.com/tonyfabeen