1. Patterns and Practices
for building resilient
serverless applications
presented by Yan Cui
@theburningmonk

2. @theburningmonk theburningmonk.com

3. @theburningmonk theburningmonk.com
“the capacity to recover quickly from difficulties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun

4. @theburningmonk theburningmonk.com
“the capacity to recover quickly from difficulties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun
it’s not about
preventing failures!

5. everything fails, all the time

6. @theburningmonk theburningmonk.com
we need to build applications that can withstand failures

7. @theburningmonk theburningmonk.com

8. @theburningmonk theburningmonk.com
don’t run your application on one server…

9. @theburningmonk theburningmonk.com
entire data centers can
go down…

10. @theburningmonk theburningmonk.com
run your application in multiple AZs and regions

11. @theburningmonk theburningmonk.com
Failures on load: exhaustion of resources

12. @theburningmonk theburningmonk.com
Failures on load: exhaustion of resources

13. @theburningmonk theburningmonk.com
latency
reqs/s
Failures on load: exhaustion of resources
CPU saturation

14. @theburningmonk theburningmonk.com
Failures in distributed systems
Service A Service B Service C
user

15. @theburningmonk theburningmonk.com
Failures in distributed systems
Service A Service B Service C
user

16. @theburningmonk theburningmonk.com
Failures in distributed systems
Service A Service B Service C
user
HTTP 502

17. @theburningmonk theburningmonk.com
Failures in distributed systems
Service A Service B Service C
user
You suck!

18. @theburningmonk theburningmonk.com
microservices death stars circa 2015

19. Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

20. Yan Cui
http://theburningmonk.com
@theburningmonk
http://bit.ly/yubl-serverless

21. Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

23. Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advisetraining delivery

24. by Uwe Friedrichsen

25. @theburningmonk theburningmonk.com
Lambda execution environment

26. @theburningmonk theburningmonk.com
Serverless - multiple AZ’s out of the box
Total resources created:
1 API Gateway
1 Lambda

27. @theburningmonk theburningmonk.com
Serverless - multiple AZ’s out of the box
Total resources created:
1 API Gateway
1 Lambda
don’t pay for idle
redundant resources!

28. @theburningmonk theburningmonk.com
Load balancing

29. @theburningmonk theburningmonk.com
Data replication in different AZ’s
DynamoDB
Global Tables

30. @theburningmonk theburningmonk.com
There are throttling everywhere!

31. @theburningmonk theburningmonk.com
Beware of timeout mismatch
API Gateway
Integration timeout
Default: 29s
Lambda
Timeout
Max: 15 minutes

32. @theburningmonk theburningmonk.com
Beware of timeout mismatch
Lambda
Timeout
Max: 15 minutes
SQS
Visibility timeout
Default: 30s
Min: 0s
Max: 12 hours

33. @theburningmonk theburningmonk.com
Beware of timeout mismatch
Lambda
Timeout
Max: 15 minutes
SQS
Visibility timeout
Default: 30s
Min: 0s
Max: 12 hours
set VisibilityTimeout to
6x Lambda timeout

34. @theburningmonk theburningmonk.com
Offload computing operations to queues

35. @theburningmonk theburningmonk.com
Offload computing operations to queues

36. @theburningmonk theburningmonk.com
Offload computing operations to queues
better absorb
downstream problems

37. @theburningmonk theburningmonk.com
Offload computing operations to queues
need way to replay
DLQ events

38. https://www.npmjs.com/package/lumigo-cli

39. @theburningmonk theburningmonk.com
Offload computing operations to queues
great for fire-and-forget tasks

40. @theburningmonk theburningmonk.com
“what if the client is waiting for a response?”

41. @theburningmonk theburningmonk.com
“Decoupled Invocation”

42. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx <null>
… … …
task results
not ready…

43. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx <null>
… … …
task results
not ready…
202

44. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx <null>
… … …
task results
reporting for duty!

45. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx <null>
… … …
task results
working hard…
not ready…

46. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx <null>
… … …
task results
202
working hard…

47. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx { … }
… … …
task results
done!

48. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx { … }
… … …
task results
done!

49. @theburningmonk theburningmonk.com
task id created at result
xxx xxx <null>
xxx xxx { … }
… … …
task results
200
{ … }

50. @theburningmonk theburningmonk.com
wait…

51. @theburningmonk theburningmonk.com
a distributed
transaction!

52. @theburningmonk theburningmonk.com
a distributed
transaction!
needs rollback

53. @theburningmonk theburningmonk.com
how do you implement distributed transactions?

54. @theburningmonk theburningmonk.com
The Saga pattern
A pattern for managing failures where each action
has a compensating action for rollback

55. @theburningmonk theburningmonk.com
The Saga pattern
https://www.youtube.com/watch?v=xDuwrtwYHu8

56. @theburningmonk theburningmonk.com
The Saga pattern
Begin transaction
Start book hotel request
End book hotel request
Start book flight request
End book flight request
Start book car rental request
End book car rental request
End transaction

57. @theburningmonk theburningmonk.com
The Saga pattern
model both actions and
compensating actions as
Lambda functions

58. @theburningmonk theburningmonk.com
The Saga pattern
use Step Functions as the
coordinator for the saga

59. @theburningmonk theburningmonk.com
The Saga pattern
Input

60. @theburningmonk theburningmonk.com
The Saga pattern

61. @theburningmonk theburningmonk.com
The Saga pattern

62. @theburningmonk theburningmonk.com
The Saga pattern

63. @theburningmonk theburningmonk.com
no distributed
transactions

64. @theburningmonk theburningmonk.com
do the work here

65. @theburningmonk theburningmonk.com
retry-until-success

66. @theburningmonk theburningmonk.com

67. @theburningmonk theburningmonk.com
24 hours data retention

68. @theburningmonk theburningmonk.com
24 hours data retention
need alerting to ensure
issue are addressed quickly

69. @theburningmonk theburningmonk.com
Mind the poison message

70. @theburningmonk theburningmonk.com
retry-until-success
needs to deal with
poinson messages

71. @theburningmonk theburningmonk.com
Mind the poison message

72. @theburningmonk theburningmonk.com
Mind the poison message
6, 3, 1, 1, 1, 1, …

73. @theburningmonk theburningmonk.com
Mind the poison message
6, 3, 1, 1, 1, 1, …
only count the “same” batch

74. @theburningmonk theburningmonk.com
Mind the poison message

75. @theburningmonk theburningmonk.com
Mind the poison message
have to fetch
from the stream

76. @theburningmonk theburningmonk.com
Mind the poison message
have to fetch
from the stream
do it before they expire
from the stream!

77. @theburningmonk theburningmonk.com
Mind the partial failures
LambdaSQS

78. @theburningmonk theburningmonk.com
Mind the partial failures
LambdaSQS Poller

79. @theburningmonk theburningmonk.com
LambdaSQS Poller
Mind the partial failures
Delete

80. @theburningmonk theburningmonk.com
Mind the partial failures
LambdaSQS Poller
Error

81. @theburningmonk theburningmonk.com
Mind the partial failures
LambdaSQS Poller
Error
DLQ

82. @theburningmonk theburningmonk.com
Mind the partial failures
LambdaSQS Poller
Error
DLQ
batch fails as a unit

83. https://lumigo.io/blog/sqs-and-lambda-the-missing-guide-on-failure-modes
Mind the partial failures

84. @theburningmonk theburningmonk.com
Mind the partial failures

85. @theburningmonk theburningmonk.com
Mind the partial failures

86. @theburningmonk theburningmonk.com
Mind the partial failures

87. @theburningmonk theburningmonk.com
Mind the retry storm
Service A

88. @theburningmonk theburningmonk.com
Mind the retry storm
Service A

89. @theburningmonk theburningmonk.com
Mind the retry storm
Service A
retry
retry
retry
retry

90. @theburningmonk theburningmonk.com
Mind the retry storm
Service A

91. @theburningmonk theburningmonk.com
Mind the retry storm
Service A

92. @theburningmonk theburningmonk.com
Mind the retry storm
Service A

93. @theburningmonk theburningmonk.com
Mind the retry storm
Service A

94. @theburningmonk theburningmonk.com
retry storm

95. @theburningmonk theburningmonk.com
circuit breaker pattern
After X consecutive timeouts, trip the circuit

96. @theburningmonk theburningmonk.com
circuit breaker pattern
After X consecutive timeouts, trip the circuit
When circuit is open, fail fast

97. @theburningmonk theburningmonk.com
circuit breaker pattern
When circuit is open, fail fast
but, allow 1 request through every Y mins
After X consecutive timeouts, trip the circuit

98. @theburningmonk theburningmonk.com
circuit breaker pattern
When circuit is open, fail fast
but, allow 1 request through every Y mins
If request succeeds, close the circuit
After X consecutive timeouts, trip the circuit

99. @theburningmonk theburningmonk.com
where do I keep the state of the circuit?

100. @theburningmonk theburningmonk.com
in-memory
PROS
simplicity
no dependency on external service
CONS
takes longer & more requests to stop all traffic
new containers would generate more traffic

101. @theburningmonk theburningmonk.com
external service
PROS
minimizes no. of total requests to trip circuit
new containers respect collective decision
CONS
complexity
dependency on an external service

102. @theburningmonk theburningmonk.com
which approach should I use?
It depends. Maybe start with the simplest solution first?

103. @theburningmonk theburningmonk.com
multi-region, active-active

104. @theburningmonk theburningmonk.com
us-east-1
API Gateway Lambda DynamoDBRoute53

105. @theburningmonk theburningmonk.com
eu-west-1
us-east-1
us-west-1

106. @theburningmonk theburningmonk.com
eu-west-1
us-east-1
us-west-1
GlobalTable

107. @theburningmonk theburningmonk.com
eu-west-1
us-east-1
us-west-1
GlobalTable

108. @theburningmonk theburningmonk.com
eu-central-1
us-east-1
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
SNS
SNS

109. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
eu-central-1
us-east-1
SNS
SNS

110. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
eu-central-1
us-east-1
SNS
SNS

111. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
eu-central-1
us-east-1
SNS
SNS
Ddedupe

112. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
us-east-1
SNS
eu-central-1
SNS
eu-central-1
SQS Lambda DynamoDB Lambda API Gateway
Global Table

113. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
us-east-1
SNS
eu-central-1
SNS
eu-central-1
SQS Lambda DynamoDB Lambda API Gateway
Global Table

114. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
us-east-1
SNS
eu-central-1
SNS
eu-central-1
SQS Lambda DynamoDB Lambda API Gateway
Global Table

115. @theburningmonk theburningmonk.com
us-east-1
SQS Lambda DynamoDB Lambda API Gateway
us-east-1
SNS
eu-central-1
SNS
eu-central-1
SQS Lambda DynamoDB Lambda API Gateway
Global Table

116. @theburningmonk theburningmonk.com
Multi-region architecture - benefits & tradeoffs
Protection against
regional failures
Higher complexity Very hard to test

117. CHAOS ENGINEERING

118. MUST KILL SERVERS!
RAWR!!
RAWR!!

119. @theburningmonk theburningmonk.com
“the discipline of experimenting on a system in order to build confidence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org

120. @theburningmonk theburningmonk.com
“You don't choose the moment, the moment chooses you!
You only choose how prepared you are when it does.”
Fire Chief Mike Burtch

121. @theburningmonk theburningmonk.com

122. @theburningmonk theburningmonk.com
“what if DynamoDB has an elevated error rate?”

123. @theburningmonk theburningmonk.com
“what if service X has elevated latency?”

124. @theburningmonk theburningmonk.com
identify weaknesses before they manifest in system-wide, aberrant behaviors
GOAL

125. everything fails, all the time

126. @theburningmonk theburningmonk.com
“the capacity to recover quickly from difficulties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun

127. @theburningmonk theburningmonk.com

128. @theburningmonk theburningmonk.com
lambdabestpractice.com

129. https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

130. @theburningmonk
theburningmonk.com
github.com/theburningmonk