1. Building Resilient Serverless Systems
with Non-Serverless Components
Jeremy Daly
CTO, AlertMe.news
@jeremy_daly

2. Jeremy Daly
• CTO at AlertMe.news
• Consultant that works with companies building in the cloud
• 20+ year veteran of technology startups
• Started migrating workloads to the cloud in 2009
• Blogger, open-source contributor, speaker
• Publish Off-by-none, a weekly newsletter about serverless
@jeremy_daly

3. Agenda
• What does it mean to be Serverless?
• Working with “less-than-scalable” RDBMS
• Using unreliable APIs
• Managing API quotas
• Other non-serverless components
@jeremy_daly

4. What does it mean to be Serverless?
• No server management
• Flexible scaling
• Pay for value
• Automated high availability
@jeremy_daly

5. What does it mean to be Serverless?
@jeremy_daly
ElastiCache
RDS
EMR Amazon ES
Redshift
Fargate
Anything “on EC2”Lambda Cognito Kinesis
S3 DynamoDB SQS
SNS API Gateway CloudWatch
AppSync IoT Comprehend
Serverless Managed Not Serverless
DocumentDB
(MongoDB)
Managed Streaming
for Kafka
Definitely

6. @jeremy_daly

7. Everything has limits!
• Reserved Concurrency 🚦
• FunctionTimeouts ⏳
• Memory Limits 🧠
• NetworkThroughput 🚰
Some components are better than others
@jeremy_daly
Know
Your
Limits

8. “I want my, I want my, I want my SQL”
~ Dire Straits

9. Simple ServerlessWeb Service
Client
API Gateway Lambda DynamoDB
@jeremy_daly
Highly Scalable Highly Scalable Highly Scalable

10. Simple ServerlessWeb Service
Client
API Gateway Lambda
@jeremy_daly
Highly Scalable Highly Scalable NotThat Scalable 😳
RDS
^
not so
RDBMS and FaaS don’t play nicely together:
• Concurrency model doesn’t allow connection pooling
• Limited number of DB connections available
• Recycled containers create zombies

11. Ways to Manage DB Connections
• Increase max_connections setting
• Limit concurrent executions
• Lower your connection timeouts
• Limit connections per username
• Close connection before function ends
@jeremy_daly
🤞
😡
⚠
🎲
😱
👎

12. BetterWays to Manage DB Connections
• Implement a good caching strategy
• Buffer events for throttling and durability
• Manage connections ourselves
@jeremy_daly
👎

13. hit
Implement a good caching strategy
Client API Gateway RDSLambda
Elasticache
Key Points:
• Create new RDS connections ONLY on misses
• Make sureTTLs are set appropriately
• Include the ability to invalidate cache
@jeremy_daly

14. Do you really need immediate feedback?
• Synchronous Communication ⏳
Services can be invoked by other services and must wait for a reply.
This is considered a blocking request, because the invoking service
cannot finish executing until a response is received.
• Asynchronous Communication 🚀
This is a non-blocking request. A service can invoke (or trigger)
another service directly or it can use another type of communication
channel to queue information.The service typically only needs to
wait for confirmation (ack) that the request was received.
@jeremy_daly

15. RDS
Buffer events for throttling and durability
Client API Gateway
SQS
Queue
SQS
(DLQ)
Lambda Lambda
(throttled)
ack
“Asynchronous”
Request
Synchronous
Request
@jeremy_daly
Key Points:
• SQS adds durability
• Throttled Lambdas reduce downstream pressure
• Failed events are stored for further inspection/replay
Limit the
concurrency to match
RDS throughput

16. Manage connections ourselves
1. Count open connections
2. Close connection if connection ratio threshold exceeded
3. Close sleeping connections with high time values
4. Retry connections with exponential back off
@jeremy_daly

17. Serverless MySQL
https://github.com/jeremydaly/serverless-mysql
@jeremy_daly

18. Count open connections
@jeremy_daly
Query the
processlist to get
the total number
of active
connections

19. Close connection if over ratio threshold
@jeremy_daly
If we exceed the
connection ratio
Calculate our timeout
Try to kill zombies
If no zombies,
terminate connection
Else, just try to kill
zombies

20. Close sleeping connections with high time values
@jeremy_daly
Query processlist for zombies
Kill zombies

21. Retry connections with exponential back off
@jeremy_daly
If error trying to connect
Retry with Jitter

22. Does this really work?
@jeremy_daly
• Aurora Serverless (2 ACUs)
• 90 connections available
• 1,024 MB of memory
• 500 users/sec for one minute
• Avg. response time was 41 ms
• ZERO ERRORS

23. We shouldn’t have to do this!
@jeremy_daly
Amazon
Aurora Serverless
Aurora Serverless
DATA API
Doesn’t solve the max_connections issue Still in PREVIEW, not ready for primetime

24. Third-Party APIs

25. Manage calls to third-party APIs
• Implement a good caching strategy
• Buffer events for throttling and durability
• Implement circuit breakers
@jeremy_daly

26. DynamoDB
Stripe API
The Circuit Breaker
Client API Gateway Lambda
Key Points:
• Cache your cache with warm functions
• Use a reasonable failure count
• Understand idempotency
Status
Check CLOSED
OPEN
Increment Failure Count
HALF OPEN
“Everything fails all the time.”
~WernerVogels
@jeremy_daly

27. What about quotas?
• Concurrency has no effect on frequency
• Stateless functions are not coordinated
• Step Functions would be very expensive
• Adding state wouldn’t prevent needless invocations
@jeremy_daly

28. Can we build a better system?
• 100% serverless
• Cost effective
• Scalable
• Resilient
• Efficient
• Coordinated
@jeremy_daly

29. Lambda Orchestrator
(concurrency 1)
The Lambda Orchestrator
DynamoDB
LambdaWorker
LambdaWorker
LambdaWorker
Concurrent Executions
of the SAME function
SQS (DLQ)
@jeremy_daly
CloudWatch Rule
(trigger every minute)
SQS QueueSQS (DLQ)
Status?
Gmail API
250 Quota Units
per minute

30. Distributing Events
Key Points:
• SNS has a “well-defined API”
• Decouples downstream processes
• Allows multiple subscribers with message filters
Client
SNS
“Asynchronous”
Request
ack
Event Service
@jeremy_daly
HTTP
SMS
Lambda
SQS
Email

31. Stripe API
@jeremy_daly
Distribute &Throttle
ack
SQS
Queue Lambda
(concurrency 25)
SNS
Topic
Client API
Gateway
Lambda
Order Service
total > $0
Key Points:
• SNS to SQS is “guaranteed” (100,010 retries)
• Filter events to selectively trigger services
• Manage throttling/quotas per service
RDS
SQS
Queue Lambda
(concurrency 10)
SMS Alerting Service
Twilio API
SQS
Queue Lambda
(concurrency 5)
Billing Service
status == ”order_complete”
Event Service

32. Other non-serverless components
• Managed Services
• Legacy Systems
• Our own serverless APIs
@jeremy_daly

33. Non-serverless components are inevitable
• Know the limits of your components
• Use a good caching strategy
• Embrace asynchronous processes
• Buffer and throttle events to distributed systems
• Utilize eventual consistency
@jeremy_daly

34. Things I’m working on…
Blog: jeremydaly.com
Newsletter: Offbynone.io
Lambda API: LambdaAPI.com
Podcast: ServerlessChats.com
GitHub: github.com/jeremydaly
Twitter: @jeremy_daly
@jeremy_daly

35. ThankYou!
Jeremy Daly
jeremy@jeremydaly.com
@jeremy_daly