Making S3 file service resilient to regional outages using Lambda@Edge

1. Making S3 more resilient
using Lambda@Edge
Júlia Biró
Contentful
@nellgwyn21

2. Where were you on February 28, 2017?
@nellgwyn21

3. Júlia, what is the
problem you are
trying to solve?
Velocity, I want
multi-region
active-active file
service on S3
@nellgwyn21

4. Why me?
Júlia Biró
Reliability Engineer
@nellgwyn21
Operating highly available
APIs with contractual SLAs

5. S3 file delivery infrastructure
@nellgwyn21

6. Go multi-region
@nellgwyn21
to limit blast radius

7. Failover can easily fail
● Detection
@nellgwyn21
● Propagation
● Switchback

8. Active-active is better with complex stacks
● Scaling
@nellgwyn21
● Health

9. GET /puppy.jpg HTTP/1.1
Host: examplebucket.s3-us-west-
2.amazonaws.com
Date: Mon, 11 Apr 2016 12:00:00 GMT
x-amz-date: Mon, 11 Apr 2016 12:00:00
GMT
Authorization: authorization string
@nellgwyn21
DNS round-robin

10. Modify requests using Lambda@Edge
+
@nellgwyn21
Original idea: for A/B testing

11. Which edge?
@nellgwyn21

12. eu-west-1
dns.resolveCname('files-origin.contentful.com', (function(err, result) {
if (result[0].includes('us-east-1')) {
bucketName = 'cf-files.s3.us-east-1.amazonaws.com';
region = 'us-east-1';
} else {
bucketName = 'cf-files.s3.eu-west-1.amazonaws.com';
region = 'eu-west-1';
}
request.origin.s3.region = region;
request.origin.s3.domain = bucketName;
request.headers['host'] = [{key: 'host', value: bucketName}];
}))
Proof of concept
@nellgwyn21

13. And then we are done, right?
@nellgwyn21
WRONG.

14. undefined
solution != implementation
@nellgwyn21

15. What makes us trust our current systems?
version
control
dash-
boards
@nellgwyn21

16. Gut feelings are valuable signals
@nellgwyn21@nellgwyn21

17. Production-readiness
criteria
Production readiness criteria
for reducing uncertainty
● Ownership
● Infrastructure as code
● Testability
● Routing
● Deployment and rollback
● Observability and operability
● Monitoring and alerts
● Reliability
● Outages
● Security
Abstract criteria status
Config change deployed in <5
minutes
New config can be triggered
without downtime
Deployment progress can be
monitored
@nellgwyn21

18. Production-readiness
criteria
PRC: Make the abstract specific
Abstract criteria status specific criteria notes
Config change deployed in <5
minutes
unverified Empirically observed
to comply
New config can be triggered
without downtime
yes Lambda is triggered
on each request, so
there is no such thing
as an unavailable
lambda
Deployment progress can be
monitored
yes Via CloudFront APIs
GetDistribution
method
@nellgwyn21

19. Production-readiness list is an ideal
RUNNIN' RHINO
Design by Allan Faustino
@nellgwyn21
The goal is to
increase confidence

20. Production-readiness
criteria
Specific decisions we made
@nellgwyn21
● Logs remain in CloudWatch
● We accept the risk of having no SLAs
● We deploy the code with Terraform
SLA

21. Expect to meet the unexpected
@nellgwyn21

22. MR active-active file delivery infrastructure
@nellgwyn21

23. What else could run on Lambda@Edge?
<image_uri>?w=600&r=75&fit=pad&bg=rgb:4b2
Request
The Contentful Images API
More code, more traffic, more risk.
@nellgwyn21

24. Use TypeScript
JavaScript
@nellgwyn21

25. Operational experience is part of production
readiness
@nellgwyn21

26. Development & release: 5 things that happened
@nellgwyn21
● Contribute to typescript library
● Ensure transpiled code matches deployed code
● Backport tooling
● Code review
● Rollbacks

27. Life with Lambda@Edge — the sun
@nellgwyn21
● Fast
● Scalable
● Cheap

28. Life with Lambda@Edge — the shade
@nellgwyn21
logs testing deployment

29. Risk-aversion is an asset
@nellgwyn21

30. One year later
@nellgwyn21

31. Literature
Orignal idea
● https://aws.amazon.com/blogs/networking-and-content-delivery/dynamically-route-viewer-requests-to-any-origin-using-
lambdaedge/
● Using Amazon CloudFront with Multi-Region Amazon S3 Origins
● Amazon S3 Region Failover — Part 2: CloudFront S3 origin failover
Failover
● Failover solution https://medium.com/cloudaholics/highly-available-multi-region-s3-website-cloudfront-distributions-29fe07195a3
● Cloudfront origin failover: Amazon CloudFront announces support for Origin Failover
Python 3 in Lambda@Edge
● Python for Lambda@edge https://aws.amazon.com/about-aws/whats-new/2019/08/lambdaedge-adds-support-for-python-37/
Multregion:
● AWS best practices https://www.youtube.com/watch?v=2e29I3dA8o4
@nellgwyn21

32. @nellgwyn21
Takeaways
Lambda@Edge is a cheap, scalable & highly reliable
platform for stateless APIs
Use production readiness lists to reduce risk
Trust your gut feelings, turn them into actions

33. Thank you!
Questions?
@nellgwyn21
Please rate this session
on the O’Reilly website
or in the conference app