In this living room session, industry leaders from ecommerce, tech, and media discuss war stories from the edge, rehashing the biggest business and technological challenges they’ve faced and how they addressed them.

1. I’m a Failure!
(and so can you)
Kenton Jacobsen
Director of Engineering,
Vogue.com, Glamour.com
at Condé Nast

3. Everyone will make mistakes
Information is imperfect
Failure is rarely a single thing
Complex systems have complex failures
Sometimes there is a single point of failure
There shouldn’t be a single button / command / event that requires manual
intervention
Fundamental surprise
Failures happen in ways that cannot be predicted

5. Some things can only be observed in production
Keep fire escapes clear
Keep the cost of failure low
Limited blast radius
http://www.kitchensoap.com/2013/09/30/learning-from-failure-at-etsy/
https://codeascraft.com/speakers/john-allspaw-outages-post-mortems-and-human-error/
https://www.slideshare.net/kellan/continuous-deployment-7300057

6. Risk
Velocity
Scientific Graph

7. Fail fast,fix faster

8. Recovering from Failure with Fastly
@sethvargo

9. Open Source
fastly.com/open-source
✓ 14,243,908 downloads
✓ 549.9 TB bandwidth
✓ 62.6 GB/hour
✓ 80 requests/minute
✓ 96% cache coverage
@sethvargo

10. The dashboard not changing color is related to
S3 issue. See the banner at the top of the
dashboard for updates.
@awscloud
Amazon Web Services
2:17 PM - 28 Feb 2017
@sethvargo

11. www.terraform.io
@sethvargo

12. HTML
HTML
HTML
@sethvargo

13. PURGE-KEY
spider-all
@sethvargo

14. PURGE-KEY
spider-all
@sethvargo

15. Spider Failed
@sethvargo
We PURGED ALL'd
@sethvargo

16. Fastly Still Had the Data
💩Well

17. Fastly Still Had the Data
But Fastly still had the data

18. HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
@sethvargo

19. HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
139043
PURGE ALL =>
@sethvargo

20. HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
PURGE ALL =>
HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
139043
@sethvargo

21. PURGE ALL =>
HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
139043
HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
@sethvargo

22. PURGE ALL =>
HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
139043
HOSTNAME
terraform.io
URL
/index.html
###GENERATION###
139042
+ +
CACHE KEY
They did that
@sethvargo

23. Mitigating Future Failures
@sethvargo

24. Cache-Control
Surrogate-Control
Cache-Control
@sethvargo

25. Stale-If-Error
Stale-While-Revalidate
@sethvargo

26. $ curl
--request PURGE
--header "Fastly-Soft-Purge:1"
https://api.fastly.com/purge/service/ab12/purge/site-tf
@sethvargo

27. Thank you!
@sethvargo

28. Poisoning
caches
Niklas Gustavsson
ngn@spotify.com
@protocol7

30. Production storage

31. Client accesspoint
Production
storage
(nginx)
Proxy
Master
storage

32. Goggle Cloud Storage

36. Accept-Ranges: bytes

37. Client accesspoint
Production
storage
(nginx)
Proxy
Master
storage
(GCS)
🔥🔥🔥🔥 ✅ ✅ ✅☑