Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2bXLlBA. Luke Kosewski describes the internals and machinations of Flow, how a similar approach may add value to a microservice architecture, and what preconditions must be met for such a recovery mechanism to succeed. He gives a real-life war story of a 2015 Q4 outage to demonstrate how this approach can turn the threat of a nasty disruption into a harmless, routine event. Filmed at qconnewyork.com. Luke Kosewski currently works as a Senior Software Engineer and a founding member of the Traffic & Chaos team at Netflix. He built "Flow", Netflix's traffic management service, to solve reliability issues at scale and continues to innovate on this, making Netflix more resilient to new threats.

1. Chaos Kong
Endowing Netflix with Antifragility
Luke Kosewski Traffic & Chaos Engineering

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
flow-microservice-antifragility

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. This is a Case Study We’ll Be Doing
TOGETHER

5. This is What AWS Failover Looks Like
us-west-2 us-east-1 eu-west-1

6. Failover is Run By This Guy
A Traffic Engineer

7. Failover is Run By This Guy
A Traffic Engineer

8. A Traffic Engineer’s Environment
● Netflix control plane

9. A Traffic Engineer’s Environment
● Netflix control plane
● Primarily in 3 AWS regions (EU, us-east-1, us-west-2)

10. A Traffic Engineer’s Environment
● Netflix control plane
● Primarily in 3 AWS regions (EU, us-east-1, us-west-2)
● They look like this:
us-west-2 us-east-1 eu-west-1

11. Casey Rosenthal
Traffic’s Teammates
Chaos
Intuition
traffic@netflix.com / chaos@netflix.com
Traffic
(management)
Lorin Hochstein, Aaron Blohowiak & Ali Basiri
Niosha Behnam & myself
Justin Reynolds

12. Our Relationship
Chaos Traffic
High Availability
Flow

13. Storytime with Luke
Once upon a time...
(August 2013)

14. 3 SREs at Netflix

15. 3 SREs at Netflix
10s of services

16. 3 SREs at Netflix
10s of services
100s of devs

17. Disaster

18. Active-Active

19. Opportunity

20. Flow

21. Fail Out of US-East-1: Case Study
➢ Outage!
➢ Scaling-up
➢ Proxying
➢ DNS design and cutover
➢ Improvisation

25. Fail Out of US-East-1: Case Study
➢ Outage!
➢ Scaling-up
➢ Proxying
➢ DNS design and cutover
➢ Improvisation

26. January 14, 2016
Stream Starts per Second – us-east region

27. January 14, 2016
Stream Starts per Second – us-east region

28. Fail Out of US-East-1: Case Study
➢ Outage!
➢ Scaling-up
➢ Proxying
➢ DNS design and cutover
➢ Improvisation

29. Diurnal Scaling

30. Y’all Ready for This?

31. What to Scale?

32. What to Scale?
● Anything absorbing incoming traffic

33. What to Scale?
● Anything absorbing incoming traffic
● Large stateless services

34. What to Scale?
● Anything absorbing incoming traffic
● Large stateless services
● Required stateful services (carefully)

35. That’s Better

36. How to Scale?

38. Two More Fallbacks
● “Time of Day” estimation

39. Two More Fallbacks
● “Time of Day” estimation
● largest observed value in the last 24h as
an intercept

40. How Much?

41. Ooze

42. Nimble

43. Fail Out of US-East-1: Case Study
➢ Outage!
➢ Scaling-up
➢ Proxying
➢ DNS design and cutover
➢ Improvisation

44. What do I mean by that?

45. Why We Proxy
Stream Starts per Second - EU

46. How do We Proxy?
Archaius dynamic properties –
regionally scoped
Zuul proxy with dynamic filters (Groovy)

47. Fail Out of US-East-1: Case Study
➢ Outage!
➢ Scaling-up
➢ Proxying
➢ DNS design and cutover
➢ Improvisation

48. Traditional DNS

49. Netflix’s DNS as a DB

50. Failover

51. Are We Done?
Stream Starts per Second - EU

52. Nope
Stream Starts per Second - EU

53. Fail Out of US-East-1: Case Study
➢ Outage!
➢ Scaling-up
➢ Proxying
➢ DNS design and cutover
➢ Improvisation

54. Recap: Proxying

55. Recap: Proxying

56. The Crowd Goes Wild
Stream Starts per Second - EU

57. This is What Success Feels Like

58. Positive Feedback Loop
The more we practice, the better and more daring we get

59. Other Takeaways

60. Thank You and Questions
Luke Kosewski – luke@netflix.com
Traffic & Chaos Engineering

61. Summary of NFLX github/techblog links
● Active/Active
http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html
http://techblog.netflix.com/2016/03/global-cloud-active-active-and-beyond.html
● Archaius
https://github.com/Netflix/archaius
http://techblog.netflix.com/2012/06/annoucing-archaius-dynamic-properties.html
● Zuul
https://github.com/Netflix/zuul
http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html
● SPS
http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-streaming.html

62. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/flow-
microservice-antifragility