At many high-growth companies, staying at the bleeding edge of innovation and maintaining the highest level of service availability often sideline financial efficiency. This problem is exacerbated in a micro-service environment, where decentralized engineering teams can spin up thousands of instances at a moment’s notice, with no governing body tracking cost. By developing a cost-conscious culture and assigning the responsibility for efficiency to the appropriate business owners, you can deliver innovation efficiently and cost effectively . At Netflix, the Finance and Operations Engineering teams bear the responsibility for ensuring that the rate of innovation is fast and that development is cost effective. In this presentation, we’ll explore the building blocks of AWS cost management and discuss Netflix’s best practices.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Andrew Park, Manager – Technology FP&A
10.7.15
Efficient Innovation
High Velocity Cost Management at Netflix

2. What to Expect from the Session
• Managing efficiency vs. innovation & availability
• How and why cloud mgmt. has changed at Netflix
• Best practices & future goals

3. The Efficiency Challenge
Netflix: world’s largest
subscription Internet TV
business
Business Stats:
>60m members
2,000+ employees
80+ countries
>100m hours watched per day
Strategy: innovation & availability
prioritized above efficiency
Engineering Stats:
1,400 Tech & Dev. engineers
40+ independent teams
500+ microservices
90,000+ instances (~15% autoscaling)

5. Cloud Cost Timeline
0
0.2
0.4
0.6
0.8
1
Aug-10
Oct-10
Dec-10
Feb-11
Apr-11
Jun-11
Aug-11
Oct-11
Dec-11
Feb-12
Apr-12
Jun-12
Aug-12
Oct-12
Dec-12
Feb-13
Apr-13
Jun-13
Aug-13
Oct-13
Dec-13
Feb-14
Apr-14
Jun-14
Aug-14
Oct-14
Dec-14
Feb-15
Apr-15
Jun-15
0
0.2
0.4
0.6
0.8
1
1.2
Aug-10
Oct-10
Dec-10
Feb-11
Apr-11
Jun-11
Aug-11
Oct-11
Dec-11
Feb-12
Apr-12
Jun-12
Aug-12
Oct-12
Dec-12
Feb-13
Apr-13
Jun-13
Aug-13
Oct-13
Dec-13
Feb-14
Apr-14
Jun-14
Aug-14
Oct-14
Dec-14
Feb-15
Apr-15
Jun-15
~Not 20x Growth
Dollars
(normalized)
Cost per stream
(normalized)
The Foundations Present Day The Dream

6. Our foundations
• Forming the Cloud Capacity Team
• Matching processes with strategy
• Developing transparency tools

7. The Who: Cloud Capacity Planners
“Serving customers, not keeping gates”

8. Main Responsibilities
Strategy & Operations
• Scalable cloud growth
• Ad-hoc internal consulting
• Capacity liaisons with AWS
Capacity Planning
• Purchasing capacity
• Planning with major teams
• Retroactive RI purchasing

9. Retroactive Reservation Purchasing
• Look back purchases of on-demand
• Bi-weekly process includes rebalancing unused RIs
• Purchasing considerations:

10. Implementing Retroactive Purchasing
• Assumptions & processes required:
• Understanding about infrastructure & usage by account
• Irregular growth to be communicated
• Robust usage dashboards & cost tools
• Benefits:
• Lowered deployment friction accelerating innovation
• Reduced operational overhead management
• Potential reduction of “capacity sandbagging”

11. On-demand in Aug’15
C*
scale-up
Edge service
migration
Global
memcache
replication

12. General Cloud Capacity Strategies
• Strat: Service oriented architecture @ massive scale
• Process: centralized cloud capacity planning function
• Strat: Unconstrained deployment capabilities
• Process: develop contextual efficiency information via tooling
• Strat: Improve overall availability
• Process: dedicated failover capacity, critical services on
general instance families

13. Transparency Through Tooling
• Invest in robust tooling for capacity team
• Reveal AWS usage & cost back to service teams
• Select a business metric to set growth context

14. Historic Cloud Cost Email

15. KPI dashboard

16. Detailed Cost Dashboards - Demo
VP Level
Director / 1st Level
Engineering Manager
Application Level

17. Detailed Cost Dashboards - Demo
VP Level
Director / 1st Level
Engineering Manager
Application Level

18. Detailed Cost Dashboards - Demo
VP Level
Director / 1st Level
Engineering Manager
Application Level

19. Netflix Today
• Decentralizing cloud cost responsibilities
• Active ROI analysis with largest service teams
• Exposing efficiency metrics beyond instance usage

20. Real Time Actionable Data

21. “ROI” Based Mindset
• Cause: today’s scale requires thoughtful deployment
strategy & service architecture
• Cloud capacity team engaged on a per-project basis

22. Pro-tip: Internal Unused RI Borrowing
Prod Heavy
RI Usage
Encoding
Heavy RI
Borrowed

23. Efficiency Score Cards

24. The Future of Cloud Cost Management
• Bin-packing at the service level through containers
• Dynamic traffic shifting between regions
• Automated ROI calculations at testing & deployment

25. Session Recap
• EC2: Everyone Contributes Cost
• S3: Smart Strategy Saves
• Data Transfer: Develop Transparency
• RDS: Remove Datacenter Sentiment
• Managing efficiency vs. innovation & availability
0
0.2
0.4
0.6
0.8
1
1.2 Aug-10
Oct-10
Dec-10
Feb-11
Apr-11
Jun-11
Aug-11
Oct-11
Dec-11
Feb-12
Apr-12
Jun-12
Aug-12
Oct-12
Dec-12
Feb-13
Apr-13
Jun-13
Aug-13
Oct-13
Dec-13
Feb-14
Apr-14
Jun-14
Aug-14
Oct-14
Dec-14
Feb-15
Apr-15
Jun-15

26. Thank you!
Contact: andrewp@netflix.com
Office Hours: 3:45-5:30pm @ Netflix Booth 326

27. Remember to complete
your evaluations!

28. Related Sessions
• SPOT302 – Availability: The New Kind of Innovator’s Dilemma
• DVO203 – A Day in the Life of a Netflix Engineer using 37% of the
Internet
• ISM301 – Engineering Netflix Global Operations in the Cloud