Netflix accounts for more than a third of all traffic heading into American homes at peak hours. Making sure users are getting the best possible experience at all times is no simple feat and performance is at the core of this experience. In order to ensure performance and maintain development agility in a highly decentralized environment/(organization?), Netflix employs a multitude of strategies, such as production canary analysis, fully automated performance tests, simple zero-downtime deployments and rollbacks, auto-scaling clusters and a fault-tolerant stateless service architecture. We will present a set of use cases that demonstrate how and why different groups employ different strategies to achieve a common goal, great performance and stability, and detail how these strategies are incorporated into development, test and DevOps with minimal overhead.

1. Ensuring Performance in a Fast-
Paced Environment
Martin Spier
Performance Engineering @ Netflix
@spiermar
mspier@netflix.com
Performance & Capacity 2014 by CMG

2. Martin Spier
● Performance Engineer @ Netflix
● Previously @ Expedia and Dell
● Performance
o Architecture, Tuning and Profiling
o Testing and Frameworks
o Tool Development
● Blog @ http://overloaded.io
● Twitter @spiermar

3. ● World's leading Internet television network
● ⅓ of all traffic heading into American homes at
peak hours
● > 50 million members
● > 40 countries
● > 1 billion hours of TV shows and movies per
month
● > 100s different client devices

4. Agenda
● How Netflix Works
o Culture, Development Model, High-level
Architecture, Platform
● Ensuring Performance
o Auto-Scaling, Squeeze Tests, Simian Army, Hystrix,
Redundancy, Canary Analysis, Performance Test
Framework, Large Scale Tests

5. Freedom and Responsibility
● Culture deck* is TRUE
o 9M+ views
● Minimal process
● Context over control
● Root access to everything
● No approvals required
● Only Senior Engineers
* http://www.slideshare.net/reed2001/culture-1798664

6. Independent Development Teams
● Highly aligned, loosely coupled
● Free to define release cycles
● Free to choose use any methodology
● But it’s an agile environment
● And there is a “paved road”

7. Development Agility
● Continuous innovation cycle
● Shorter development cycles
● Automate everything!
● Self-service deployments
● A/B Tests
● Failure cost close to zero
● Lower time to market
● Innovation > Risk

9. Architecture
● Scalable and Resilient
● Micro-services
● Stateless
● Assume Failure
● Backwards Compatible
● Service Discovery

10. Zuul & Dynamic Routing
● Zuul, the front door for all requests from devices and
websites to the backend of the Netflix streaming
application
● Dynamic Routing
● Monitoring
● Resiliency and Security
● Region and AZ Failure
* https://github.com/Netflix/zuul

11. Cloud
● Amazon’s AWS
● Multi-region Active/Active
● Ephemeral Instances
● Auto-Scaling
● Netflix OSS (https://github.com/Netflix)

12. Performance Engineering
● Not a part of any development team
● Not a shared service
● Through consultation improve and maintain the
performance and reliability
● Provide self-service performance analysis utilities
● Disseminate performance best practices
● And we’re hiring!

13. What about Performance?

14. Auto-Scaling
● 5-6x Intraday
● Auto-Scaling Groups (ASGs)
● Reactive Auto-Scaling
● Predictive Auto-Scaling (Scryer)

15. Squeeze Tests
● Stress Test, with Production Load
● Steering Production Traffic
● Understand the Upper Limits of Capacity
● Adjust Auto-Scaling Policies
● Automated Squeeze Tests

16. Red/Black Pushes
● New builds are rolled out as new
Auto-Scaling Groups (ASGs)
● Elastic Load Balancers (ELBs)
control the traffic going to each
ASG
● Fast and simple rollback if issues
are found
● Canary Clusters are used to test
builds before a full rollout

17. Monitoring: Atlas
● Humongous, 1.2 billion distinct time
series
● Integrated to all systems, production
and test
● 1 minute resolution, quick roll ups
● 12-month persistence
● API and querying UI
● System and Application Level
● Servo (github.com/Netflix/servo)
● Custom dashboards

18. Vector
● 1 second Resolution
● No Persistence
● Leverages Performance Co-
Pilot (PCP)
● System-level Metrics
● Java Metrics (parfait)
● ElasticSearch, Cassandra
● Flame Graphs (Brendan Gregg)

19. Mogul
● ASG and Instance Level
● Resource Demand;
● Performance
Characteristics;
● And Downstream
Dependencies.

20. Slalom
● Cluster Level
● High-level Demand Flow
● Cross-application Request
Tracing
● Downstream and Upstream
Demand

21. Canary Release
“Canary release is a technique to reduce the risk
of introducing a new software version in
production by slowly rolling out the change to a
small subset of users before rolling it out to the
entire infrastructure and making it available to
everybody.”

22. Automatic Canary Analysis (ACA)
Exactly what the name implies. An automated
way of analyzing a canary release.

23. ACA: Use Case
● You are a service owner and have finished
implementing a new feature into your application.
● You want to determine if the new build, v1.1, is
performing analogous to the existing build.
● The new build is deployed automatically to a canary
cluster
● A small percentage of production traffic is steered to the
canary cluster
● After a short period of time, canary analysis
is triggered

24. Automated Canary Analysis
● For a given set of metrics, ACA will compare
samples from baseline and canary;
● Determine if they are analogous;
● Identify any metrics that deviate from the
baseline;
● And generate a score that indicates the overall
similarity of the canary.

25. Automated Canary Analysis
● The score will be associated
with a Go/No-Go decision;
● And the new build will be
rolled out (or not) to the rest
of the production
environment.
● No workload definitions
● No synthetic load

26. What about pre-production
Performance
Testing?
When is it appropriate?

27. Not always!
Sometimes it doesn't make sense to run
performance tests.

28. Remember the short release cycles?
With the short time span between production builds,
pre-production tests don’t warn us much sooner.
(And there’s ACA)

29. So when?
When it brings value. Not just because is
part of a process.

30. When? Use Cases
● New Services
● Large Code Refactoring
● Architecture Changes
● Workload Changes
● Proof of Concept
● Initial Cluster Sizing
● Instance Type Migration

31. Use Cases, cont.
● Troubleshooting
● Tuning
● Teams that release less frequently
o Intermediary Builds
● Base Components (Paved Road)
o Amazon Cloud Images (AMIs)
o Platform
o Common Libraries

32. Who?
● Push “tests” to development teams
● Development understands the product, they
developed It
● Performance Engineering knows the tools
and techniques (so we help!)
● Easier to scale the effort!

33. How? Environment
● Free to create any environment configuration
● Integration stack
● Full production-like or scaled-down environment
● Hybrid model
o Performance + integration stack
● Production testing

34. How? Test Framework
● Built around JMeter

35. How? Test Framework
● Runs on Amazon’s EC2
● Leverages Jenkins for orchestration

36. How? Analysis
● In-house developed web analysis tool and API
● Results persisted on Amazon’s S3 and RDS

37. How? Analysis
● Automated analysis built-in (thresholds)
● Customized alerts
● Interface with monitoring tools

39. Large Scale Tests
● > 100k req/s
● > 100 of load generators
● High Throughput Components
o In-Memory Caches
● Component scaling
● Full production tests

40. Large Scale Tests: Problems
● Your test client is likely the first bottleneck
● Components are (often) not designed to
scale
o Great performance per node;
o But they don’t scale horizontally.
o Controller, data feeder, load generator*, result
collection, result analysis, monitoring
* often the exception

41. Large Scale Tests: Single Controller
● Single controller, multiple load generators
● Controller also serves as data feeder
● Controller collects all results synchronously
● Controller aggregates monitoring data
● Batch and async might alleviate the problem
● Analysis of large result sets is heavy (think
percentiles)

42. Large Scale Tests: Distributed Model
● Data Feeding and Load Generation
o No Controller
o Independent Load Generators
● Data Collection and Monitoring
o Decentralized Monitoring Platform
● Data Analysis
o Aggregation at node level
o Hive/Pig
o ElasticSearch

43. Takeaways
● Canary analysis
● Testing only when it brings VALUE
● Leveraging cloud for tests
● Automated test analysis
● Pushing execution to development teams
● Open source tools

44. Martin Spier
mspier@netflix.com
@spiermar
http://overloaded.io/

45. References
● parfait (https://code.google.com/p/parfait/)
● servo (https://github.com/Netflix/servo)
● hystrix (https://github.com/Netflix/Hystrix)
● culture deck (
http://www.slideshare.net/reed2001/culture-1798664)
● zuul (https://github.com/Netflix/zuul)
● scryer (
http://techblog.netflix.com/2013/11/scryer-netflixs-predictive-
auto-scaling.html)

46. Backup Slides

47. Simian Army
● Ensures cloud handles failures
through regular testing
● The Monkeys
o Chaos Monkey: Resiliency
o Latency: Artificial Delays
o Conformity: Best-practices
o Janitor: Unused Instances
o Doctor: Health checks
o Security: Security Violations
o Chaos Gorilla: AZ Failure
o Chaos Kong: Region Failure

48. “... is a latency and fault
tolerance library designed to
isolate points of access to
remote systems ...”
● Stop cascading failures.
● Fallbacks and graceful degradation
● Fail fast and rapid recovery
● Thread and semaphore isolation with
circuit breakers
● Real-time monitoring and
configuration changes
* https://github.com/Netflix/Hystrix

49. Real-time Analytics Platform (RTA)
● ACA runs on top of RTA
● Compute Engines
o OpenCPU (R)
o OpenPY (Python)
● Data Sources
o Real-time Monitoring Systems
o Big Data Platforms
● Reporting, Scheduling, Persistence

50. Slow Performance Regression
● Deviation => “acceptable” regression
● Small performance regressions might sneak in
● Short release cycle = many releases
● Many releases = cumullative regression

51. Slow Performance Regression

52. Testing Lower Level Components
● Base AMIs
o OS (Linux), tools and agents
● Common Application Platform
● Common Libraries
● Reference Application
o Leverages a common architecture (front, middle,
data, memcache, jar clients, Hystrix)
o Implements functions that stress
specific resources (cpu, service, db)