Netflix uses containers to run both batch jobs and services. For batch jobs, containers simplify resource management and allow jobs like model training and media encoding to easily share resources. Services are more complex to run in containers due to challenges like constant resizing, statefulness, and networking. Netflix addresses these challenges through solutions like a VPC networking driver and reusing existing infrastructure services for containers. Looking ahead, Netflix aims to run more containers at larger scale for areas like developer experience, continuous integration, and internal resource optimization.

1. Containers @ Netflix
How they add to a proven cloud architecture

3. https://www.flickr.com/photos/hinnosaar/2655128664

4. Datacenters
Java monolithic app
Oracle database

6. AWS cloud
Java microservices
Cassandra

7. Netflix Open
Source Software
http://netflix.github.io

9. Containers @ Netflix
How they add to a proven cloud architecture

10. batch applications

11. Multi-tenant (cgroups/Mesos)
historically used for batch
Linux cgroups

12. What do batch users want?
● Simple shared resources, run till done, job files
● NOT
○ EC2 Instance sizes, autoscaling, AMI OS’s
● WHY
○ Offloads resource management ops, Simpler

13. Titus
Batch
Job Management
Resource Management & Optimization
Container Execution
Integration
Workflow, Data Analysis, Adhoc
Upstream Systems

14. Netflix Batch Job Examples
● Algorithm Model Training (with GPU’s)

15. Netflix Batch Job Examples
● Media Encoding
● Digital Watermarking
1 1

16. Netflix Batch Job Examples
Open Connect
CDN Reporting
Adhoc
Reporting

17. Lessons Learned from Batch
● Docker helped generalize use cases

18. Lessons Learned from Batch
● Docker helped generalize use cases
● Advanced scheduling required

19. Lessons Learned from Batch
● Docker helped generalize use cases
● Advanced scheduling required
● Initially ignored failures (with retries)

20. ● Docker helped generalize use cases
● Advanced scheduling required
● Initially ignored failures (with retries)
● Time sensitive batch came later
Lessons Learned from Batch

21. Current Container Usage - Batch
● 100 containers / hour
● Peaks of 1000’s per hour
● Large spikes of CI testing and Digital Watermarking
A random day’s worth of containers

22. service applications

23. Why Services in containers?
Theory Reality

24. “Why is Apache and Tomcat running on
my NodeJS server”

25. “Why is Apache and Tomcat running on
my NodeJS server”
Problem:
BaseAMI optimized for Java, not
easily customizable

26. “Why do I need java, gradle, ospackage
after my non-Java build?”

27. “Why do I need java, gradle, ospackage
after my non-Java build?”
Problem:
Reuse of Java-centric AMI tooling

28. “I want an instance with a single core to
run my lightweight server”

29. “I want an instance with a single core to
run my lightweight server”
Problem:
Small instances are not reliable

30. Enter Docker
● Have a new language?
● Have a build tool you like?
● Want to resource isolate easily?
Come one, come all

31. Services are just long running batch?
Services
Job Management
Resource Management & Optimization
Container Execution
Integration
Service Apps
Batch

32. Services more complex
● Services resize constantly and run forever
○ Autoscaling
○ Hard to upgrade underlying hosts

33. Services more complex
● Services resize constantly and run forever
○ Autoscaling
○ Hard to upgrade underlying hosts
● Have more state
○ Ready for traffic vs. just started/stopped
○ Even harder to upgrade

34. Services more complex
● Services resize constantly and run forever
○ Autoscaling
○ Hard to upgrade underlying hosts
● Have more state
○ Ready for traffic vs. just started/stopped
○ Even harder to upgrade
● Existing well defined dev, deploy, runtime & ops tools

35. Real networking is hard

36. Multi-tenant
Need IP per container - in VPC
Using security groups
Using IAM roles
Considering network resource isolation

37. Solutions
● VPC Networking driver
○ Supports ENI’s - full IP functionality
○ Scheduled security groups
○ Support traffic control (isolation)
● EC2 Metadata proxy
○ Adds container “node” identity
○ Delivers IAM roles

38. Reuse existing infrastructure services
VMVM
EC2
AWSAutoScaler
VMs
App
Cloud Platform
(metrics, IPC, health)
VPC
Netflix Cloud Infrastructure (VM’s + Containers)
Atlas Eureka Edda

39. Enable them for containers
VMVM
EC2
AWSAutoScaler
VMs
App
Cloud Platform
(metrics, IPC, health)
VPC
Netflix Cloud Infrastructure (VM’s + Containers)
VMVM
Atlas
TitusJobControl
Containers
App
Cloud Platform
(metrics, IPC, health)
Eureka Edda
VMVM
Batch
Containers

40. Spinnaker

41. Deploy based
on new images
tags

42. Basic resource
requirements
IAM Roles & Sec
Groups per
container
Deploy
Strategies
Same as VM’s

43. Easily see
health &
discovery

46. Current Container Usage - Service
● Still small - 100’s of containers
● NodeJS Device UI Scripts Apps
● Stream Processing Jobs - Flink
● Various Internal Dashboards

47. developer experience

48. ● Consistent Mac setup
● Consistent workflows
● Netflix integration
The Docker experience

49. NEWT (Netflix Workflow Toolkit)

50. dev machine bootstrap

51. project scaffolding

52. setup dev pipelines

53. run locally

54. build history
start pipeline

55. beyond java

56. beyond java

58. Node.js support
before Newt:
● Install Java
● Install Nebula (Netflix Gradle)
● Add a build.gradle
● Run gradlew wrapper
● Add deb instr to build.gradle
● Install Vagrant + VBox
● Test deb locally
● Create Stash repo
● Create Jenkins job
● Create Spinnaker pipelines
● git push
after Newt:
● Install Newt
● newt init --app-type nodejs
● git push
Nebula ospackage is
hidden inside a local
Docker container
managed by Newt

59. https://en.wikipedia.org/wiki/Data_visualization#/media/File:Social_Network_Analysis_Visualization.png
dependencies @ scale

60. Project Niagara

62. 100,000 builds a day, peak

63. where are we going?

64. Future of containers @ Netflix
● More scale!
● Guaranteed capacity (service)
● Fair scheduling (batch)
● Local integration test env (devex)
● Next generation CI (devex)
● Internal RI spot market of trough

65. Questions?