Come hear about our container management platform, Titus. Titus launches over 2 millions containers per week for service and batch workloads. Come to learn what applications are powered by Titus and what values the developers are getting from containers. Also, we will cover some of the Titus unique aspects of reliability, control plane, scheduling, and container runtime technologies. We will also cover our integrations with Netflix systems such as Spinnaker as well as Amazon concepts such as VPC and IAM. https://www.meetup.com/Netflix-Open-Source-Platform/events/247776324/

1. Netflix Open Source
Season 6, Episode 1
Netflix Open Source - @NetflixOSS

2. Introduction

3. Titus
● Netflix’s Container
Management Platform
Scheduling
● Service & batch lifecycle
● Resource management
Container Execution
● AWS Integration
● Netflix Infra Support
Service
Job and Fleet Management
Resource Management & Optimization
Container Execution
Integration
Batch

4. The Titus Team
● Design
● Develop
● Operate
● Support

5. Those who contributed to Titus
Andrei SharmaManasJigishDiptanu
Partners
● Common infrastructure teams
● Internal users and platforms on top of Titus
Previous Titus codebase contributors

6. Why containers?
● Simpler management of compute resources
● Simpler deployment packaging artifacts for compute jobs
● Need for a consistent local developer environment

7. ● 1000+ Applications
● Netflix API, Node.js Backend UI Scripts
● Machine Learning (GPUs) for personalization
● Encoding and Content use cases
● Netflix Studio use cases
● CDN tracking and planning
● Massively parallel CI system
● Data Pipeline routing and SPaaS
● Big Data platform use cases
Growing set of container use cases
Batch
Q4 15
Basic
Services
1Q 16
Production
Services
4Q 16
Customer
Facing
Services
2Q 17
shadow

8. Q1 2018 Container Usage
Common
Jobs Launched 176K jobs / day
Different applications 1K+ different images
Regional isolated Titus stacks 7
Services
Single App Cluster Size 5K (real), 12K containers (benchmark)
Agents managed 16K VMs
Batch
Containers launched 430K / day
Agents autoscaled 350K VMs / month

9. Titus Product Management
● Ordered priority focus on
○ Developer Velocity
○ Reliability
○ Cost Efficiency
● Easy migration from VM’s to containers
● Easy container integration with VM’s and Amazon Services
● Focus on just what Netflix needs

10. High Level Titus Architecture
Cassandra
Titus Control Plane
● API
● Scheduling
● Job Lifecycle Control
EC2 Autoscaling
Fenzo
container
container
container
docker
Titus Agents
Mesos agent
Docker
Docker Registry
containercontainerUser Containers
AWS Virtual Machines
Mesos
Titus System ServicesBatch/Workflow
Systems
Service
CI/CD

11. Control Plane & Scheduling

12. Running Containers
API
Request
Job Task(s) Container(s)

13. Client Integrations
Titus Gateway
REST
GRPC

14. Control Plane Overview
Cassandra
MesosTitus Gateway
Zookeeper
Titus Master
Job
Management
Scheduler
Agent
Management
Capacity
Management
AWS

15. Job Management
Manages the lifecycle of jobs
Batch
● Run to completion
● Retry policies
Service
● Run to termination
● Similar to AWS ASG
● min/max/desired

16. The Scheduling Problem
Titus Agents
Containers

17. Scheduling Tradeoffs
Reliability
Launch
Time
Cost
Trade
offs

18. How is a container scheduled?
Tasks
Agents
Filtering Scoring
Agent
Assignment
Fenzo

19. How is a container scheduled? - Filtering
● Does it fit?
○ CPU
○ Memory
○ Disk
○ Network Bandwidth
○ Network Interface
● Is the instance group active?
● Is the agent healthy?
● Is the agent launching a task?

20. How is a container scheduled? - Scoring
● Bin packing vs spreading
● Images
● Security groups
Different workloads have different needs

21. Scheduling Around Issues
Agents manage the lifecycle of a container
● Setup networking
● Image pull
● Run container
Realization: Schedule around common issues

22. Agent Management
Agent Management
Scheduler
Health checks
Cluster lifecycle
Other signals
Unified component for providing a single consistent view of
agent information
Autoscaler
Capacity
Management

23. Agent Management
Agent B
Healthy
Scheduler does not schedule to
unhealthy agents
Agent A
Unhealthy
Agent Management
Scheduler
Agent Health

24. Capacity Management
User configures “capacity groups” based on workload type
Critical (RIs)
● Pre-warmed instances in order to achieve fast launch times
● Buffer to support temporary extra capacity needs for deployments
Flex (On-Demand)
● Autoscaled instances based on demand
Opportunistic (Spot) - Coming Soon
● Utilize unused instances with the ability to preempt or evict the workload

25. Container Execution

26. Tenants of the The Titus Container Runtime
● Workload Agnostic
○ We’ll work equally well with code from anywhere
■ Laptops
■ Dockerhub
○ Enable easy lift-and-shift
○ Enable the broadest set of applications
● Security
○ The container runtime should not expose excessive security risk
○ Security should have cognitive overhead
● Infrastructure Force Multiplier
○ The container runtime should allow loose coupling with the underlying
infrastructure and app

27. How Does Titus Uniquely Solve These
Problems?

28. Orchestration Process
Composition of off the shelf
components

29. Orchestration Process
Composition of off the shelf
components
● Custom Coordinator

30. Orchestration Process
Composition of off the shelf
components
● Custom Coordinator

31. Orchestration Process
Composition of off the shelf
components
● Custom Coordinator
● Injected:
○ Metrics
○ IAM service
● Custom mount tool (for NFS)
● Specific-purpose networking
● Auxiliary components
○ Log Viewer
○ Log Uploader
● Self-contained

32. Let It Crash™: Process Model
Tightly tied to runtime orchestration model

33. Let It Crash™: Process Model
Tightly tied to runtime orchestration model

34. Let It Crash™: Process Model
Tightly tied to runtime orchestration model

35. Let It Crash™: Process Model
Tightly tied to runtime orchestration model

36. Let It Crash™: Process Model
Tightly tied to runtime orchestration model

37. Approach:
Compose the best available techniques

38. Logs: Everybody Has Them
Enable All Approaches

39. Services
Daemons

40. Services: Daemons a la Nginx
S3

41. Services: Daemons a la Nginx
S3

42. Services: Log Uploading
S3

43. Log to Stdio
Services that write to stdio and simplistic batch

44. Docker Logs!

45. Docker Logs.

46. 100% CPU
Why is logging using?

47. Stdio Management: Own Stdout / Stderr

48. Stdio Management: FDs are files with O_APPEND

49. Stdio Management: FDs are files with O_APPEND

50. Approach:
Unopinionated Logging

51. Networking

52. Goals of Networking
● VPC networking security
● Prevent interference of mice and elephants
● VPC Routable IP per container
○ Enable lift-and-shift applications
● Rapid network reconfiguration
○ Security groups
○ Releasing / leasing IP addresses

53. The Building Block
● Elastic Network Interfaces
○ One for control plane
○ Limit per machine is 1-15
○ Not performance isolating
● Union of Security Groups
● Multiple IPs available per ENI
○ Bounded to <30

54. Simple Networking
Why doesn’t this work?
● No public IPs
● Overhead of NAT
● All containers share the
same Security Group
● No QoS

55. Basic Networking
Why doesn’t this work?
● No QoS
● One container per ENI
● Incredibly inefficient
○ Allows packing 6 containers
○ Versus 50-60 we can do
now

56. Intermediate Networking
Why doesn’t this work?
● No QoS

57. Advanced Networking
This Works!
Leverages:
● BPF
● HTB
○ ECN
○ BBR
○ FQ_CODEL

58. Approach:
Make it Just Work

59. Security

60. Container User Security

61. Container User Insecurity

62. User Namespaces

63. User Namespaces Security Isolation

64. titus-ssh
● Provides transparent interactive access to container
● Without compromising host access

65. Vector: Self-service performance troubleshooting

66. Approach:
Infra Security Should be Invisible

67. Integrations

68. Containers Should Be Transparent
● Applications should not need to be rewritten
● Developers should not need to relearn operations
Remove barriers to adoption!
Existing infrastructure and tools should work with
containers

69. Building and Deploying

70. ● Eureka supports VM and Titus container service discovery
Service Discovery
App A
App AVM
app
App A
App Acontainer
app
Eureka Service
Discovery
Registration
Connectivity
Discovery is decoupled from virtualization

71. Load Balancing
● Services also use AWS Load Balancers

72. Load Balancing
● Netflix drove IP Target Groups with AWS ALB team
● Enables routing to Titus containers
Titus Control
Plane
Task Task
IP Target
Group
Associate
Target Group
Update Target
Group State
Route Traffic
Ingress Traffic

73. Auto Scaling
● Many services are built to be auto scaled
● Apps can be automatically scaled up/down based on metrics
Titus Control
Plane
Task Task
Create Auto
Scaling Policy
Create Auto
Scaling Policy
CloudWatch
Metrics
Adjust instance count
CloudWatch
Alarms
● Titus drove (upcoming) AWS Auto Scaling feature

74. Secure AWS Service Access
● Apps need AWS IAM credentials
○ EC2 MDS provides EC2 instance credentials
EC2 Virtual Machine Host
IAM=appInstanceProfile
Titus Container AWS EC2
Metadata
Service
mds interfaceeth0
169.254.169.254
aux netns
lo interface
169.254.169.254
AssumeRole to
appInstanceProfile
veth
mds proxy
server
bind
● Route requests through metadata proxy
○ Provide container level incredials via STS AssumeRole

75. Insight and Telemetry
● Atlas supports collecting and emitting container metrics
○ Container metrics & app metrics
App A
App AVM
app
App A
App Acontainer
app
● Developers should not need to relearn operations

76. But one more question....
Thank You!

79. Demo Stations