Slides from Diptanu Choudhury's talk at DockerCon SF 2015 Talk Description: Netflix has a complex micro-services architecture that is operated in an active-active manner from multiple geographies on top of AWS. Amazon gives us the flexibility to tap into massive amounts of resources, but how we use and manage those is a constantly evolving and ever-growing task. We have developed Titan to make cluster management, application deployments using Docker and process supervision much more robust and efficient in terms of CPU/memory utilization across all of our servers in different geographies. Titan, a combination of Docker and Apache Mesos, is an application infrastructure gives us a highly resilient and dynamic PAAS, that is native to public clouds and runs across multiple geographies. It makes it easy for us to manage applications in our complex infrastructure and gives us the ability to make changes in the IAAS layer without impacting developer productivity or sacrificing insight into our production infrastructure.

1. Reliably shipping containers in a
resource rich world using Titan
Diptanu Choudhury
Software Engineer, Netflix
@diptanu

2. A Cloud Native Application built on
Micro Services Architecture

4. Active-Active
Multi- Geography

5. Titan is a Compute service
distributed across multiple
geographies

6. A need for a common
resource scheduler for
domain specific
distributed systems

7. The operational
benefits of a PaaS
without the dilemmas
of sandboxing
technologies.

8. Consistent tooling and
operational control
plane for SREs across
all technology stacks

9. Faster turn around
time from
development to
production

10. Auto Scaling Groups
are harder to adopt
for event based
orchestration systems

11. Increasing density of
application processes
per server

13. Why we chose Docker
• Process isolation
• Immutable deployment artifacts
• Ability to package dependencies of an application in
a single binary
• Tooling around the runtime for building and
deploying
• Scalable distribution of binaries across clusters

14. Docker Containers are
the deployment
artifacts and process
runtime for Titan

15. The Titan API

16. AutoScaling
• Two Levels of Autoscaling
- Scaling of underlying compute
resources
- Application Scaling based on
business and performance
metrics

17. AutoScaling
• Two Types of Autoscaling
- Predictive
• Titan scales up infrastructure based on
historical data on statistical modeling.
- Reactive
• Scaling activities are triggered based on pre-
defined thresholds

18. Disk
• Titan manages volumes for containers.
• We use ZFS on Linux to create volumes
• Data volumes are mounted within containers

19. Logging
• Titan allows users to stream logs of a Task from a
running container in a location transparent manner
• Logs are archived off-instance and Titan provides
API to stream logs of finished tasks

20. Network
• In EC2 Classic, Titan exposes ports
on containers on the host machine.
- Mesos is used as a broker for port
allocation

21. Network
• In VPC, every container gets its own IP
address.
- Mesos is completely out of the
picture
- We use ENIs and move them into the
network namespace of containers
- Developing a custom network plugin

22. Monitoring
• cgroup metrics published by the kernel are
pushed to Atlas.
• Users can see all the cgroup metrics per task.
• cgroup notification API for alerting

23. Failover
• Titan allows SREs to drain a cluster
of containers into newer compute
nodes
• Underlying VMs are automatically
terminated when containers crashes
for hardware/OS problems
• Allows failover across multiple data
centers

24. From a 1000 Feet

25. From a 5000 Feet

26. Where are we with Titan at Netflix
Prototype for
running cron
jobs
Non Mission
Critical
Algorithms
Mission Critical
Batch Jobs in
Production
Prototypes for
running online
processes and
web services
Parts of Netflix
Data Pipeline
The Netflix API
and Edge
Systems
May ‘14
Future
Near Term

27. Thank you
Diptanu Choudhury
@diptanu
www.linkedin.com/in/diptanu