Mesos and Container Schedulers Landscape and Practical Experiences This document discusses Piksel's evolution from static VM partitioning to using Mesos as a container scheduler. It summarizes their experiences using different frameworks like YARN and Mesos directly. Apache Aurora is highlighted as a framework that provides capabilities for multi-tenancy, quotas, and preemption needed for Piksel's heterogeneous workloads and multiple customers. Customizing Aurora's job definition language is discussed.

1. Mesos and Container
Schedulers
Landscape and Practical Experiences

2. Piksel: We do video stuff

3. Piksel’s big data toolset

4. Multi-tenant PaaS for Heterogeneous Workloads - E.g. Metadata Enrichment

5. Piksel Video
Platform - Cluster
Evolution

6. Previous - Static Partitioning at VM Level

7. Impossible to
efficiently utilise
resources across
machines

8. Too much EC2
We estimated that this was about 20-30% utilized

9. Large
maintenance
overhead and
cognitive load

10. Puppet - VM Focused Deployment
With provisioning
tools VMs are
provisioned with a
specific role in
mind.

11. Teams are
thinking about
machines rather
than applications

12. Cluster Manager: VMs are provisioned to contribute resources
With cluster
manager the VMs
are provisioned to
contribute
resources

14. Java big data frameworks

15. We dipped our
toe in the cluster
pool

16. We ran with YARN and with Mesos

17. API services: Docker, marathon, mesos

18. Heterogeneous API services, homogeneous workload

19. Flink, Spark, YARN

20. Multiple clusters - Myriad

21. Where we wanted to get to
● Single platform for deploying applications
● Development teams don’t care about machines
● Development teams do care about applications
● Development teams describe their application and
operational characteristics declaratively

22. Heterogeneous Workload, single PaaS

23. Let’s Talk about
Workload Types

24. Video Platform at a Glance

25. Everything On Mesos

26. But Mesos Needs
a Framework to
Schedule work

27. The Role of a Framework
Mesos provides
resource offers to
frameworks which
can choose to
accept or decline
them subject to
various constraints

28. Mesos is Low-Level - Requires a Framework
Credit: https://twitter.com/wendigo/status/646394656700375040

29. Framework capabilities - the BIG THREE

30. Framework capabilities
● Choice between Aurora and the
others boils down to the need for
batch jobs and pre-emption
● Do you run DAG jobs?
● Do you have a large engineering
team? Or multiple isolated
customers that need quotas?

31. Apache Aurora
- Self-Service
multi-tenancy

32. Noisy neighbours
are a problem!

33. Customers….
● Some customers pay more than
others, and need more capacity
● Some jobs are higher priority than
others
● We don’t need all capacity, all the
time

34. Aurora - Roles
Aurora has a concept of a Role, e.g.
a Customer or Internal Team… Work
is submitted into a Role. Quotas can
be assigned to Roles.

35. Aurora - Goal
● Ensure high priority work gets done
● Provide quota isolation across Roles
● Allow low priority work to use all
resources unless high priority work arrives

36. Aurora - Pre-emption
Aurora supports pre-emption; killing
and rescheduling work when higher
priority work arrives. It does this
under 2 conditions...

37. Aurora - Priority killing
1. Role submits a job with higher
priority that existing job in same
Role

38. Aurora - Intra-Role Prioritization
2. A production job requires quota
from a non-production job of any
Role

39. Multi-Tenancy Scheduling - Production Quotas
● Aurora allows
us to give a
quota to Roles
for their critical
work

40. Multi-Tenancy Scheduling - Environments and Pre-emption

41. Multi-Tenancy Scheduling - Slot Non-Critical Work Around Critical Work
Aurora allows us to slot low-priority
(typically batch) work around critical
work, yet discard it if more critical
work arrives

42. Frameworks that
provision their
own resources
need Aurora
integration

43. Problem: Workloads that provision their own resources

44. Problem: Workloads that provision their own resources

45. PVP - Current state of affairs

46. Questions??
p.s. We’re hiring!

47. How do we submit work to Aurora

48. Job DAGs
Aurora allows Jobs to contain parallel
and sequential stages, resulting a
DAG. However all processes will be
ran in the same container. It is not a
high level job orchestrator.

49. Domain Specific Job Definition Language With Templating
transcode_proc = Process(
name=’transcode’,
cmdline=’ffmpeg -i {{input}} {{output}} ’
)
transcode_task = Task(
name = ‘transcode_task ’,
processes = [transcode_proc]
resources = Resources(cpu = 1, ram = 128*MB, disk=1*GB)
)
jobs = [
Job(cluster = 'devcluster',
environment = 'sandbox',
role = 'customer_a',
name = 'transcode_job',
task = transcode_task,
container = Docker(image = 'ffmpeg_image:16.04 ')
)
]

50. Customer Extensions - Extending the DSL
transcode_task = transcode()
.input(“s3://input.mp4”)
.output(“s3://output.avi”)
jobs = Flow.create(transcode_task)
No nice hook exists to plug in DSL extensions.
We need to patch and build the Aurora client and then register this within the
configuration loader.