David Greenberg QCon 16

1. Scaling Container Architectures
with OSS & Mesos
David Greenberg
QCon NY
6/13/2016

2. Who am I?
Engineer, Architect, and Operator

4. Formerly with Two Sigma

5. Open Source Fan

6. Today
 Why does a hedge fund need a cluster?
 What did we have before?
 What did we replace it with?
 What challenges did we solve?

7. Today
 Why does a hedge fund need a cluster?
 What did we have before?
 What did we replace it with?
 What challenges did we solve?

8. What’s a quant fund do?
Math

9. Trading system
Data
Feed
Machine
Learning
Trade

10. How do we choose data
feeds?

11. Experimentation

12. How do we design models?

13. We need a platform

14. Today
 Why does a hedge fund need a cluster?
 What did we have before?
 What did we replace it with?
 What challenges did we solve?

15. A long, long time ago
In a galaxy far, far away

16. Send work to job system
Abstracts scheduling & resources

17. Scaled beautifully

18. Limitations
It’s not a feature, it’s a bug

19. Limitation 1: Quotas

20. Limitation 1: Quotas

21. Limitation 2: API

22. Limitation 3:
Isolation, in theory

23. Limitation 3:
Isolation, in reality

24. Today
 Why does a hedge fund need a cluster?
 What did we have before?
 What did we replace it with?
 What challenges did we solve?

25. Requirements
• Scalability to 1000+
machines
• Throughput guarantee
• Strong isolation
• High utilization

26. What even scales?

27. Alternatives

28. OpenStack

30. Alternative Schedulers
Service goals differ between
organizations

31. Torque, Moab, Slurm

32. Introducing: Cook
Our DIY, not NIH scheduler

33. How to replace a critical
system?
Slay a few dragons

35. SLA v1
What SLA?

36. SLA v2
All that work for no improvement

37. SLA v3
The easiest thing

38. SLA v4
Enhanced Mesos’s DRF
with CRS

40. Today
 Why does a hedge fund need a cluster?
 What did we have before?
 What did we replace it with?
 What challenges did we solve?

41. Scaling is painful

42. How do you debug?
SSH and grep logs

43. =

49. How do you automate manual
processes?
Sometimes 100% isn’t good
enough

50. Retrospective
That went well

51. Scaled to thousands of nodes

52. Supports Spark & dozens of
internal tools

53. Automatically recovers from
common issues

54. Takeaways

55. Does your workload fit the tool?

56. Architecural Refactoring

59. Questions?
Want to ask a question,
but don’t know what to ask?
• Understanding your workload
• Dragonslaying/architectural refactoring
• Mesos
• Operations at scale