These are the slides from talk given by Mateusz Gajewski at AWS UG meeting in Warsaw. Content: Apache Mesos, Cluster Scheduling, Hybrid Environment, Scalability, Fault Tolerance.

1. @wendigo
Mateusz „Serafin” Gajewski • AWS UG Meetup
Scaling infrastructure beyond containers

2. @wendigo
Agenda
• Evolution of infrastructure at Allegro,
• Why Apache Mesos™?
• Apache Mesos key concepts,
• Future of datacenter and cloud computing?

3. @wendigo
History of scaling
infrastructure @ Allegro

4. @wendigo
Infrastructure 1.0

5. @wendigo
Job allocation problem

6. @wendigo
Web Scale Resource management
100s dots 100s dots

7. @wendigo
Infrastructure 2.0
1000s dots another 1000s dots

8. @wendigo
Infrastructure 2.1
1000s dots another 1000s dots

9. @wendigo
Challenges
• cloud not used as cloud ;),
• high cost of virtualization,
• effective resource utilization,
• microservice architecture,
• spread of new technologies,
• heterogenous resources,
• scalability, fault tolerance & HA,
• performance isolation,
• data processing at scale

10. @wendigo
Beyond cloud computing

11. @wendigo
Holly Grail of TCO

12. @wendigo
Infrastructure 3.0
A Platform for Fine-Grained
Resource Sharing in the Data Center

13. @wendigo
Scheduling

14. @wendigo
Cluster scheduling

15. @wendigo
Mesos architecture

16. @wendigo
Mesos frameworks

17. @wendigo
Offers

18. @wendigo
Execution isolation

19. @wendigo
External Containerizers

20. @wendigo
Mesos HA
• master election/failover with ZooKeeper,
• master maintains soft-state,
• framework state reconciliation,
• slave checkpointing,
• slave recovery,
• framework checkpointing

21. @wendigo
Beyond offers
• offer filters (constraints),
• static (pre-startup) reservations,
• dynamic (post-startup) reservations,
• oversubscription,
• persistent volumes,
• pluggable allocator scheduling policy (fair, priority based)

22. @wendigo
Mesos frameworks

24. @wendigo
Mesos recap
• „programming against the datacenter",
• distributed datacenter kernel,
• two-level multi-resource scheduler,
• scalable, highly-available & fault-tolerant,
• performance isolation with containers,
• exposes homogeneous resources,
• elastic, dynamic partitioning,
• high resource utilization

25. @wendigo
Future Datacenter

26. @wendigo
IaaC

27. @wendigo
Efficient utilization

28. @wendigo
Google’s Omega
source: Omega: flexible, scalable schedulers for large compute clusters

30. Questions?

31. http://meetup.com/allegro.tech
http://allegro.tech
@AllegroTechBlog
Work with us