Build Highly scalable systems demand special care in which regards Engineering and Operation. Maintain/support a cluster with hundred/thousand of servers is a quite complex task. This is a presentation about Apache Mesos and how It helps to manage clusters in an efficient way.

1. Managing Computational resources with Apache Mesos
Jackson Oliveira
@cyber_jso

2. Software Architect, more than 13 years working with I.T
Currently Working at ilegra as consultant.
SOA specialist.
Believe on agile, devops principles, people transformation and
open source community ideas behind.
Like videogames and watch series.
Football fun.
Blog: http://jackson-s-oliveira.blogspot.com.br/
Linkedin: https://www.linkedin.com/in/jacksonsoliveira
facebook:https://www.facebook.com/jackson.dossantos.5

3. Storymap traditional data centers resource usage ….
90s… A different scenario

4. Storymap traditional data centers resource usage ….
90s… Changes for datacenter adminstration

5. Storymap traditional data centers resource usage ….
Virtualization, a bit game changing

6. Siloed cluster (static partition,low granularity ) no shared resource ...
Brought a different issue… The siloed cluster

7. Siloed cluster (static partition,low granularity ) no shared resource ...
Siloed cluster (static partitioning, low granularity) no shared resource
Siloed data center Data center managed by mesos
mesos

8. Resilience ...
Resilience

9. Management complexity (technology , hardware facts ) ...
Management complexity (technology, hardware, facts) ..

10. Distributed systems added complexity

11. Releases demand more effort

12. Orchestration complexity

13. Heterogeneous Archs on the same datacenter

14. Failures

15. Service Discovery

16. Big Data - distribute is needed

17. Mesos - The datacenter operational system

18. Resources as abstractions

19. Mesos Architecture

20. Mesos Architecture

21. Mesos Architecture

22. Mesos Architecture

23. Mesos Architecture

24. Slave node anatomy

25. Master node responsibilities

26. Master nodes High Availability

27. Frameworks responsibility

28. Frameworks Ecosystem
Long Running jobs
Big Data Processing
Batch Scheduling
Data Storage

29. Frameworks can coexist on the same datacenter

30. Resource Offering process

31. Resource Offering process

32. Resource Offering process

33. Resource Offering process

34. Resource Isolation

35. Native isolation using Linux containers

36. Isolation Mechanisms

37. CPU Share, Disk quotas and Bandwith limits
CPU Disk Network
- Core Isolation - Enforce maximum space usage
limits
- Limit I/O bandwidth usage
- Ports

38. Resource allocation

39. How to prevent frameworks starving?
Before workload After workload

40. How to prevent frameworks starving?

41. Static reservation - Good for Stateful services

42. Dynamic reservation - Good for Stateful Scheduled tasks

43. Resource preemption

44. Resource preemption

45. Frameworks: Marathon!

46. Built to support long running jobs

47. Specificing contranstraints
Good to ensure tasks are:
● Running all slaves
● Running At least one per
datacenter
● Running on specific slaves
● Running at least one per
cluster group

48. Scaling applications out
New tasks can
be allocated
dinamically

49. Handling failures

50. Handling failures
Timeout!

51. Handling failures
Report the
frameworks!

52. Handling failures
Reschedule the tasks!

53. ● Automation is heavily needed on this environment
● Troubleshooting can be tricky
● Monolithic systems may not take full advantage from this solution
● Ops: supporting the platform rather than specific products
● Applications that demands specific SO and hardware improvements may
wont leverage the benefits from this approach
Considerations

54. Managing Computational resources with Apache Mesos
Jackson Oliveira
@cyber_jso
Cesar Mesquita
@cmesquita00
Thank you!