"How can you reliably schedule tasks in an unreliable, autoscaling cloud environment? This presentation talks about the design of our Fenzo scheduler, built on Apache Mesos, that serves as the core of our stream-processing platform, Mantis, designed for real-time insights. We focus on the following aspects of the scheduler: - Resource granularity - Fault tolerance - Bin packing, task affinity, stream locality - Autoscaling of the cluster and of individual service jobs - Constraints (hard and soft) for individual tasks such as zone balancing, unique, and exclusive instances This talk also includes detailed information on a holistic approach to scheduling in a distributed, autoscaling environment to achieve both speed and advanced scheduling optimizations."

12. Analytics
System health
Customer experience
Insights
Anomaly detection

16. User
Job 1
User
Job 2
User
Job 3
Discovery

17. Mantis
Mantis
Apache Mesos
Apache Mesos
Mantis
Apache Mesos
ASG
ASG
ASG
Fenzo
Mesos
Framework

18. Job
Job
Job

22. Mesosslave
FrmWrk2 executor
Task
Task
Mesosslave
FrmWrk2 executor
FrmWrk1 executor
Task
Task
Mesosmaster
Standby master
Standby master
Mesosslave
FrmWrk1 executor
Task
Task
FrmWrk1
FrmWrk2

24. Instance 1
Instance 1
Task A
Instance 2
Task B
Instance 1
Task A
Instance 1
Task A
Task B
Task C
Task D

35. Data stream
Host A
Task1
Host B
Task2
Host C
Task3

36. Data stream
Data stream
Host X
Task1
Task2
Task3
Host A
Task1
Host B
Task2
Host C
Task3

41. Mantis
Mantis
Mantis
Fenzo
Mesos
Framework

42. Mesosslave
FrmWrk1 executor
FrmWrk1 executor
Mesosslave
Framework executor
Task
Task
Task
Framework executor
Task

43. Apache Mesos
Apache Mesos
MesosMaster
Apache Mesos
Framework
Persistence

44. Apache Mesos
Apache Mesos
MesosMaster
Apache Mesos
Framework
Persistence

45. .setName(name)
.setFailoverTimeout(to)
.setId(id)
.setCheckpoint(true)
.build();

50. Heterogeneous
Autoscale
Visibility
Plugins for
Constraints, Fitness
High speed

52. Mesosmaster
Mesosframework
Tasks
requests
Available
resource
offers
Fenzotask
scheduler
Persistence

53. Fitness
Urgency

54. Speed
Accuracy
First fit assignment
Optimal assignment
Real world trade-offs
~ O (1)
~ O (N * M)1
1Assuming tasks are not reassigned

69. ✔

71. 750
950
1150
1350
1550
1750
1950
2150
#Hosts
No bin packing used
#Full
#Partial
#Empty
750
950
1150
1350
1550
1750
1950
2150
#Hosts
With bin packing
#Full
#Partial
#Empty

72. 1450
1550
1650
1750
1850
1950
2050
#Hosts
No task runtime-based packer
Differentruntimes
Sameruntimes
Unused
1450
1550
1650
1750
1850
1950
2050
#Hosts
Using task runtime-based packer
Differentruntimes
Sameruntimes
Unused

78. ASG/Cluster: mantisagent
MinIdle: 8
MaxIdle: 20
CooldownSecs: 360
ASG/Cluster: mantisagent
MinIdle: 8
MaxIdle: 20
CooldownSecs: 360
ASG/cluster: mantisagent
MinIdle: 8
MaxIdle: 20
CooldownSecs: 360
Fenzo
ScaleUpaction:
Cluster, N
ScaleDownaction:
Cluster, HostList

81. .withLeaseOfferExpirySecs(60)
.withLeaseRejectAction( (lease) -> {
mesosDriver.declineOffer(lease.getOffer().getId());
})

82. .withLeaseOfferExpirySecs(60)
.withLeaseRejectAction( (lease) -> {
mesosDriver.declineOffer(lease.getOffer().getId());
} )
.withFitnessCalculator(
BinPackingFitnessCalculators.cpuBinPacker)

85. Heterogeneous
Autoscale
Visibility
Plugins for
Constraints, Fitness
High speed

86. Mantis
Mantis
Apache Mesos
Apache Mesos
Mantis
Apache Mesos
ASG
ASG
ASG
Fenzo
Mesos
Framework

88. Talk
Time
Title
PFC-305
Wednesday, 1:15pm
Embracing Failure: Fault Injection and ServiceReliability
BDT-403
Wednesday, 2:15pm
Next Generation Big Data Platform at Netflix
PFC-306
Wednesday, 3:30pm
Performance Tuning EC2
DEV-309
Wednesday, 3:30pm
From Asgardto Zuul, How Netflix’s proven Open Source Tools can accelerateand scale your services
ARC-317
Wednesday, 4:30pm
Maintaining a ResilientFront-Door at Massive Scale
PFC-304
Wednesday, 4:30pm
Effective Inter-process Communicationsin the Cloud: The Pros and Cons of Micro Services Architectures
ENT-209
Wednesday, 4:30pm
Cloud Migration, Dev-Ops and Distributed Systems
APP-310
Friday, 9:00am
Scheduling using Apache Mesosin the Cloud

89. Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals