This document discusses Elastic Streams at scale using Apache Flink and Mesos. It describes how Flink jobs can be deployed on Mesos clusters by having the Flink master process request resources from the Mesos resource manager. The resource manager then allocates Mesos containers for the Flink master and task managers, allowing the Flink processes to be deployed and tasks to run on the Mesos cluster resources. A Mesos dispatcher can be used to start and monitor the Flink master process.

1. Elastic Streams at Scale
@stsffap @joerg_schad

3. time
Workload
Resources
time
Workload
Resources

5. •
•

6. •

7. •
•
•
•

8. •
•
•

9. •
•

10. •
•
•
#1 #2
#3

11. partitionId: 1, offset: 42
partitionId: 3, offset: 10
partitionId: 6, offset: 27
partitionId: 1, offset: 42
partitionId: 3, offset: 10
partitionId: 6, offset: 27
• Store offset for each partition
• Individual entries are repartitionable

12. •
•
•
•

14. •
•
•

15. •
•
•
•
•
•
•

17. •
•
•
•
•
•

19. •
•
•
•
•
•
•
•
•
•
•
•

20. © 2018 Mesosphere, Inc. All Rights Reserved.
Flink Mesos Integration
Mesos Master
Mesos Cluster
Client
(2) HTTP POST
JobGraph/Jars
Flink Master Process
Flink Mesos
ResourceManager
JobManager
(4) Start Process
(and supervise)
(8) Deploy
Tasks
(7) Register
(5) Request slots
Flink Mesos
Dispatcher
(3) Allocate
container
for Flink master
(6) Allocate
containers
for TaskManagers
Marathon
(1) Start and
monitor
dispatcher
Mesos Task
TaskManager
Mesos Task
TaskManager

21. Mesos
Master
Mesos Cluster
Cluster Client
(1) Submit Mesos App.
(JobGraph / JARs)
Application Master
Flink-Mesos
ResourceManager
JobManager TaskManager
TaskManager
TaskManager
(2) Spawn
Application Master
(4) Start
TaskManagers
(6) Deploy
Tasks
(5) Register
(3) Request slots

22. •
Monitor

23. •
•
•
•
•

24. THANK YOU!
@stsffap
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers