Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elastic Streams at Scale @ Flink Forward 2018 Berlin

445 views

Published on

One of the big operational challenges when running streaming applications is to cope with varying workloads. Variations, e.g. daily cycles, seasonal spikes or sudden events, require that allocated resources are constantly adapted. Otherwise, service quality deteriorates or money is wasted. Apache Flink 1.5 includes a lot of enhancements to support full resource elasticity on cluster management frameworks such as Apache Mesos. With the latest version, it is now possible to build elastic applications which can be programmatically scaled up or down in order to react to changing workloads. In this talk, we will discuss recent improvements to Flink's deployment model which also enables full resource elasticity. In particular, we will discuss how Flink leverages cluster management frameworks, e.g. Mesos, and already-introduced features like scalable state to support elastic streaming applications. We will conclude the presentation with a short demo showing how a stateful Flink application can be rescaled on top of Mesos.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Elastic Streams at Scale @ Flink Forward 2018 Berlin

  1. 1. Elastic Streams at Scale @stsffap @joerg_schad
  2. 2. time Workload Resources time Workload Resources
  3. 3. • •
  4. 4.
  5. 5. • • • •
  6. 6. • • •
  7. 7. • •
  8. 8. • • • #1 #2 #3
  9. 9. partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 • Store offset for each partition • Individual entries are repartitionable
  10. 10. • • • •
  11. 11. • • •
  12. 12. • • • • • • •
  13. 13. • • • • • •
  14. 14. • • • • • • • • • • • •
  15. 15. © 2018 Mesosphere, Inc. All Rights Reserved. Flink Mesos Integration Mesos Master Mesos Cluster Client (2) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos ResourceManager JobManager (4) Start Process (and supervise) (8) Deploy Tasks (7) Register (5) Request slots Flink Mesos Dispatcher (3) Allocate container for Flink master (6) Allocate containers for TaskManagers Marathon (1) Start and monitor dispatcher Mesos Task TaskManager Mesos Task TaskManager
  16. 16. Mesos Master Mesos Cluster Cluster Client (1) Submit Mesos App. (JobGraph / JARs) Application Master Flink-Mesos ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn Application Master (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots
  17. 17. • Monitor
  18. 18. • • • • •
  19. 19. THANK YOU! @stsffap @dataArtisans @ApacheFlink WE ARE HIRING data-artisans.com/careers

×