Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Running
Apache Flink®
Everywhere
Stephan Ewen (@StephanEwen)
How is Flink deployed?
2
Standalone Cluster Embedded Service (OSGI)
YARN Sessions
Standalone Cloud
Docker on Mesos
Docker/...
How is Flink deployed?
3
Standalone Cluster Embedded Service (OSGI)
YARN Sessions
Standalone Cloud
Docker on Mesos
Docker/...
How is Flink deployed?
4
Standalone Cluster Embedded Service (OSGI)
YARN Sessions
Standalone Cloud
Docker on Mesos
Docker/...
Resource Management
5
Standalone Cluster Embedded Service (OSGI)
YARN Sessions
Standalone Cloud
Docker on Mesos
Docker/Kub...
More dimensions coming up…
6
Dynamic Resources
• Number of TaskManagers changes
over job lifetime
"Trusted" processes
• Ru...
Reworking the Flink
Process Model
7
Flink Improvement Proposal 6
8
Currently driving parties:
Core Idea
• Creating composable building blocks
• Create differe...
Recap: Current status (Standalone)
9
Standalone Flink Cluster
Client (2) Submit Job
JobManager
TaskManager
TaskManager
Tas...
Recap: Current status (YARN)
10
YARN
ResourceManager
YARN Cluster
Client
(1) Submit YARN App.
(FLINK)
Application Master
J...
The Building Blocks
11
• ClusterManager-specific
• May live across jobs
• Manages available Containers/TaskManagers
• Used...
The Building Blocks
12
ResourceManager
(1) Request slots TaskManager
JobManager
(2) Start
TaskManager
(3) Register
(4) Dep...
Building Flink-on-YARN
13
YARN
ResourceManager
YARN Cluster
YARN Cluster
Client
(1) Submit YARN App.
(JobGraph / JARs)
App...
Building Flink-on-YARN
Main differences from current YARN mode
 All containers started with JARs, config files in classpa...
Building Flink-on-YARN (separate RM)
15
YARN
ResourceManager
YARN Cluster
YARN Cluster
Client
(1) Submit YARN App.
(JobGra...
Building Flink-on-YARN (w/ dispatcher)
16
YARN
ResourceManager
YARN Cluster
YARN Cluster
Client
(1) HTTP POST
JobGraph/Jar...
Building Flink-on-Mesos
17
Mesos Master
Mesos Cluster
Mesos Cluster
Client
(1) HTTP POST
JobGraph/Jars
Flink Master Proces...
Building Standalone
18
Standalone Cluster
Flink Cluster
Client
(1) Submit
JobGraph/Jars
Flink Master Process
Standalone
Re...
Master Container
Flink Master Process
Building Flink-on-Docker/K8S
19
Flink-Container
ResourceManager
JobManager
Program R...
Building Flink-on-Docker/K8S
 This is a blueprint for all setups where external services control
resources and start new ...
Multi-Job Sessions
21
Example: YARN session
ApplicationMaster
Flink-YARN
ResourceManager
(5) Request
slots
JobManager
(A)
JobManager
(B)
Dispatc...
Sessions vs. Jobs
 For each Job submitted, the session will spawn its own JobManager
 All jobs run under session-user cr...
Wrap-up
24
More stuff
 Dynamically acquire/release resources
• Slots are allocated/released from Resource Manager as needed
• Resour...
Wrapping it up
 It’s a zoo of cluster managers out there
• Following different paradigms
 Usage patterns vary because of...
Appendix
27
Flink Streaming cornerstones
28
Low latency
High Throughput
Well-behaved
flow control
(back pressure)
Make more sense of d...
Upcoming SlideShare
Loading in …5
×

Stephan Ewen - Running Flink Everywhere

1,620 views

Published on

http://flink-forward.org/kb_sessions/running-apache-flink-everywhere-standalone-yarn-mesos-docker-kubernetes-etc/

The world of cluster managers and deployment frameworks is getting complicated. There is zoo of tools to deploy and manage data processing jobs, all of which have different resource management and fault tolerance slightly different. Some tools have a only per-job processes (Yarn, Docker/Kubernetes), while others require some long running processes (Mesos, Standalone). In some frameworks, streaming jobs control their own resource allocation (Yarn, Mesos), while for other frameworks, resource management is handled by external tools (Kubernetes). To be broadly usable in a variety of setups, Flink needs to play well with all these frameworks and their paradigms. This talk describes Flink’s new proposed process and deployment model that will make it work together well with the above mentioned frameworks. The new abstraction is designed to cover a variety of use cases, like isolated single job deployments, sessions of multiple short jobs, and multi-tenant setups.

Published in: Data & Analytics
  • Be the first to comment

Stephan Ewen - Running Flink Everywhere

  1. 1. Running Apache Flink® Everywhere Stephan Ewen (@StephanEwen)
  2. 2. How is Flink deployed? 2 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs A two minute search on the mailing list reveals
  3. 3. How is Flink deployed? 3 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs A two minute search on the mailing list reveals Mesos Sessions Mesos Jobs (soon!)
  4. 4. How is Flink deployed? 4 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs Users run mostly isolated jobs or multi-job sessions Mesos Sessions Mesos Jobs
  5. 5. Resource Management 5 Standalone Cluster Embedded Service (OSGI) YARN Sessions Standalone Cloud Docker on Mesos Docker/Kubernetes YARN->Myriad->Mesos YARN Jobs Resources controlled by the framework or another service. Mesos Sessions Mesos Jobs
  6. 6. More dimensions coming up… 6 Dynamic Resources • Number of TaskManagers changes over job lifetime "Trusted" processes • Run under superuser credential and dispatch jobs No blocking on any process type • YARN job needs to continue while ApplicationMaster is down Uniform vs. Heterogeneous Resources • Run different functions in different size containers • E.g., simple mapper in small container, heavy window operator in large container Avoiding "Job Submit" step
  7. 7. Reworking the Flink Process Model 7
  8. 8. Flink Improvement Proposal 6 8 Currently driving parties: Core Idea • Creating composable building blocks • Create different compositions for different scenarios FLIP-6 design document: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077
  9. 9. Recap: Current status (Standalone) 9 Standalone Flink Cluster Client (2) Submit Job JobManager TaskManager TaskManager TaskManager (3) Deploy Tasks (1) Register
  10. 10. Recap: Current status (YARN) 10 YARN ResourceManager YARN Cluster Client (1) Submit YARN App. (FLINK) Application Master JobManager TaskManager TaskManager TaskManager (2) Spawn AppMaster (4) Start TaskManagers (8) Deploy Tasks (3) Poll status (6) All TaskManager started (5) Register (7) Submit Job
  11. 11. The Building Blocks 11 • ClusterManager-specific • May live across jobs • Manages available Containers/TaskManagers • Used to acquire / release resources ResourceManager TaskManagerJobManager • Registers at ResourceManager • Gets tasks from one or more JobManagers • Single job only, started per job • Thinks in terms of "task slots" • Deploys and monitors job/task execution Dispatcher • Lives across jobs • Touch-point for job submissions • Spawns JobManagers • May spawn ResourceManager
  12. 12. The Building Blocks 12 ResourceManager (1) Request slots TaskManager JobManager (2) Start TaskManager (3) Register (4) Deploy Tasks
  13. 13. Building Flink-on-YARN 13 YARN ResourceManager YARN Cluster YARN Cluster Client (1) Submit YARN App. (JobGraph / JARs) Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn AppMaster (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots
  14. 14. Building Flink-on-YARN Main differences from current YARN mode  All containers started with JARs, config files in classpath  Credentials & Secrets are strictly bound to a single job  Slots are allocated/released as needed/freed • Basic building block for elastic resource usage  Client disconnects after submitting job, does not need to wait until TaskManagers are up 14
  15. 15. Building Flink-on-YARN (separate RM) 15 YARN ResourceManager YARN Cluster YARN Cluster Client (1) Submit YARN App. (JobGraph / JARs) Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn AppMaster (4) Start TaskManagers (6) Deploy Tasks (5) Register(4) Request slots (3) Start JobMngr
  16. 16. Building Flink-on-YARN (w/ dispatcher) 16 YARN ResourceManager YARN Cluster YARN Cluster Client (1) HTTP POST JobGraph/Jars Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (3) Spawn AppMaster (5) Start TaskManagers (7) Deploy Tasks (6) Register Flink YARN Dispatcher (2) Submit YARN App. (JobGraph / JARs) (4) Request slots
  17. 17. Building Flink-on-Mesos 17 Mesos Master Mesos Cluster Mesos Cluster Client (1) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos ResourceManager JobManager TaskManager TaskManager TaskManager (3) Start Process (and supervise) (5) Start TaskManagers (7) Deploy Tasks (6) Register (4) Request slots Flink Mesos Dispatcher (2) Allocate container for Flink master
  18. 18. Building Standalone 18 Standalone Cluster Flink Cluster Client (1) Submit JobGraph/Jars Flink Master Process Standalone ResourceManager TaskManager TaskManager TaskManager (7) Deploy Tasks (1) Register (3) Request slots JobManager JobManager Dispatcher (2) Start JobMngr Standby Master Process Standby Master Process
  19. 19. Master Container Flink Master Process Building Flink-on-Docker/K8S 19 Flink-Container ResourceManager JobManager Program Runner (2) Run & Start Worker Container TaskManager Worker Container TaskManager Worker Container TaskManager (3) Register (1) Container framework starts Master & Worker Containers (4) Deploy Tasks
  20. 20. Building Flink-on-Docker/K8S  This is a blueprint for all setups where external services control resources and start new TaskManagers • For example AWS EC2 Flink image with auto-scaling groups  Can be extended to have N equal containers, out of which one becomes master, remainder workers  With upcoming dynamic-scaling feature (see Till's talk), JobManager scales job to use all available resources 20
  21. 21. Multi-Job Sessions 21
  22. 22. Example: YARN session ApplicationMaster Flink-YARN ResourceManager (5) Request slots JobManager (A) JobManager (B) Dispatcher (4) Start JobMngr YARN ResourceManager YARN Cluster Client (1) Submit YARN App. (FLINK – session) TaskManager TaskManager TaskManager (2) Spawn AppMaster (6) Start TaskManagers (8, 12) Deploy Tasks (7) Register (3) Submit Job A (11) Request slots (10) Start JobMngr (9) Submit Job B 22
  23. 23. Sessions vs. Jobs  For each Job submitted, the session will spawn its own JobManager  All jobs run under session-user credentials  ResourceManager holds on to containers for a certain time • Jobs quickly following one another reuse containers (quicker response)  Internally, sessions build on the dispatcher component 23
  24. 24. Wrap-up 24
  25. 25. More stuff  Dynamically acquire/release resources • Slots are allocated/released from Resource Manager as needed • ResourceManager allocates/releases containers over time • Strong interplay with "Dynamic Scaling" (rf. talk by Till yesterday)  Resource Profiles: Containers of different size • Requests can pass a "profile" (CPU / memory / disk), or simply use "default profile" • Resource Managers YARN & Mesos can allocate respective containers 25
  26. 26. Wrapping it up  It’s a zoo of cluster managers out there • Following different paradigms  Usage patterns vary because of Flink's broad use cases • Isolated long running jobs vs. many short-lived jobs • Shared clusters vs. per-user authenticated resources  We are making "jobs" and "sessions" explicit constructs  Flexible building blocks, composed in various ways to accommodate different scenarios 26
  27. 27. Appendix 27
  28. 28. Flink Streaming cornerstones 28 Low latency High Throughput Well-behaved flow control (back pressure) Make more sense of data Works on real-time and historic data Performant Streaming Event Time APIs Libraries Stateful Streaming Globally consistent savepoints Exactly-once semantics for fault tolerance Windows & user-defined state Flexible windows (time, count, session, roll-your own) Complex Event Processing

×