Seven ways to run Apache Flink on AWS resources

2. whoami
Software Engineer with
Focus on
Data-Intensive Applications
Site Reliability Engineering

3. 
Stateful Computations over Data Streams
Deployment
YARN
Mesos
Kubernetes
Apache Flink
Apache Flink
apache/flink

4. Stream Processor
Stream Processor
1. "Functional" DSL
1. "Functional" DSL
3. Parallel Execution Graph
3. Parallel Execution Graph
2. Logical Execution Graph
2. Logical Execution Graph
from("json-events").map(MyPojo::fromJson)
.keyBy(MyPojo::getUserId)
.timeWindow(SECONDS.of(10))
.count()
.to("events-per-user")

5. Flink stack
Flink stack

6. Flink cluster
Flink cluster
Job Manager
Task Manager Task Manager Task Manager

7. Use case
Use case

9. The recommended way
The recommended way
EMR
EMR

10. https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/aws.html

12. EMR Architecture
EMR Architecture
Job Manager
Task Manager Task Manager Task Manager
YARN Container YARN Container YARN Container
YARN Container

14. 👍
👍
Recommended
Battle-tested
👎
👎
Need to manage Hadoop cluster
Flink version lag (currently 1.8.0)

15. The do-it-yourself way
The do-it-yourself way
EC2
EC2

16. EC2 Architecture
EC2 Architecture
Job Manager
Task Manager Task Manager Task Manager

17. Reproducibility
Reproducibility


ansible/ansible
hashicorp/packer

18. 👍
👍
Fully customizable
👎
👎
Need to build everything on your own
Need to manage EC2 instances

19. The containerized way
The containerized way
ECS + EC2
ECS + EC2

20. ECS Architecture (with EC2)
ECS Architecture (with EC2)
Job Manager
Task Manager Task Manager Task Manager
ECS Task
ECS Service
ECS Task ECS Task ECS Task
ECS Service

21. 👍
👍
Useful model
👎
👎
Proprietary primitives
Not common, little community support
Need to manage EC2 instances

22. The CaaS way
The CaaS way
ECS + Fargate
ECS + Fargate

23. ECS Architecture (with Fargate)
ECS Architecture (with Fargate)
Job Manager
Task Manager Task Manager Task Manager
ECS Task
ECS Service
ECS Task ECS Task ECS Task
ECS Service

24. 👍
👍
Useful model
No instance-level management required
👎
👎
Proprietary primitives
Not common, little community support

25. The cloud-native way
The cloud-native way
EKS
EKS

26. Kubernetes
Kubernetes
… is an
open source system for
managing containerized applications
across multiple hosts;
providing basic mechanisms for
deployment,
maintenance, and
scaling of applications.

27. EKS Architecture
EKS Architecture
Job Manager
Task Manager Task Manager Task Manager
Kubernetes pod Kubernetes pod Kubernetes pod
Kubernetes pod
Kubernetes deployment
Kubernetes deployment

29. 

weaveworks/eksctl
lyft/flinkk8soperator
Ververica Platform

30. 👍
👍
Powerful primitives
De-facto standard
Strong ecosystem
👎
👎
Need to manage Kubernetes cluster

31. The hosted way
The hosted way
Kinesis Data Analytics for Java
Kinesis Data Analytics for Java

32. Kinesis Data Analytics for Java Architecture
Kinesis Data Analytics for Java Architecture

34. 👍
👍
Fully hosted
State migration feature
👎
👎
Restricted to specific use case
Failure scenarios
Flink version lag (currently 1.6)

35. The serverless way
The serverless way
Lambda
Lambda

36. Lambda Architecture
Lambda Architecture

38. 👍
👍
Fun experiment
👎
👎
Not a good fit

39. Recap
Recap
EMR
The recommended way
EC2
The do-it-yourself way
ECS + EC2
The containerized way
ECS + Fargate
The CaaS way
EKS
The cloud-native way
Kinesis Analytics for Java
The hosted way
Lambda
The serverless way

40. What's next?
What's next?
State management & fault tolerance



Monitoring & alerting


Continuous Deployment
Infrastructure as Code

Tuning



Checkpointing
Upgrading Applications and Flink Versions
Job Manager High Availability
mbode/flink-prometheus-example
Monitoring Flink with Prometheus (Flink Forward 2018)
hashicorp/terraform
Data Types & Serialization
Tuning Checkpoints and Large State
Improving throughput and latency with Flink's network stack (Flink Forward 2018)

41.  
 
maximilian.bode@tng.tech @mxpbode
maximilianbo.de mbode