DA Platform is a turnkey solution designed for stateful stream processing using Apache Flink, addressing challenges like job upgrades and configuration changes. It integrates with Kubernetes to manage deployments dynamically and offers tools for logging, metrics, and lifecycle management. The platform aims to simplify streaming applications while ensuring scalability and ease of integration within existing infrastructures.

1. Robert Metzger
Patrick Lucas
dA Platform
Production-ready stream processing with Apache Flink®

2. © 2018 data Artisans2
WHY WE BUILD DA PLATFORM
The need for
streaming platforms
The right tools from
day one
Operations for stateful
streaming applications

3. © 2018 data Artisans3
THE NEED FOR STREAMING PLATFORMS
DevOps model
Integration with
internal infrastructure
Integrate once,
scale adoption
within organization
Focus on use-cases
instead of infrastructure
• Logging
• Metrics
• State storage
• Data access
• Security
• Auditing

4. © 2018 data Artisans4
THE RIGHT TOOLS FROM DAY ONE

5. © 2018 data Artisans5
STATEFUL STREAMING OPERATIONS
Managing a stateful stream processor
on stateless containers is challenging.
STATEFUL STREAM
PROCESSING
Stateful stream processing
is disrupting analytics and
application development.
KUBERNETES AND THE
WORLD OF CONTAINERS
Kubernetes offers easy to use
container orchestration. But
containers are stateless.

6. © 2018 data Artisans6
STATEFUL STREAMING OPERATIONS
STATEFUL STREAM
PROCESSING
Stateful stream processing
is disrupting analytics and
application development.
KUBERNETES AND THE
WORLD OF CONTAINERS
Kubernetes offers easy to use
container orchestration. But
containers are stateless.

7. © 2018 data Artisans7
WHY WE BUILD DA PLATFORM
The need for
streaming platforms
Easy integration and
focus on use-cases
The right tools from
day one
Get started with
stream processing in
no time with a turnkey
solution
Operations for stateful
streaming applications
Bringing stateful
streaming into the world
of containers

8. © 2018 data Artisans8
HOW DOES IT WORK?

9. © 2018 data Artisans9
DECLARATIVE CONTROL OF DEPLOYMENTS
Kubernetes
Specify your Flink streaming deployment, dA Platform
takes care of configuring, deploying and operating it.
Status report /
Event Log
Log messages
from Flink
System
Metrics
Flink User
Interface
Deployment spec
state: running
job: fraud-detection.jar
parallelism: 60
flinkConfiguration: {…}
cpu: 8
…

10. © 2018 data Artisans10
DECLARATIVE CONTROL IN ACTION
Deployment spec
state: cancelled
job: fraud-detection.jar
parallelism: 60
flinkConfiguration: {…}
cpu: 8
…
In this case, the specification of a deployment
changed to “cancelled”.
The Flink cluster in Kubernetes will be deallocated.
Kubernetes
Changing the specification of a deployment will be
reflected in the cluster.

11. © 2018 data Artisans11
DECLARATIVE CONTROL IN ACTION
Deployment spec
state: running
job: fraud-detection.jar
parallelism: 60 -> 80
flinkConfiguration: {…}
cpu: 8
…
The Flink cluster will scale up to the specified size.
Kubernetes
Changing the specification of a deployment will be
reflected in the cluster.
Scale up cluster

12. © 2018 data Artisans12
MANAGING STREAMING APPS AND STATE
dA Platform allows to perform stateful
deployment changes.
Deployment changes:
• Upgrading the Flink job
• Upgrading Flink
• Changing the parallelism
• Changing resource allocations
• Changing a configuration parameter
• Suspending / resuming a deployment
• …
Scale up cluster
Kubernetes
Migrate StateMigrate state

13. © 2018 data Artisans13
VERSIONED APPLICATIONS, NOT JOBS/JARS
STREAM PROCESSING
APPLICATION
Version 3
Version 2
Version 1
Code and Application
State
upgrade
upgrade
NEW APPLICATION
Version 3a
Version 2a
fork /
duplicate

14. © 2018 data Artisans14
IMPLEMENTATION

15. © 2018 data Artisans15
ARCHITECTURE
Apache Flink
Stateful stream processing
Kubernetes
Container platform
Logging
Metrics
Real-time
Analytics
Anomaly- &
Fraud Detection
Real-time Data
Integration
Reactive
Microservices
dA
Application
Manager
Application lifecycle
management

16. © 2018 data Artisans16
ARCHITECTURE
All components of dA Platform are shipped as Docker containers
We provide an installer for setting up the platform
on Kubernetes
The containers for Flink follow the same versioning scheme as the
Apache Flink releases. The dAP distribution Flink has minor patches:
compatible with the open source releases.
Metrics and logging components are provided for demonstration
purposes. We recommend integration with existing systems.

17. © 2018 data Artisans17
APPLICATION MANAGER
Application Manager is the central orchestration and lifecycle management
component of dA platform.
Resource Management
Application
Manager
REST API
CI/CD Integration
Web interface
Resource
Allocation
Job
Control

18. © 2018 data Artisans18
APPLICATION MANAGER
Rich, interactive user interface
Based on the main REST API
Deployment of Flink clusters
on Kubernetes
Control Flink jobs
Generate configuration for metrics,
logging, HA, state backups,
security, …
Trigger savepoints, cancel jobs, start
jobs with savepoints, check Flink
health, …
Keep track of Flink jobs,
configuration parameters,
savepoints, events
Stored on a persistent volume
Declarative REST API
For integration with existing
systems such as CI/CD pipelines

19. © 2018 data Artisans19
HOOKS FOR CI/CD PIPELINES
Kubernetes Cluster
Application
Version 1
CI Service
dA Application
Manager
Trigger
CI
Call
API
Application
Version 1
Stateful Upgrade
Push
Update
Application
Version 2

20. © 2018 data Artisans20
DEMO

21. © 2018 data Artisans21
WRAP-UP

22. © 2018 data Artisans22
AVAILABILITY
• dA Platform has been announced at Flink
Forward Berlin 2017, with a closed beta program
‒ 190 beta signups, successful collaboration with a
large number of beta users
• As of March 2018, dA Platform is generally
available
• Trial versions are available for download:
data-artisans.com/download
‒ Recommendation: Download the trial virtual
machine!
• dA Platform license includes Enterprise Support
for Apache Flink
dA Platform Trial VM
Download Now

23. © 2018 data Artisans23
ROADMAP
Improved Access Control
Introduce ownership for resources
(deployments, savepoints, jobs), control
access based on ownership.
Metadata management
Awareness for data sources such as Kafka
topics, files in S3 or HDFS.
SQL
SQL will be a first class citizen of dA Platform
with interfaces for interactive queries and
integration with existing SQL tools.
Multi-Datacenter fail-over
Run dA Platform in redundant data centers,
automatically moving the processing to a
standby data center in case of a failure.

24. © 2018 data Artisans24
SUMMARY
dA Platform is a turnkey
solution for stateful stream
processing with Apache Flink.
Apache Flink
Stateful stream processing
Kubernetes
Container platform
Logging
Metrics
dA Application
Manager
Application lifecycle
management
It solves common challenges
such as job upgrades,
configuration changes or
state migrations..

25. © 2018 data Artisans25
Q & A

26. © 2018 data Artisans26
data-artisans.com
data-artisans.com/blog
@dataArtisans
data Artisans
data-artisans
STAY IN
TOUCH