Did you like it? Check out our blog to stay up to date: https://getindata.com/blog This workshop focuses on creating a data streaming platform from scratch using an empty Kubernetes (or even Minikube) cluster. During the workshop, we go through the installation process, deploy the basic components for the platform, start Apache Flink, and monitor the process, using SQL to query available data. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com​

1. Flink SQL
Workshop
Author: Albert Lewandowski

2. Overview
Apache Flink

3. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Data Streaming vs. Batch
Events
1 2 3 4 5 6
Batch
Stream

4. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Use cases
User activity
Fraud detection
Logistics
Industrial IoT
Location data
Recommendations

5. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
One tool, multiple languages
Java 8 or 11
Python
SQL
Scala 2.11 or 2.12

6. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Where should I install?
Standalone
Kubernetes
YARN cluster
● CICD process
● Service Discovery - monitoring with Prometheus
● Scalability
● Managing resources
● A/B Testing

7. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
High Availability of Flink
JobManager level
Storage level
● ZooKeeper
● Kubernetes (beta)
● High Availability of
storage to/from
which Flink
writes/reads
savepoints and
checkpoints
● Performance of
storage
Job Strategy
● Data reprocessing
policy
● How to deploy new
job?

8. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Flink K8S Operator
Kubernetes Operator for
Apache Flink
Ververica Platform
Native Kubernetes -
Apache Flink
CRDs Yes Yes No No
CICD Kubernetes API Kubernetes API REST API or Web UI Kubernetes API
Installation
Helm chart or raw
Kubernetes manifests
Helm chart or raw
Kubernetes manifests
Helm chart or raw
Kubernetes manifests
No need to install any
component
SQL Editor No No Yes No
Dependencies No No
Persistence volume for
database
Object storage for
artifactory
No
Status beta beta production beta

9. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Why Flink on Kubernetes?
Simpler deployment process Flexible jobs management
Simple Service Discovery -
Prometheus
Flexible testing

10. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Installation & Configuration
Helm
A package manager
for Kubernetes
CICD tool
Example: Gitlab CI
Kubernetes API
Flink jobs

11. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Testing
Incubating Mode
A/B Testing
Blue Green
Deployment
Production
data
Production
job
Job
Incubating
mode
Separated
output
Standard
output
Dedicated TaskManagers
Dedicated TaskManagers
savepoi
nt
Proxy
Flink Job
#1
Flink Job
#2
Result #1
Result #2

12. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Job Cluster & Session Cluster
Job Cluster Session Cluster
Full set of Flink cluster for
each individual job
Standalone Flink cluster on
Kubernetes
Short running tasks Ad-hoc queries
Long running tasks
Separate images for
different jobs

13. Overview
Ververica Platform

14. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Ververica Platform components

15. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Ververica Platform components
● AppManager
It sits as an orchestrator between a user’s requests,
Kubernetes, and Apache Flink
● Gateway
Container hosting the SQL service and processing any
SQL-related task.
● UI
App responsible for the Web User Interface

16. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Ververica Platform external resources (optional)
● Blob storage (like AWS S3)
It can be used to store artifacts if required
● IAM permissions
We can define which Kubernetes ServiceAccount with
associated IAM role we can use.
● Logging & Metrics
We can set up links to external components in which we
can read logs and metrics from Flink jobs.

17. © Copyright. All rights reserved. Not to be reproduced without prior written consent.

18. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Ververica Platform dictionary
● Deployment Target
DeploymentTargets correspond to different Kubernetes
namespaces in the same Kubernetes cluster.
● Deployment
A Deployment specifies the desired state of a Flink job and its
configuration. Ververica Platform tracks and reports it.
● Namespaces (in Ververica, not Kubernetes!)
Namespaces are the primary means to isolate resources between
different groups of users and grant access to resources, allowing for
multi-tenancy.

19. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Ververica Platform dictionary
● Session Cluster
Your Deployment will be executed in a Flink Session Cluster that
may be shared with other Deployments. The lifecycle of the Flink
cluster is independent of the Deployment lifecycle.
● Application Cluster
Your Deployment will be executed in a separate Flink cluster. The
lifecycle of the Flink cluster is tied to the lifecycle of the Deployment.

20. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Lifecycle Management
● Upgrade or suspend job
When a Deployment is suspended or a stateful upgrade is triggered,
Ververica Platform will submit a Stop-with-Savepoint command to
Apache Flink. This command will atomically trigger a Savepoint and
stop the job.
● Drain the pipeline
Draining emits the maximum watermark before stopping the job.
When the watermark is emitted, all event time timers will fire,
allowing you to process events that depend on this timer.
This is useful when you want to fully shut down your job without
leaving any unhandled events or state.

21. Quick start

22. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Repo with examples
git clone
https://github.com/getindata/ververica-platform-flink-workshop

23. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Prerequisites
● Container Runtime
● Installed Kubectl
● Installed Helm
Instruction can be found on Confluence.

24. Observability

25. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Observability
Observability is about measuring how well internal states of the
system can be inferred from knowledge of its external outputs
(according to the control theory).

26. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Part One: Metrics
Get metrics from environment and application - but how?

27. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Prometheus - Kubernetes-native solution
open-source systems
monitoring and alerting toolkit
joined the Cloud Native Computing
Foundation in 2016 as the second
hosted project, after Kubernetes
a lot of exporters
you can write your own easily
mature ecosystem
PushGateway, Blackbox, AlertManager, etc.

28. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Pull vs. push-based monitoring
Pull Push
Collector takes metrics Agents push metrics
Workload on central poller increases with the number of
devices polled.
Polling task fully distributed among agents, resulting in
linear scalability.
Polling protocol can potentially open up system to
remote access and denial of service attacks.
Push agents are inherently secure against remote
attacks since they do not listen for network connections.
Flexible: poller can ask for any metric at any time.
Relatively inflexible: pre-determined, fixed set of
measurements are periodically exported.

29. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Prometheus - Stories
service discovery
simple on k8s
limited security
archived data
how old data is required?
monitor monitoring

30. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Part Two: Logs analytics
1. Get logs from app or environment.
2. Save logs.
3. Query them.
4. Make your system self-healing and discover what’s
happening inside your platform.

31. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Logs analytics - which tool should I choose?
Logs Analytics for Developers Logs Analytics for Business
Loki ElasticSearch

32. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
ELK vs. Loki
ELK Loki + Promtail/Fluentd
Indexing Keys and content of each key Only labels
Query language Query DSL or Lucene QL LogQL
Tool for data visualisation Kibana Grafana
Query performances Faster due to indexed all the data Slower due to indexing only labels
Resource requirements Higher due to the need of indexing Lower due to index only labels

33. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
What about alerts?
Alerts signify that
a human needs to take action
immediately
in response to something that is
either happening or about to
happen, in order to improve the
situation.

34. Usage

35. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
How to deploy job?
Web UI REST API

36. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Local Artifactory
● Ververica provides its own artifactory
● JARs can be also downloaded from the available source
like Nexus

37. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Flink job & AWS
● We can connect Flink job with our IAM user or IAM role
● Flink supports AWS IAM credentials by default
● We can run Flink via EMR or Kinesis

38. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Flink job & GCP
● We can connect Flink job with our IAM user or IAM role
● Flink supports AWS IAM credentials by default
● We can run Flink via Dataproc or Dataflow (as Beam
supports Flink jobs)

39. Useful
Commands

40. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Check Flink jobs status
● Check status of TaskManagers and JobManager
kubectl -n flow-ververica-jobs get po
● Check any issues with deployment
TaskManagers
kubectl -n flow-ververica-jobs get deployment
kubectl -n flow-ververica-jobs describe deployment
DEPLOYMENT_NAME - check Events tab
JobManagers
kubectl -n flow-ververica-jobs get job
kubectl -n flow-ververica-jobs describe job
DEPLOYMENT_NAME - check Events tab

41. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Check Flink jobs status
● Check volume claim used by Ververica
kubectl -n flow-ververica-jobs get pvc
● Check volume used by Ververica
kubectl get pv

42. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Get logs from application
● Ververica compontents
kubectl -n flow-ververica logs $(kubectl -n flow-ververica
get po | awk 'FNR == 2 {print $1}') CONTAINER_NAME
Here you can pass: appmanager, gateway or ui
● Flink job
kubectl -n flow-ververica-jobs get po
kubectl -n flow-ververica-jobs logs
POD_NAME_FROM_1st_COMMAND
To follow all newly appeared logs, add flag -f just after logs, example:
kubectl -n flow-ververica-jobs logs -f POD

43. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Get logs from application
● Access Exception from Flink Web User Interface

44. © Copyright. All rights reserved. Not to be reproduced without prior written consent.
Q&A
https://linkedin.com/in/albert-lewandowski
albert.lewandowski@getindata.com

45. Thank you for your
attention!