Flink Forward Berlin 2017: Dominik Bruhn - Deploying Flink Jobs as Docker Containers

•

3 likes•751 views

This talk will focus on how to package, distribute and deploy Flink Jobs by leveraging existing docker technology: Previously deploying of Flink Jobs has been a manual job which leads into errors. In this talk, we present an approach which works well in an CI/CD environment by automating most steps: From the code of a Flink Job in a repository to a running Job on an YARN cluster.

Data & Analytics

Deploying Flink Jobs
as Docker Containers
Dominik Bruhn - Director of Platform Engineering Relayr
Flink Forward Sep 12-14 2017 Berlin

Who am I?
● Director of Platform Engineering
● At Relayr
○ Industrial IOT Platform
○ Real Time Sensor Data is processed, stored, analyzed and presented
● Contributor to Apache Flink

Production
Production
What is this talk about?
Source
Flink Job Production???

What are the Requirements?
● Streaming Jobs
● Endless Jobs
● Deployed in Multiple Environments (Configuration Files)
● Single Deployed Artefact
● Deploy to YARN (EMR) cluster
● Repeatable Builds

What is the Idea?
“If it behaves like a service, package it like a service”
● Docker Container contains the Job + Flink + Some
scripting
● Docker Container submits and monitors the Flink Job in
the YARN cluster.
● Actual computation happens in the YARN cluster
● Docker Container stays attached to the job

Steps Necessary I - Building
Source
Flink Job
Docker
ContainerCompile
Test
Package
Upload

Packaging
● Result from compliation: Job Fat JAR
○ Exclude Flink from the Fat JAR
● Package into a Docker Container:
○ Java
○ Apache Flink Distribution
○ Job Fat JAR
○ Configuration File (Templates)
○ If needed: Utilities for Configuration Fetching
○ Entrypoint Shell Script

Steps Necessary II - Executing
Docker
Container
Docker Runtime YARN Cluster
Flink Job
Configure
Submit
Monitor

What does the Container do?
1. Fetch Configuration Values + YARN Credentials
2. Get YARN Configuration
3. Update Configuration File of Job + Flink
4. List YARN Jobs, find old running
a. If found, kill
5. Find out the last savepoint on HDFS
6. Start job on YARN from savepoint
7. Stay attached to the Job

What do we get from this?
● Flink Job deployed as all other services
● Adapting to different environment
● Monitoring and failing like other services

What was left out?
● Packaging as stand alone
docker container (i.e. for
testing)
What could be done in the
future?
● Approach independent of
YARN on a stand alone Flink
cluster.
● Use other resource
management tools instead of
YARN, i.e. kubernetes.

What's hot

Many stream processing applications can benefit from or need to rely on the prediction made with machine learning (ML) methods. In this presentation, new features of Apache Samoa are presented with a real data processing scenario. These features make Apache SAMOA fully accessible for Apache Flink users: (1) the data stream processed within Apache Flink is forwarded to Apache Samoa stream mining engine to perform predictions with stream-oriented ML models, (2) ML models evolve after every labelled instance and, at the same time, new predictions are sent back to Apache Flink. In both cases, Apache Kafka is used for data exchange. Hence, Apache Samoa is used as stream mining engine, provided with input data from, and sending predictions to Apache Flink. During the presentation, real life aspects are illustrated with code examples, such as input and prediction stream integration and monitoring latency of data processing and stream mining.

Flink Forward Berlin 2017: Piotr Wawrzyniak - Extending Apache Flink stream p...

Flink Forward

Apache Beam was open sourced by the big data team at Google in 2016, and has become an active community with participants from all over. Beam is a framework to define data processing workflows and run them on various runners (Flink included). In this talk, I will talk about some cool things you can do with Beam + Flink such as running pipelines written in Go and Python; then I’ll mention some cool tools in the Beam ecosystem. Finally, we’ll wrap up with some cool things we expect to be able to do soon - and how you can get involved.

Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...

Flink Forward

The windowing capabilities offered by most stream processing engines are limited to aligned windows of a fixed duration. However, many real-world event processing use cases don’t fit this rigid structure, resulting in awkward processing pipelines. There haven’t been good alternatives, until recently that is. Apache Flink offers a rich Window API that supports implementing unaligned windows of varying duration. In this talk, Matt Zimmer will discuss using this API at Netflix to aggregate events into windows customized along varying definitions of a session. He will talk about implementation details such as: * Handling out-of-order events * Limiting state build-up while aggregating a subset of events from an event stream * Periodically emitting early results * Creating windows bounded by a type of event Attendees will leave this talk with practical techniques and knowledge to implement their own custom windows in Apache Flink.

Flink Forward Berlin 2017: Matt Zimmer - Custom, Complex Windows at Scale Usi...

Flink Forward

During last two major versions (1.9 & 1.10), Apache Flink community spent lots of effort to improve the architecture for further unified batch & streaming processing. One example for that is Flink SQL added the ability to support multiple SQL planners under the same API. This talk will first discuss the motivation behind these movements, but more importantly will have a deep dive into Flink SQL. The presentation shows the unified architecture to handle streaming and batch queries and explain how Flink translates queries into the relational expressions, leverages Apache Calcite to optimize them, and generates efficient runtime code for execution. Besides, this talk will also describe the lifetime of a query in detail, how optimizer improve the plan based on relational node patterns, how Flink leverages binary data format for its basic data structure, and how does certain operator works. This would give audience better understanding of Flink SQL internals.

Virtual Flink Forward 2020: A deep dive into Flink SQL - Jark Wu

Flink Forward

Modern software development is increasingly taking a “microservice” approach that has resulted in an explosion of complexity at the network level. We have more applications running distributed across different datacenters. Distributed tracing, events, and metrics are essential for observing and understanding modern microservice architectures. This talk is a deep dive on how to monitor your distributed system. You will get tools, methodologies, and experiences that will help you to realize what your applications expose and how to get value out from all these information. Gianluca Arbezzano, SRE at InfluxData will share how to monitor a distributed system, how to switch from a more traditional monitoring approach to observability. Stay focused on the server’s role and not on the hostname because it’s not really important anymore, our servers or containers are fast moving part and it’s easy to detach it from the right in case of trouble than call the server by name as a cute puppet. How to design a SLO for your core services and now to iterate on them. Instrument your services with tracing using tools like Zipkin or Jaeger to measure latency between in your network.

OSDC 2018 - Distributed monitoring

Gianluca Arbezzano

Flink Connector Development Tips & Tricks

Eron Wright

At Lyft we dynamically price our rides with a combination of various data sources, machine learning models, and streaming infrastructure for low latency, reliability and scalability. Dynamic pricing allows us to quickly adapt to real world changes and be fair to drivers (by say raising rates when there's a lot of demand) and fair to passengers (by let’s say offering to return 10 mins later for a cheaper rate). The streaming platform powers pricing by bringing together the best of two worlds using Apache Beam; ML algorithms in Python and Apache Flink as the streaming engine. https://sf-2019.flink-forward.org/conference-program#streaming-your-lyft-ride-prices

Streaming your Lyft Ride Prices - Flink Forward SF 2019

Thomas Weise

Nowadays the Kappa Architecture is surely one of the best architectural pattern to implement a streaming system. While the choice for the log / journal side is usually straightforward thanks to engines like Apache Kafka, DistributedLog and Pravega, perfectly fitting the write side of this architecture, we didn’t find an open source counterpart able to fully satisfy all the requirements we believe are essential for a time series database such as: high availability, partition tolerance, optimized time series management, security, out of the box Apache Flink integration, ad-hoc front-end streaming features based on WebSocket protocol and natural real-time Analytics readiness. For this reason we took the decision to start the development of NSDB (Natural Series DB). During this talk we will introduce the main concepts behind the ideation of NSDB focusing on our starting goals and its architecture giving an overview of its first draft implementation. We will eventually provide an explanation on how it leverages Akka cluster and how it partitions data on a time basis.

Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...

Flink Forward

Developing and operating real-time applications with Oceanus The Tencent Data team (data.qq.com) is responsible to build a reliable and scalable infrastructure for various products at Tencent including Tencent Games, Tencent Video and WeChat. We have to process over 1.6 trillion events per day and the peak throughput reaches 210 million per second. Such a large scale exposes great challenges in the developing and operating of real-time applications. In this talk, we will give an introduction to Oceanus, a platform built at Tencent to facilitate the development of real-time applications. We will introduce typical user cases and how applications are developed, deployed and monitored with Oceanus. We will also introduce the management of Flink jobs in Oceanus, together with some experience gained in our operating practice. Finally, we will discuss some efforts we have done to enhance the reliability and efficiency, e.g. master failover and local keyed streams.

Flink Forward San Francisco 2019: Developing and operating real-time applicat...

Flink Forward

Managing Flink on Kubernetes - FlinkK8sOperator The goal of Lyft is to “Improve people’s lives with the world’s best transportation”. Our product is fundamentally real-time and building a reliable platform that consumes and processes massive amounts of streaming data empowers us to achieve our mission. The advent of containers and Kubernetes has completely changed how we deploy and manage stateless services. At Lyft we have doubled down on Docker containers and Kubernetes for all the services in production. To achieve a homogenous infrastructure we decided to extend Kubernetes to manage stateful streaming services like Flink. We developed the FlinkK8sOperator which leverages Kubernetes CustomResourceDefinition to enable native management of Flink applications on Kubernetes. FlinkK8sOperator employs a state machine that transitions the application through a series of states, until a stable state is attained. Each Flink application on Kubernetes spins up a separate Flink Cluster, with its own UI, providing clear isolation for monitoring and debugging. This talk provides an overview of running Flink applications on Kubernetes using FlinkK8sOperator, showcasing the entire lifecycle of the application from creation to execution, with focus of transitions during deployments and stateful updates, concluding with a demo.

Flink Forward San Francisco 2019: Managing Flink on Kubernetes - FlinkK8sOper...

Flink Forward

Moving from Lambda and Kappa Architectures to Kappa+ at Uber Kappa+ is a new approach developed at Uber to overcome the limitations of the Lambda and Kappa architectures. Whether your realtime infrastructure processes data at Uber scale (well over a trillion messages daily) or only a fraction of that, chances are you will need to reprocess old data at some point. There can be many reasons for this. Perhaps a bug fix in the realtime code needs to be retroactively applied (aka backfill), or there is a need to train realtime machine learning models on last few months of data before bringing the models online. Kafka's data retention is limited in practice and generally insufficient for such needs. So data must be processed from archives. Aside from addressing such situations, enabling efficient stream processing on archived as well as realtime data also broadens the applicability of stream processing. This talk introduces the Kappa+ architecture which enables the reuse of streaming realtime logic (stateful and stateless) to efficiently process any amounts of historic data without requiring it to be in Kafka. We shall discuss the complexities involved in such kind of processing and the specific techniques employed in Kappa+ to tackle them.

Flink Forward San Francisco 2019: Moving from Lambda and Kappa Architectures ...

Flink Forward

Container technology experiences an ever increasing adoption throughout many industries. Not only does this technology make your applications portable across different machines and operating systems, it also allows to scale applications in a matter of seconds. Moreover, it significantly simplifies and speeds up deployments which decreases development and operation costs. Consequently, more and more Flink deployments run in containerized environments which poses new challenges for Flink. In this talk, we will take a look at Flink's current and future container support which will make it a first class citizen of the container world. First of all, we will explain how the new reactive execution mode will solve the problem of seamless application scaling and how it blends in with any environment. Complementary to the reactive mode, the active execution mode demonstrates its strengths when it comes to changing workloads such as batch jobs. Last but not least, we will take a look beyond Flink's own nose and investigate how Flink can be used together with Kubernetes operators or data Artisans' Application Manager. We will conclude the talk with a short demo of Flink's native Kubernetes support and giving an outlook on future developments in the container realm.

Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...

Till Rohrmann

Last year we (TouK) introduced Flink in one of the biggest polish telcoms in the domain of real time marketing and fraud detection. One of the most significant problems in adoption was lack of programming skills at our client - the users were supposed to be analytics/business people. Therefore, we developed a custom platform - TouK Nussknacker - which allows users to design processes with GUI by drawing diagrams. Our project is going to be open-sourced soon - this will happen before Flink Forward. We believe it can make stream processing with Flink more accessible in many use cases, especially in companies that don't have their own development teams. During the talk I’m going to describe architecture of our platform, why we made certain design decisions and about our future plans. I’ll also describe our experiences - when being able to use GUI is great and when it’s better to develop jobs as normal code. If time permits I’ll also show a quick demo of our solution.

Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...

Flink Forward

End to-end example: consumer loan acceptance scoring using kubeflow

Radovan Parrak

AirStream is a realtime stream computation framework that supports Flink as one of its processing engines. It allows engineers and data scientists at Airbnb to easily leverage Flink to build real time data pipelines and feedback loops. Multiple mission critical applications have been built on top of it. In this talk, we will start with an overview of AirStream, and describe how we have designed Airstream to leverage SQL support in Flink to allow users to easily build real time data pipelines. We will go over a few production use cases such as building a user activity profiler and building user identity mapping in realtime. We will also cover how we have integrated Airstream into the data infrastructure ecosystem at Airbnb through easily configurable connectors such as Kafka and Hive that allow users to easily leverage these components in their pipelines.

Flink Forward San Francisco 2019: Building production Flink jobs with Airstre...

Flink Forward

Kubeflow Control Plane 中文

Weiqiang Zhuang

At Yelp we run hundreds of Flink jobs to power a wide range of applications: push notifications, data replication, ETL, sessionizing and more. Routine operations like deploys, restart, and savepointing for so many jobs would take quite a bit of developers’ time without the right degree of automation. The latest addition to our toolshed is a Kubernetes operator managing the deployment and the lifetime of Flink clusters on PaaSTA, Yelp’s Platform As A Service. We replaced our deployment framework launching Flink clusters on top of AWS EMR with a Kubernetes operator managing fully Docker-ized Flink clusters. Compared to EMR, this architecture allowed us to both drastically reduce the deployment time of our Flink clusters and to share our hardware resources more efficiently. In addition, we now offer to our developers the same interface they are used to for running REST services, batch jobs and many other workloads on PaaSTA. This talk will give a brief overview of Yelp’s PaaSTA before diving into the details of how the Kubernetes operator has been implemented and how it has been integrated with Yelp developers’ workflow (deploy, logs, savepoints, upgrades, etc), to end with a glimpse of the future features we are planning for the operator (Flink as a library, autoscaling, etc.).

Kubernetes + Operator + PaaSTA = Flink @ Yelp - Antonio Verardi, Yelp

Flink Forward

OSDC 2018 | Hardware-level data-center monitoring with Prometheus by Conrad H...

NETWAYS

What's hot (18)