Task Parallel Library Data Flows

This document provides an overview of Apache Flink, an open-source stream processing framework. It discusses Flink's capabilities in supporting streaming, batch, and iterative processing natively through a streaming dataflow model. It also describes Flink's architecture including the client, job manager, task managers, and various execution setups like local, remote, YARN, and embedded. Finally, it compares Flink to other stream and batch processing systems in terms of their APIs, fault tolerance guarantees, and strengths.

Maximilian Michels – Google Cloud Dataflow on Top of Apache Flink

This document discusses Google Cloud Dataflow and how it can be executed using Apache Flink. It provides an overview of Dataflow and its API, which is similar to batch and streaming concepts in Flink. It then describes how a Dataflow program is translated to an Abstract Syntax Tree (AST) and how the AST is converted to a Flink execution graph by implementing translators for specific Dataflow transforms like ParDo and Combine. Finally, it mentions the FlinkPipelineRunner that is available on GitHub to execute Dataflow pipelines using Flink.

Apache Flink Deep Dive

Vasia Kalavri

Flink provides concise summaries of key points: 1) After submitting a Flink job, the client creates and submits the job graph to the JobManager, which then creates an execution graph and deploys tasks across TaskManagers for parallel execution. 2) The batch optimizer chooses optimal execution plans by evaluating physical execution strategies like join algorithms and data shipping approaches to minimize data shuffling and network usage. 3) Flink iterations are optimized by having the runtime directly handle caching, state maintenance, and pushing work out of loops to avoid scheduling overhead between iterations. Delta iterations further improve efficiency by only updating changed elements in each iteration.

Apache Flink internals

Kostas Tzoumas

This document provides an overview of Apache Flink internals. It begins with an introduction and recap of Flink programming concepts. It then discusses how Flink programs are compiled into execution plans and executed in a pipelined fashion, as opposed to being executed eagerly like regular code. The document outlines Flink's architecture including the optimizer, runtime environment, and data storage integrations. It also covers iterative processing and how Flink handles iterations both by unrolling loops and with native iterative datasets.

Flink Forward SF 2017: Dean Wampler - Streaming Deep Learning Scenarios with...

As a low-latency streaming tool, Flink offers the possibility of using machine learning, even "deep learning" (neural networks), with low latency. The growing FlinkML library provides some of the infrastructure support required for this goal, combined with third-party tools. This talk is a progress report on several scenarios we are developing at Lightbend, which combine Flink, Deeplearning4J, Spark, and Kafka to analyze cluster telemetry for anomaly detection, predictive autoscaling, and other scenarios. I'll focus on the pragmatics of training deep learning models in a streaming context, using batch and mini-batch training, combined with low-latency application of those models. I'll discuss the architecture we're using and highlight trade offs of particular tools for certain design problems in the implementation. I'll discuss the drawbacks and workarounds of our design and finish with a look at how future developments in Flink could improve its support for scenarios like ours.

The document provides an overview of Apache Flink, an open source stream processing framework. It discusses Flink's programming model using DataSets and transformations, real-time stream processing capabilities, windowing functions, iterative processing, and visualization tools. It also provides details on Flink's runtime architecture, including its use of pipelined and staged execution, optimizations for iterative algorithms, and how the Flink optimizer selects execution plans.

FlinkML: Large Scale Machine Learning with Apache Flink

Theodoros Vasiloudis

Scalable Algorithm Design with MapReduce

Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow

This session will introduce a new open-source project - Flink TensorFlow - that enables Flink programs to operate on data using TensorFlow machine learning models. Applications include real-time image processing, NLP, and anomaly detection. The session will: - Introduce TensorFlow and describe its component model which allows for model reuse across environments - Demonstrate how to use TensorFlow models in Flink ML and Flink Streaming environments - Present a roadmap and provide opportunities to contribute

Productionizing Machine Learning - Bigdata meetup 5-06-2019

Iulian Pintoiu

Cluster Schedulers

Braxton McKee, Founder & CEO, Ufora at MLconf SF - 11/13/15

MLconf

Is Machine Learning Code for 100 Rows or a Billion the Same?: We have built an automatically distributed, implicitly parallel data science platform for running large scale machine learning applications. By abstracting away the computer science required to scale machine learning models, The Ufora platform lets data scientists focus on building data science models in simple scripting code, without having to worry about building large-scale distributed systems, their race conditions, fault-tolerance, etc. This automatic approach requires solving some interesting challenges, like optimal data layout for different ML models. For example, when a data scientist says “do a linear regression on this 100GB dataset”, Ufora needs to figure out how to automatically distribute and lay out that data across a cluster of machines in the cluster in order to minimize travel over the wire. Running a GBM against the same dataset might require a completely different layout of that data. This talk will cover how the platform works, in terms of data and thread distribution, how it generates parallel processes out of single-threaded programs, and more.

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Till Rohrmann

Parallel Programming: Beyond the Critical Section

Tony Albrecht

Flink Forward Berlin 2017: Boris Lublinsky, Stavros Kontopoulos - Introducing...

The Machine Learning (ML) library for Flink is a new effort to bring scalable ML tools to the Flink community in order to make Flink a premier platform for ML. This effort covers a wide area of ML related activities including data analysis, model training and model serving. In this talk we will cover approaches to using Flink for model serving. We will present a framework and actual code for serving and updating models in real time. Although presented examples cover only two model types - Tensorflow and PMML, the framework is applicable for any type of externalizable models. We will also show how this implementation fits into overall ML learning implementation and demonstrate integration with SparkML and Tensorflow/Keras and specifically show how models can be exported from these systems. This initial implementation can be further enhanced to support additional model representations, for example PFA and introduction of model serving specific semantics into Flink framework.

Parallel Programming Primer

Sri Prasanna

The document provides an introduction to parallel computing and parallel programming. It discusses Moore's Law and the need for parallelism to continue increasing performance. It outlines some common parallel architectures like SIMD and MIMD. It also describes different parallel programming models including message passing and shared memory, and different parallel algorithm patterns such as data parallel, task graph, and master-slave models. Finally, it briefly introduces MapReduce as a parallel programming paradigm.

A Domain-Specific Embedded Language for Programming Parallel Architectures.

This document proposes a domain-specific embedded language (DSEL) for programming parallel architectures. The DSEL aims to enable parallelism while avoiding issues like deadlocks, race conditions, and complex APIs. It presents the grammar and properties of the proposed DSEL, including that it generates schedules that are deadlock-free and race-condition free. Examples demonstrating data flow and data parallelism using the DSEL are also provided.

SICS: Apache Flink Streaming

Turi, Inc.

This document provides an introduction to stream processing with Apache Flink. It discusses why streaming is important, the key parts of a streaming infrastructure, and gives an example of how Bouygues Telecom uses Flink for stream processing. It then provides an overview of what Apache Flink is, including its unified batch and stream processing capabilities. The rest of the document focuses on stream processing features in Flink, including its DataStream API, flexible windowing options, support for iterative processing, fault tolerance mechanisms, and exactly-once processing semantics.

Suneel Marthi - Deep Learning with Apache Flink and DL4J

http://flink-forward.org/kb_sessions/deep-learning-with-apache-flink-and-dl4j/ Deep Learning has become very popular over the last few years in areas such as Image Recognition, Fraud Detection, Machine Translation etc. Deep Learning has proved to be very useful in handling unstructured data and extracting value from them. A big challenge with having to build deep learning models was the high cost of training them. With the recent advent of distributed frameworks like Apache Flink, Apache Spark etc.. it’s faster to train Deep Learning models in parallel on modern platform architecture. In this talk, we’ll be showing how to use Apache Flink Streaming with the open source Deep Learning framework, DeepLearning4j to perform large scale deep learning model training. We will show a demo of a Recurrent Neural Net that is trained for language modeling and have it generate text.

Netflix machine learning

Amer Ather

ML gives machines the ability to learn from data without being explicitly programmed. At Netflix, machine learning is used across many areas including recommendation systems, streaming quality, resource management, regional failover, anomaly detection, and capacity forecasting. Netflix uses various ML algorithms like decision trees, neural networks, and regression models to optimize the customer experience and infrastructure operations.

C++ Data-flow Parallelism sounds great! But how practical is it? Let’s see ho...

The document summarizes a presentation on using data-parallelism in C++ to parallelize the SPEC2006 benchmark suite. It discusses the design of a data-flow library called Parallel Pixie Dust (PPD) and how it was used to parallelize some STL algorithms. Analysis of two SPEC2006 benchmarks found that only a small number of STL algorithm usages could be easily parallelized due to the functional decomposition of the code into small blocks and potential side effects. Larger functions and avoidance of side effects may enable more opportunities for parallelization.

Introduction to OpenMP (Performance)

Akhila Prabhakaran

Message-passing concurrency in Python

Sarah Mount

Concurrency and parallelism in Python are always hot topics. This talk will look the variety of forms of concurrency and parallelism. In particular this talk will give an overview of various forms of message-passing concurrency which have become popular in languages like Scala and Go. A Python library called python-csp which implements similar ideas in a Pythonic way will be introduced and we will look at how this style of programming can be used to avoid deadlocks, race hazards and "callback hell".

Spring 5 Webflux - Advances in Java 2018

Trayan Iliev

The document discusses a presentation on functional reactive services with Spring 5 WebFlux. It introduces functional reactive programming (FRP), Project Reactor, building REST services with Spring 5 WebFlux including routers, handlers, filters, and reactive repositories. It also covers end-to-end non-blocking reactive service-oriented architecture with Netty, reactive WebClients, and real-time event streaming to JavaScript clients using server-sent events (SSE). The presentation code examples are available on GitHub.

Synapse 2018 Guarding against failure in a hundred step pipeline

Calvin French-Owen

Control store (ctlstore) is a new infrastructure component that solves the "n+1 problem" of independent database failures bringing down a distributed system. It replicates control data from a system of record to local SQLite databases on each service node. Ctlstore loads data transactionally from the system of record using a loader, executive, and control data log. It then replicates changes to local databases using a reflector. Snapshots to object storage allow new instances to start up with the latest data. This approach provides high availability and fast querying of shared control data across many services.

Zero-Copy Event-Driven Servers with Netty

Daniel Bimschas

Netty is a Java framework that provides tools for developing high performance and event-driven network applications. It uses non-blocking I/O and zero-copy techniques to minimize overhead and maximize throughput and scalability. Netty provides buffers, codecs, pipelines and handlers that allow building applications as a stack of processing layers. Example applications include a discard server and an HTTP file server that demonstrate Netty's core features and event-driven architecture.

What's hot

Ufuc Celebi – Stream & Batch Processing in one System

Apache Flink: API, runtime, and project roadmap

Kostas Tzoumas

FlinkML: Large Scale Machine Learning with Apache Flink

Theodoros Vasiloudis

Scalable Algorithm Design with MapReduce

Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow

Productionizing Machine Learning - Bigdata meetup 5-06-2019

Iulian Pintoiu

Cluster Schedulers

Braxton McKee, Founder & CEO, Ufora at MLconf SF - 11/13/15

MLconf

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Till Rohrmann

Parallel Programming: Beyond the Critical Section

Tony Albrecht

Flink Forward Berlin 2017: Boris Lublinsky, Stavros Kontopoulos - Introducing...

Parallel Programming Primer

Sri Prasanna

A Domain-Specific Embedded Language for Programming Parallel Architectures.

SICS: Apache Flink Streaming

Turi, Inc.

Suneel Marthi - Deep Learning with Apache Flink and DL4J

Netflix machine learning

Amer Ather

C++ Data-flow Parallelism sounds great! But how practical is it? Let’s see ho...