Large scale stream processing with Apache Flink

•

1 like•220 views

In today’s world it’s no longer enough to build systems that process big volumes of information. We now need applications that can handle large continuous streams of data with very low latency so we can react to the ever-changing environment around us. To efficiently handle such problems we need to deploy a stream processing solution. During the talk we’ll explore one of the most popular frameworks for stream processing – Apache Flink. We’ll see what unique capabilities it provides and how they apply to some real world problems. And we’ll also explore how it works under the hood and how to get the scalable and fault-tolerant stream processing that Flink provides.

Software

Stream Processing?
User Interaction Logs

Stream Processing?
User Interaction Logs
Application Logs

Stream Processing?
User Interaction Logs
Application Logs
Sensor Data

Stream Processing?
User Interaction Logs
Application Logs
Sensor Data
Database Commit Logs

High-latency & accurate
vs.
Low-latency & approximation

https://www.oreilly.com/ideas/questioning-the-lambda-architecture

Use Apache Kafka
Durable, scalable, fault-tolerant

Metrics we want to track
Net payout
Daily items sold
Weekly items sold
Order acceptance rate
Order preparation speed
Item rating

Order Stream
Payment Stream
User Rating Stream

Order Stream
Payment Stream
User Rating Stream
Stream Processor
OLAP

Everything is a batch
vs.
Everything is a stream

Single JVM Cluster Cloud
Runtime
DataSet API DataStream API

Source
Source
Operator
Operator
Operator Sinc
OLAP

https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

Flink Program
Optimizer
Graph Builder
Client

Flink Program
Optimizer
Graph Builder
Client Job Manager
Task Manager Task Manager

Flink Program
Optimizer
Graph Builder
Client Job Manager
Task Manager Task Manager
Snapshot Store

Lightweight Asynchronous Snapshots for
Distributed Dataflows
Paris Carbone,
Gyula Fóra,
Stephan Ewen
Seif Haridi
Kostas Tzoumas

Barrier Msg Msg Barrier Msg Msg Barrier
Operator

Barrier Msg Msg BarrierMsg Msg
Operator
Msg
Snapshot Store

Flink Program
Optimizer
Graph Builder
Client Job Manager
Task Manager Task Manager
Snapshot Store
Job
Manager
Job
Manager
Zookeeper

Flink Program
Optimizer
Graph Builder
Client
Task Manager Task Manager
Snapshot Store
Job
Manager
Job
Manager
Zookeeper

Order Stream
User Rating Stream
Local Join
Local Join

Apache Flink
● Can join streams
● Fault tolerant
● Exactly Once Processing
● Combines stream and batch processing

SQL → what data to analyze
Flink → how to analyze it

Monitoring and automatic
failure recovery

Large scale stream processing with Apache Flink

What's hot

Kafka is the bedrock of Wix’s distributed Mega Microservices system. Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices. In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These Include: * How to increase dev velocity of event-driven style code. * How to optimize working with Kafka in polyglot setting * How to migrate from request-reply to event-driven * How to tackle multiple DCs environment.

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Natan Silnitsky

We wanted to embed a Kafka producer/consumer in C++ and decided to use ""librdkafka"", a robust C/C++ library that is open source, well-maintained, and widely used. The C++ interface of ""librdkafka"" is confined to C++ 98 for compatibility, which makes it less object-oriented and user-friendly. Raw pointers in the interface complicates object lifecycle management and limited encapsulation leading to a more complex API. That is why we built a C++ Kafka API using modern C++ as a wrapper on top of ""librdkafka"". This header-only library is quite similar to the Java API, object-oriented and user-friendly. It frees us from the burdens mentioned above so even a novice can pick it up and work out an efficient solution. The ""modern-cpp-kafka"" project on github has been thoroughly tested within our company. After using it to replace a legacy implementation, throughput for a key middleware system gained 26%!

A Modern C++ Kafka API | Kenneth Jia, Morgan Stanley

HostedbyConfluent

Ingesting Healthcare Data, Micah Whitacre

confluent

Robust Stream Processing with Apache Flink

Jamie Grier

Kat Grigg, Confluent, Senior Customer Success Architect + Jen Snipes, Confluent, Senior Customer Success Architect This presentation will cover tips and best practices for Apache Kafka. In this talk, we will be covering the basic internals of Kafka and how these components integrate together including brokers, topics, partitions, consumers and producers, replication, and Zookeeper. We will be talking about the major categories of operations you need to be setting up and monitoring including configuration, deployment, maintenance, monitoring and then debugging. https://www.meetup.com/KafkaBayArea/events/270915296/

Tips & Tricks for Apache Kafka®

confluent

We present Apache Kafka’s core design for stream processing, which relies on its persistent log architecture as the storage and inter-processor communication layers to achieve correctness guarantees. Kafka Streams, a scalable stream processing client library in Apache Kafka, defines the processing logic as read process-write cycles in which all processing state updates and result outputs are captured as log appends. Idempotent and transactional write protocols are utilized to guarantee exactly once semantics. Furthermore, revision-based speculative processing is employed to emit results as soon as possible while handling out-of-order data. We also demonstrate how Kafka Streams behaves in practice with large-scale deployments and performance insights exhibiting its flexible and low-overhead trade-offs.

Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...

Guozhang Wang

Kafka est un système de messagerie distribué, en mode publish-subscribe, persistant les données qu'il reçoit, conçu pour facilement monter en charge et supporter des débits de données très importants. Originellement développé chez LinkedIn, et maintenu au sein de la fondation Apache depuis 2012, son adoption n'a cessé de croitre pour en faire un quasi de-facto standard dans les pipelines de traitement de données. Venez découvrir cet outil durant ce Hand's on de 3h où vous installerez un mini cluster Kafka et explorerez ses différentes API. En bonus, vous aurez la possibilité d'analyser vos données en temps réel avec Spark Streaming.

Devoxx fr 2016 - Apache Kafka - Stream Data Platform

Publicis Sapient Engineering

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Paul Brebner

Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of continuously changing data in real time? The answer is stream processing, and one system that has become a core hub for streaming data is Apache Kafka. This presentation will give a brief introduction to Apache Kafka and describe its usage as a platform for streaming data. It will explain how Kafka serves as a foundation for both streaming data pipelines and applications that consume and process real-time data streams. It will introduce some of the newer components of Kafka that help make this possible, including Kafka Connect, a framework for capturing continuous data streams, and Kafka Streams, a lightweight stream processing library. This is talk 1 out of 6 from the Kafka Talk Series. http://www.confluent.io/apache-kafka-talk-series/introduction-to-stream-processing-with-apache-kafka

Introduction To Streaming Data and Stream Processing with Apache Kafka

confluent

In order for our systems to scale continuously and be resilient, they need to be constantly evolving. In this talk, I’m going to tell the store of how my team migrated a data-intensive microservice from Python to Go. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.

ISTA 2019 - Migrating data-intensive microservices from Python to Go

Nikolay Stoitsev

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

HostedbyConfluent

Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 200M messages per second. In this talk, we will share the learning and challenges in our journey moving to Apache Kafka.

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

HostedbyConfluent

What’s easier than building a data pipeline nowadays? You add a few Apache Kafka clusters and a way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse… wait, this looks like a lot of things, doesn’t it? And you probably want to make it highly scalable and available too. Join this session to learn best practices for building a data pipeline, drawn from my experience at Activision/Demonware. I’ll share the lessons learned about scaling pipelines, not only in terms of volume, but also in terms of supporting more games and more use cases. You’ll also hear about message schemas and envelopes, Apache Kafka organization, topics naming conventions, routing, reliable and scalable producers and the ingestion layer, as well as stream processing.

Building Scalable and Extendable Data Pipeline for Call of Duty Games (Yarosl...

confluent

Apache Incubator Samza: Stream Processing at LinkedIn

Chris Riccomini

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

confluent

ApacheCon 2021 Apache Deep Learning 302 Tuesday 18:00 UTC Apache Deep Learning 302 Timothy Spann This talk will discuss and show examples of using Apache Hadoop, Apache Kudu, Apache Flink, Apache Hive, Apache MXNet, Apache OpenNLP, Apache NiFi and Apache Spark for deep learning applications. This is the follow up to previous talks on Apache Deep Learning 101 and 201 and 301 at ApacheCon, Dataworks Summit, Strata and other events. As part of this talk, the presenter will walk through using Apache MXNet Pre-Built Models, integrating new open source Deep Learning libraries with Python and Java, as well as running real-time AI streams from edge devices to servers utilizing Apache NiFi and Apache NiFi - MiNiFi. This talk is geared towards Data Engineers interested in the basics of architecting Deep Learning pipelines with open source Apache tools in a Big Data environment. The presenter will also walk through source code examples available in github and run the code live on Apache NiFi and Apache Flink clusters. Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. * https://github.com/tspannhw/ApacheDeepLearning302/ * https://github.com/tspannhw/nifi-djl-processor * https://github.com/tspannhw/nifi-djlsentimentanalysis-processor * https://github.com/tspannhw/nifi-djlqa-processor * https://www.linkedin.com/pulse/2021-schedule-tim-spann/

ApacheCon 2021 Apache Deep Learning 302

Timothy Spann

Apache kafka-a distributed streaming platform

confluent

Building a modern Software as a Service platform brings a lot of interesting engineering challenges. During this talk, I’m going to share my team’s journey of building a SaaS from scratch in 2020. First, we are going to start with the technologies and the architecture we picked. Then, we’ll go over the interesting challenge of implementing multitenancy. And we'll see how we benchmarked three different options and picked one. And last but not least, we’ll explore how every startup can use open source technologies to build observability infrastructure. And how to run their SaaS in production.

Building a modern SaaS in 2020

Nikolay Stoitsev

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

confluent

Logging ingestion infrastructure at Pinterest is built around Apache Kafka to support thousands of pipelines with over 1 trillion (1PB) new messages generated by hundreds of services (written in 5 different languages) and transported to data lake (AWS S3) every day. In the past, we have focused on scalability and auto operation of the infrastructure to help internal teams quickly onboard new pipelines (Kafka Summit 2018, 2020). However, we had constantly observed data loss and data corruption due to the design decisions we made to favor scalability and availability over durability and consistency. To tackle these problems, we designed and implemented logging auditing framework which consists of (1) audit client library integrated into every component of the infrastructure to detect data corruption for every message and send out audit events for randomly picked messages, (2) Kafka clusters receiving audit events, and (3) realtime and batch application processing audit events to generate insights for alerting and reporting. Focusing on zero negative impact to existing ingestion pipelines, scalability and cost efficiency led us to make various design decisions to eventually achieve auditing rollout to every pipeline with zero downtime and fundamentally improve the data ingestion quality at Pinterest in general by tracking data loss and removing data corruption which in the past can block downstream applications for hours and often lead to severe incidents.

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

HostedbyConfluent

What's hot (20)

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

A Modern C++ Kafka API | Kenneth Jia, Morgan Stanley

Ingesting Healthcare Data, Micah Whitacre

Robust Stream Processing with Apache Flink

Tips & Tricks for Apache Kafka®

Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...

Devoxx fr 2016 - Apache Kafka - Stream Data Platform

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Introduction To Streaming Data and Stream Processing with Apache Kafka

ISTA 2019 - Migrating data-intensive microservices from Python to Go

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

Twitter’s Apache Kafka Adoption Journey | Ming Liu, Twitter

Building Scalable and Extendable Data Pipeline for Call of Duty Games (Yarosl...

Apache Incubator Samza: Stream Processing at LinkedIn

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

ApacheCon 2021 Apache Deep Learning 302

Apache kafka-a distributed streaming platform

Building a modern SaaS in 2020

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

Similar to Large scale stream processing with Apache Flink

Microservices became the new black in enterprise architectures. APIs provide functions to other applications or end users. Even if your architecture uses another pattern than microservices, like SOA (Service-Oriented Architecture) or Client-Server communication, APIs are used between the different applications and end users. Apache Kafka plays a key role in modern microservice architectures to build open, scalable, flexible and decoupled real time applications. API Management complements Kafka by providing a way to implement and govern the full life cycle of the APIs. This session explores how event streaming with Apache Kafka and API Management (including API Gateway and Service Mesh technologies) complement and compete with each other depending on the use case and point of view of the project team. The session concludes exploring the vision of event streaming APIs instead of RPC calls.

Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...

HostedbyConfluent

Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?

Kai Wähner

Building Scalable Data Pipelines - 2016 DataPalooza Seattle

Evan Chan

HDInsight for Architects

Ashish Thapliyal

Real time cloud native open source streaming of any data to apache solr Utilizing Apache Pulsar and Apache NiFi we can parse any document in real-time at scale. We receive a lot of documents via cloud storage, email, social channels and internal document stores. We want to make all the content and metadata to Apache Solr for categorization, full text search, optimization and combination with other datastores. We will not only stream documents, but all REST feeds, logs and IoT data. Once data is produced to Pulsar topics it can instantly be ingested to Solr through Pulsar Solr Sink. Utilizing a number of open source tools, we have created a real-time scalable any document parsing data flow. We use Apache Tika for Document Processing with real-time language detection, natural language processing with Apache OpenNLP, Sentiment Analysis with Stanford CoreNLP, Spacy and TextBlob. We will walk everyone through creating an open source flow of documents utilizing Apache NiFi as our integration engine. We can convert PDF, Excel and Word to HTML and/or text. We can also extract the text to apply sentiment analysis and NLP categorization to generate additional metadata about our documents. We also will extract and parse images that if they contain text we can extract with TensorFlow and Tesseract.

Real time cloud native open source streaming of any data to apache solr

Timothy Spann

Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...

confluent

MQ, ETL and ESB middleware are often used as integration backbone between legacy applications, modern microservices and cloud services. This introduces several challenges and complexities like point-to-point integration or non-scalable architectures. This session discusses how to build a completely event-driven streaming platform leveraging Apache Kafka’s open source messaging, integration and streaming components to leverage distributed processing, fault-tolerance, rolling upgrades and the ability to reprocess events. Learn the differences between a event-driven streaming platform leveraging Apache Kafka and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

confluent

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies. The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.

Apache Kafka as Event Streaming Platform for Microservice Architectures

Kai Wähner

The requirements of many modern data platforms develop along two directions: (1) Low latency, i.e. the shift from batch-oriented to event-driven processes, which facilitate much more timely and reactive insights; and (2) complex analytics, i.e. the ability to efficiently apply analytic functions or models to the incoming data streams. However, many companies don't start from scratch, and already have well-established data infrastructure and processes with various degrees of affinity and compatibility to these novel paradigms. Based on extensive experience of building data platforms with customers, we describe in this talk some key challenges and aspects of introducing streaming-based approaches in real-world productive environments. These include e.g. integrating existing batch-oriented data sources and APIs, checking consistency when using event sourcing to exchange data, and building realtime analytical visualizations. For all cases, architectural options are discussed, and the final solution is explained, including technologies like Apache Nifi, Airflow, Phoenix, Druid and the Confluent Platform. We close the talk by describing non-technical aspects like building up an event-driven mindset among analysts. Speaker: Dr. Dominik Benz, inovex Event: Confluent Meetup, 08.10.2018 Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/ Mehr Tech-Artikel: https://www.inovex.de/blog/

Down the event-driven road: Experiences of integrating streaming into analyti...

inovex GmbH

Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration. This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.

Kafka for Real-Time Replication between Edge and Hybrid Cloud

Kai Wähner

Flink in action

Artem Semenenko

Apache NiFi: latest developments for flow management at scale

Abdelkrim Hadjidj

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Big Data Analytics Platforms by KTH and RISE SICS

Big Data Value Association

Kafka for data scientists

Jenn Rawlins

Batch querying and reporting is no longer enough for many organizations. Reducing time to insight – the time it takes to turn data into actionable insights – is becoming increasingly important to remain competitive. That’s why organizations are quickly evolving their data applications to support a broader set of real-time analytic use cases. In this webinar, we will review some of the common use cases for real-time analytics such as click-stream analysis, event data processing, and real-time analytics. We will show proven architectures for collecting, storing, and processing real-time data using a combination of AWS managed services, including Amazon Kinesis Streams, Amazon Kinesis Firehose, Amazon EMR, and AWS Lambda, as well open source tools, such as Apache Spark. Then, we will discuss common approaches and best practices to incorporate real-time analytics into your existing batch applications. Learning Objectives: • Understand how to incorporate real-time analytics into existing applications • Best practices to combine batch with real-time data flows • Learn common architectures and use cases for real-time analytics

Evolving Your Big Data Use Cases from Batch to Real-Time - AWS May 2016 Webi...

Amazon Web Services

Server Monitoring (Scaling while bootstrapped)

Ajibola Aiyedogbon

Data Streaming with Apache Kafka & MongoDB

confluent

Data Streaming with Apache Kafka & MongoDB - EMEA

Andrew Morgan

Webinar: Data Streaming with Apache Kafka & MongoDB

MongoDB

Similar to Large scale stream processing with Apache Flink (20)

Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...

Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?

Building Scalable Data Pipelines - 2016 DataPalooza Seattle

HDInsight for Architects

Real time cloud native open source streaming of any data to apache solr

Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...

Apache Kafka as Event Streaming Platform for Microservice Architectures

Down the event-driven road: Experiences of integrating streaming into analyti...

Kafka for Real-Time Replication between Edge and Hybrid Cloud

Flink in action

Apache NiFi: latest developments for flow management at scale

Cloud lunch and learn real-time streaming in azure

Big Data Analytics Platforms by KTH and RISE SICS

Kafka for data scientists

Evolving Your Big Data Use Cases from Batch to Real-Time - AWS May 2016 Webi...

Server Monitoring (Scaling while bootstrapped)

Data Streaming with Apache Kafka & MongoDB

Data Streaming with Apache Kafka & MongoDB - EMEA

Webinar: Data Streaming with Apache Kafka & MongoDB

More from Nikolay Stoitsev

As engineers, we like to solve problems by building solutions from scratch. Even though on some occasions it’s better to buy and integrate existing software. But how come? Are engineers who don’t always deliver from scratch real engineers? The goal of this talk is to answer all important questions about making build vs buy decisions. We’ll see how to define a clear strategy for making such decisions. And we’ll explore how to select and integrate existing software efficiently. Even if your company doesn’t have the habit of doing it.

Building vs Buying Software

Nikolay Stoitsev

How and why to manage your manager

Nikolay Stoitsev

From programming to management

Nikolay Stoitsev

Good observability is essential for modern software. It gives us confidence that our systems are working properly. And it also allows us to debug issues efficiently. In this talk, we’ll explore everything you need to know to start applying good observability to your projects. And we’ll see the most common pitfalls you need to be aware of. We will start with the tools and basic concepts in monitoring. And we’ll go over the 3 most common mistakes people make with it. Then we’ll see how to have automatic alerts to detect issues. And, we’ll touch on the principles for setting up good alerts. As a final step, we’ll see how to build our logging system and how to apply it in the most efficient way to debug issues easily.

A practical introduction to observability

Nikolay Stoitsev

The database is usually the heart of a software system. And there are many database technologies that we can pick from. In this talk, we’ll explore where RDBMS and NoSQL fall short and how NewSQL fills the gap. We’ll see what types of NewSQL databases exist and how they work. And we’ll also go over different NewSQL solutions that we can pick for our projects. By the end of the talk, we’ll have a good understanding of when and how to apply a NewSQL database in our big scale applications.

Everything You Need to Know About NewSQL in 2020

Nikolay Stoitsev

Effective communication is one of the most important skills we need. It greatly improves our productivity. And multiplies the positive impact that we have on the products we build and the people we work with. In this talk, we are going to explore three lessons on better communication. First, we’ll start with key principles for building trust and good relationships with the people around us. Then, we’ll see why and how to manage expectations. And we’ll explore how requirements facilitation can make our work easier. We are also going to see how to apply code reviews to our communication and scale it to amplify our impact. And most importantly, we’ll go over some real-world examples of how to apply these lessons in our everyday work to become better engineers.

3 lessons on effective communication for engineers

Nikolay Stoitsev

Microservices are a well-established architecture applied by many organizations around the world to build scalable and fault-tolerant backend systems. But as these systems grow so does the number of services in them. And this brings many challenges when we want to introduce new functionality. For a simple feature, engineers may need to spend a lot of time designing the end to end flow, changing code in multiple services and setting up complex test scenarios. During this talk, we’ll explore how to evolve a microservice architecture to be easily extensible based on some lessons learned from running 5000 microservices in production. We’ll go over different architectural patterns and open source tools that we can use to make it easy for all engineers to understand, extend and be more and more productive in such big complex systems.

Evolving big microservice architectures

Nikolay Stoitsev

The career path of software engineers and how to navigate it

Nikolay Stoitsev

As Uber is hyper-growing as a company so does our need for scalable and resilient systems. In this talk, I’m going to tell the story of how my team migrated from Python to Go, a microservice that processes millions of events every day. First, we are going to start with the rationale behind the migration. Then we are going to go over the Python and Go tech stacks that we use. Last but not least, I’m also going to share our approach for migrating the service while running in production, adding new features and making sure there are no regressions.

Migrating a data intensive microservice from Python to Go

Nikolay Stoitsev

The database has always been one of the key components in every architecture. There is a great variety of tradeoffs we should consider and implementation that we can pick from. If we need consistency and correctness in exchange of availability and performance, we should pick a relational database. If we need scale and increased availability by sacrificing transactional and consistency guarantees, we should use a NoSQL database. And if we need both horizontal scalability and transactions, we need to pick a NewSQL database. During this talk we’ll explore what guarantees a NewSQL system provides. We’ll go over the different approaches in building such a system. And we’ll see some open source projects that implements each approach. At the end of the talk we’ll have a good understanding of when and how to apply a NewSQL database in our big scale applications.

NewSQL: what, when and how

Nikolay Stoitsev

How to read the v8 source code?

Nikolay Stoitsev

The applications that we build for today's world have a lot of requirements. They need to provide the best user experience and to be always up and running. To achieve this in a massive scale you need a multi data center architecture. When we have more than one data centers, even if one of them goes down, the other can handle the traffic and your users will continue to use your application uninterrupted. Also by having datacenters in different locations around the world you can you take advantage of lower latencies and provide a better usability. But to take advantage of all those benefits you need to architect your application in a special way. During the talk we’ll explore the different multi data center configurations and the tradeoff of each one of them. We’ll also go over the ways to do failover and some useful processes to facilitate it better. Moreover, we’ll see how each layer of the application is affected by such architecture, all the way down to the database and the data model. Finally, I’ll share what technologies help Uber to run in multiple data centers and the lessons we learned by doing so.

Running in multiple data centers

Nikolay Stoitsev

As our systems grow, so does our software architecture complexity. As we scale and add more and more components, the interactions occurring between them become very complex and we start to lose visibility into the system. Traditional monitoring tools such as metrics and distributed logging still have their place, but they often fail to provide visibility across services. This is where distributed tracing thrives. During the talk we’ll explore what distributed tracing is, what open tools we can use to facilitate it and some lessons learned while implementing distributed tracing at Uber and how it helps us build big and impactful systems.

Distributed tracing for big systems

Nikolay Stoitsev

The shipping containers were introduced around 1830s and since then they play a crucial role in the modern society by providing efficient packaging, storage and transportation. Today we see the same revolution happening in the DevOps world with the adoption of container technologies like Docker. They allow us to package, distribute and run software in a scalable and efficient way. In this talk we’ll see how we can abstract the common problem we solve everyday when building scalable Java APIs with Docker into design patterns to create reusable solutions. We’ll explore the good practices of writing Java applications with Docker. Then we’ll see how each design pattern can be applied in real scenarios that address different concerns that a large system creates. We’ll see some real life implementations of those patterns and how they help us solve problems in scalable systems. By the end of the talk we’ll have a very powerful abstraction to tackle the everyday problems we face in building big and impactful systems.

Reusable patterns for scalable APIs running on Docker @ Java2Days

Nikolay Stoitsev

Everyday tools and tricks for scaling Node.js

Nikolay Stoitsev

Node.js at Uber

Nikolay Stoitsev

Tracing python applications

Nikolay Stoitsev

Distributed tracing for Node.js

Nikolay Stoitsev

Design Patterns for Docker Applications

Nikolay Stoitsev

As software engineers we do trade-offs every day. We often need to pick between things like space vs time or budget vs scope. Or sometimes the amount of creative waste we can afford to have. And when we make the decision we need to be in full comprehension of both the upside and downside of a particular choice. In this talk we will discuss why our organization decided to move from Python to Java. We will go over each tradeoff we decided to do and the motivation behind it.

From Python to Java

Nikolay Stoitsev

More from Nikolay Stoitsev (20)

Building vs Buying Software

How and why to manage your manager

From programming to management

A practical introduction to observability

Everything You Need to Know About NewSQL in 2020

3 lessons on effective communication for engineers

Evolving big microservice architectures

The career path of software engineers and how to navigate it

Migrating a data intensive microservice from Python to Go

NewSQL: what, when and how

How to read the v8 source code?

Running in multiple data centers

Distributed tracing for big systems

Reusable patterns for scalable APIs running on Docker @ Java2Days

Everyday tools and tricks for scaling Node.js

Node.js at Uber

Tracing python applications

Distributed tracing for Node.js

Design Patterns for Docker Applications

From Python to Java

Recently uploaded

Crafting the Perfect Measurement Sheet with PLM Integration

Wave PLM

AI/ML Infra Meetup | Perspective on Deep Learning Framework

Alluxio, Inc.

Workforce Efficiency with Employee Time Tracking Software.pdf

DeskTrack

INGKA DIGITAL: Linked Metadata by Design

Neo4j

How to pick right visual testing tool.pdf

Testgrid.io

How to download files safely from the internet ? Source: <a href="https://www.downloadsafely.com/">Download Safely</a> -Welcome to Secure Downloading&Surfing. Learn how to safely download files and protect your data while browsing the web. Let's dive into the world of secure surfing! Understanding the Risks Discover the potential threats associated with downloading files from the internet and the impact of malicious software on your device.

how-to-download-files-safely-from-the-internet.pdf

Mehmet Akar

JustNaik Solution Deck (stage bus sector)

Max Lee

Agnieszka Andrzejewska - BIM School Course in Kraków

bim.edu.pl

AI Hackathon.pptx

ERRORhackerboy

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio, Inc.

A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1

KnowledgeSeed

This is a classic migration case study (the past, current and the future) at scale from a world-wide company transitioning from Confluent Platform and Confluent Cloud to self-managed Apache Kafka on Kubernetes using Strimzi. At Maersk, we have been architecting, designing and implementing our 3rd generation Event Streaming Platform. This platform is based on Kubernetes in Azure and using Strimzi to operate Apache Kafka at large scale, highly reliable, segregating data based on isolated use cases. Our 2nd generation was based on OnPrem Confluent Platform and Confluent Cloud and this presentation is the story of this migration and reasoning behind it. Furthermore, we would get into details on how we monitor (Grafana, Prometheus), alert (GoAlert and alert as code), operate and provide self-service solutions on top of Strimzi to enable business critical application in Maersk, implemented in GoLang using the GitOps deployment model with Flux and Kustomization among others. Finally, if time allows we will end with a demo of an open-source self service tool to monitor and explore the cluster with most wanted features such as topic message browsing and configuring and restarting connectors.

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf

steffenkarlsson2

KLARNA - Language Models and Knowledge Graphs: A Systems Approach

Neo4j

In his book, The Nature of the Physical World, Sir Arthur Eddington commented that “We have to appeal to the one outstanding law — the second law of thermodynamics — to put some sense into the world.” This sense-making goes beyond the physical world, too. Entropy is also essential in the fields of information and communication theory. During this lecture for the Princeton Plasma Physics Laboratory, lecturer Andrea Goulet discussed the application of entropy-related concepts in two communication systems: software and collaborative teams. She examined how concepts that help us understand systemic statistical disorder, such as ergodic systems, Lyapunov exponents, Kolmogorov-Sinai entropy, and Shannon-entropy can help us optimize for both software quality and innovation. She also provided several domain-specific models: Lehman’s Laws and Conway’s Law for software, as well as new models from her own research that relate to entropy and innovation. Entropy helps us understand the world and achieve great things. There is an underlying beauty in its principles that we can use to advance scientific discovery. When we understand the subtleties related to balancing surprise and structure, we increase our chances for effective collaboration and finding novel solutions to complex problems.

Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...

Andrea Goulet

OpenChain @ LF Japan Executive Briefing - May 2024

Shane Coughlan

GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates

Neo4j

What need to be mastered as AI-Powered Java Developers

EmilyJiang23

10 Essential Software Testing Tools You Need to Know About.pdf

kalichargn70th171

Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...

rajkumar669520

Scenarios are the central artifact of the Behaviour Driven Development (BDD) process. Although many teams use scenarios and tools like Cucumber or SpecFlow to automate them, in many cases their scenarios contain a lot of details, particularly test data, and therefore they become too complicated to support collaboration with the business. The "essential" principle of scenario writing (scenario formulation) states that only those details should be included in the scenario that are relevant for the outcome. This talk provides help for those who struggle implementing this principle or would be interested to learn how you can create brief and maintainable scenarios.

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)

Gáspár Nagy

Recently uploaded (20)

Crafting the Perfect Measurement Sheet with PLM Integration

AI/ML Infra Meetup | Perspective on Deep Learning Framework

Workforce Efficiency with Employee Time Tracking Software.pdf

INGKA DIGITAL: Linked Metadata by Design

How to pick right visual testing tool.pdf

how-to-download-files-safely-from-the-internet.pdf

JustNaik Solution Deck (stage bus sector)

Agnieszka Andrzejewska - BIM School Course in Kraków

AI Hackathon.pptx

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf

KLARNA - Language Models and Knowledge Graphs: A Systems Approach

Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...

OpenChain @ LF Japan Executive Briefing - May 2024

GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates

What need to be mastered as AI-Powered Java Developers

10 Essential Software Testing Tools You Need to Know About.pdf

Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)

Large scale stream processing with Apache Flink

1. Large scale stream processing with Apache Flink Nikolay Stoitsev Sr. Software Engineer at Uber Tech Sofia

2. Stream Processing?

3. Stream Processing? User Interaction Logs

4. Stream Processing? User Interaction Logs Application Logs

5. Stream Processing? User Interaction Logs Application Logs Sensor Data

6. Stream Processing? User Interaction Logs Application Logs Sensor Data Database Commit Logs

7. Infinite Dataset

8. Producer Stream

9. Producer Stream HDFS

10. Producer Stream HDFS Hive

11. Producer Stream HDFS Hive Big Latency

12. Producer Stream HDFS Real-time service

13. Apache Storm storm.apache.org

14. High-latency & accurate vs. Low-latency & approximation

15. Lambda architecture

16. https://www.oreilly.com/ideas/questioning-the-lambda-architecture

17. Kappa Architecture

18. Use Apache Kafka Durable, scalable, fault-tolerant

19. Producer Kafka Stream Processor

20.

21.

22. Metrics we want to track Net payout Daily items sold Weekly items sold Order acceptance rate Order preparation speed Item rating

23. Real time

24. Scalable

25. Granular

26. Highly available

27. Order Stream Payment Stream User Rating Stream

28. Order Stream Payment Stream User Rating Stream Stream Processor OLAP

29. samza.apache.org

30. Apache Flink flink.apache.org

31. Everything is a batch vs. Everything is a stream

32. Single JVM Cluster Cloud Runtime DataSet API DataStream API

33. Dataflow graph

34. Source Source Operator Operator Operator Sinc OLAP

35. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

36. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

37. https://ci.apache.org/projects/flink/flink-docs-release-1.6/concepts/programming-model.html

38. Flink Program Optimizer Graph Builder Client

39. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager

40. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store

41. Fault tolerant

42. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store

43. Lightweight Asynchronous Snapshots for Distributed Dataflows Paris Carbone, Gyula Fóra, Stephan Ewen Seif Haridi Kostas Tzoumas

44. Barrier Msg Msg Barrier Msg Msg Barrier Operator

45. Barrier Msg Msg BarrierMsg Msg Operator Msg Snapshot Store

46. Exactly Once Processing

47. Can handle very large state

48. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store

49. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store Job Manager Job Manager Zookeeper

50. Flink Program Optimizer Graph Builder Client Job Manager Task Manager Task Manager Snapshot Store Job Manager Job Manager Zookeeper

51. Flink Program Optimizer Graph Builder Client Task Manager Task Manager Snapshot Store Job Manager Job Manager Zookeeper

52. Joining Streams

53. Order Stream User Rating Stream

54. Order Stream User Rating Stream

55. Order Stream User Rating Stream Local Join Local Join

56. Order Stream User Rating Stream Local Join Local Join

57. Apache Flink ● Can join streams ● Fault tolerant ● Exactly Once Processing ● Combines stream and batch processing

58. … but it requires Java/Scala code

59. Scalable, efficient and robust

60. github.com/uber/AthenaX

61. SQL → what data to analyze Flink → how to analyze it

62.

63.

64.

65.

66.

67.

68.

69. Resource estimation and auto scaling

70. Monitoring and automatic failure recovery

71. eng.uber.com/athenax

72. Thanks! Nikolay Stoitsev @ Uber

Large scale stream processing with Apache Flink

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Large scale stream processing with Apache Flink

Similar to Large scale stream processing with Apache Flink (20)

More from Nikolay Stoitsev

More from Nikolay Stoitsev (20)

Recently uploaded

Recently uploaded (20)

Large scale stream processing with Apache Flink