Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highly Reliable Logging at Airbnb

•

1 like•1,468 views

This document discusses Airbnb's use of Kafka as the foundation for its highly reliable logging system. It describes the types of data Airbnb collects, including product events, database exports, service events, and derived data. Airbnb uses a simple logging pipeline where events are delivered reliably through Kafka in real time. Key components of its production logging pipeline include Jitney for standardized messaging, a central schema repository, client SDKs, producer and consumer agents, and a self-service portal. The pipeline provides continuous integration through schema authoring, deployment, implementation, processing and storage, and monitoring. It handles large volumes of data reliably with 150 brokers processing 1 million messages per second and 10 billion events collected daily with very low

Software

Every Message Counts
Youssef Francis / Jun He / Kafka Summit 2017
Kafka as a Foundation for
Highly Reliable Logging at
Airbnb

Overview
Data we collect at Airbnb
• Product Events: Click streams, metrics, experiment
reporting.
• Database Exports: Streaming and batch replicas of
production databases.
• Service Events: Services uses these to communicate with
each other. Events act as a log of mutations.
• Derived Data: Computed from a combination of streaming
and batch data.

Schemas are a contract Events are delivered
reliably
Events are available
in real time
Producers only emit
well-formed events.
Errors detected in
development.
Predictable routing and
storage. Automatic
recovery from transient
failures.
Monitoring enables
immediately actionable
alerts. Events can be
inspected as they flow
through the pipeline.
Easy to find schemas
in use.
Ownership and
metadata provide
useful context.
Schemas and data
are discoverable
Tenets of Reliable Logging

What is Jitney?
Standardized message bus at
Airbnb
Jitney is a unified solution for
propagating business context
with standardized events.

Jitney is an
ecosystem
• Central schema repository
• Client SDK
• Producer proxy
• Consumer agent
• Self-service portal

Jitney Schemas
Schemas
• Contract owned by the producer and shared by consumers.
• Rich metadata provided by the Jitney API.
Central Schema Repository
• Single source of truth.
• Schemas are easily discoverable.
• All schemas undergo code review and automated validation.
Why Thrift?
• Rich ecosystem; write once, use everywhere.
• Good performance in Java and Ruby.
• Efficient binary encoding.

Jitney client SDK
Jitney everywhere
• Standardized routing logic.
• Automatic topic creation.
• Cluster discovery.
• Seek and Replay API: fast bootstrap and recovery.
• Built-in metric reporting.
• Available on all platforms in use at Airbnb.

Jitney producer
proxy and
consumer agents
Powered by Kafka
Jitney producer agent
• Local and distributed proxy modes
• Open to the public internet
• Local rate limiting and event filtering
• Single and batch publishing modes
• Horizontally scalable
Jitney consumer agent
• Local Java agent to support non-Java consumers
• Delivery at least once guarantee

Self-service portal
• Real time message tailing
• Topic creation
• Offset management

We built our logging pipeline using Jitney

Logging Pipeline
Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Processing and
Storage
Monitoring and
Auditing
Continuous Integration Workflow

Schema
Authoring
Engineers and
Data Scientists
define their
expectations in a
thrift schema,
which undergoes
code review.
Logging Pipeline
Continuous Integration Workflow

Schema
Authoring
Schema
Deployment
Once approved,
schema changes
are published
automatically in
new client
packages and
deployed to
consumers.
Logging Pipeline
Continuous Integration Workflow

Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Engineers
implement logging
using client SDKs.
Schemas enforced
automatically.
Logging Pipeline
Continuous Integration Workflow

Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Events are
consumed by
services and
stream processing
apps, and stored
in the warehouse.
Processing and
Storage
Logging Pipeline
Continuous Integration Workflow

Schema
Authoring
Schema
Deployment
Implementation
and Publishing
Processing and
Storage
Monitoring and
Auditing
Continuous loss
reporting and
anomaly detection.
Logging Pipeline
Continuous Integration Workflow

Some numbers
• 150 brokers across 5 clusters. 1 million messages per
second at peak.
• 10 billion logging events collected per day. Highly seasonal.
• 95% of logging events are available for querying in the
warehouse within 3 minutes.
• SLA guarantees no more than 0.01% loss. Less than
0.000001% end-to-end loss in practice.

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

Simplifying Event Streaming: Tools for Location Transparency and Data Evoluti...

confluent

At Under Armour Connected Fitness, we’ve built an event streaming platform on top of Kafka and the Confluent stack that makes it easy for developers to produce and consume schema-based events without requiring direct knowledge of Kafka. We are constantly trying to improve the developer experience. The platform consists of multiple federated Kafka clusters, a schema registry, a topology service, an archiver and specialized client libraries and Web / CLI tools that assist developers with producer and consumer workflows. In this talk, we will take a deeper dive into the design and implementation of a Scala/Java implementation of our client library that allows developers to produce or consume events without worrying about the underlying infrastructure and their location while enjoying the benefits of data compatibility through schemas. We’ll also look at an HTTP based client proxy that exposes the same API but for languages without our native support. Finally, we’ll walk through Web and CLI tools we built to make working with the platform easier. The content of this talk will be primarily aimed at software developers looking for ideas on how to build Kafka client tools that allow producer/consumer interactions protected by schema-based event definitions while hiding details of the underlying infrastructure.

Real-Time Dynamic Data Export Using the Kafka Ecosystem

confluent

(Preston Thompson, Braze) Kafka Summit SF 2018 If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy. We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors. In this talk, I will discuss: -How we started our journey in designing this large-scale streaming architecture -Why streaming technologies were necessary to solve our technology and business issues -The lessons we learned along the way that can help you with your Kafka-based architecture

Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story

confluent

Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber

confluent

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

HostedbyConfluent

As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase. Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances. This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.

Kafka for Real-Time Event Processing in Serverless Environments

confluent

(Jeff Sharpe + Alex Srisuwan, Capital One) Kafka Summit SF 2018 Using Kafka as a platform messaging bus is common, but bridging communication between real-time and asynchronous components can become complicated, especially when dealing with serverless environments. This has become increasingly common in modern banking where events need to be processed at near-real-time speed. Serverless environments are well-suited to address these needs, and Kafka remains an excellent solution for providing the reliable, resilient communication layer between serverless components and dedicated stream processing services. In this talk, we will examine some of the strengths and weaknesses of using Kafka for real-time communication, some tips for efficient interactions with Kafka and AWS Lambda, and a number of useful patterns for maximizing the strengths of Kafka and serverless components.

In this presentation, I will talk about my firsthand experience dealing with the unique challenges of running Kafka at a massive scale. If you ever thought that running Kafka is difficult, this talk may change your mind and provide you with valuable insights into how to configure a Kafka cluster efficiently, how to manage Kafka for enterprise customers and how to measure, monitor and maintain the Quality of Kafka Service. Our production Kafka cluster runs over 1500+ VMs, and serves over 10 GBPS data spread across hundreds of topics for multiple teams across Microsoft. We built a self-serve Kafka management service to make the process manageable and scalable across many teams. In this talk, I will also share insights about running Kafka in Private vs multi-tenant mode, supporting failover and disaster recovery requirements, and how to make Kafka Compliant with regulatory certifications such as ISO, SOC, FEDRAMP, etc. Presented by Nitin Kumar, Microsoft Systems Track

Apache kafka-a distributed streaming platform

confluent

Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale

confluent

Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer

confluent

Do you think that analytics are run on stored data sets? Think again, the combination of Apache Kafka and Stream Processing enables analytics on real-time data streams. First, we give a brief overview of stream processing and how it differs from the request response model of analytics on stored data. Next, we cover the characteristics of Kafka which make it such a good fit for Stream processing and why they matter. Finally, we show a number of use cases which highlight how stream processing is being used to do real-time analytics at scale with very low latency.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

confluent

Today, many companies that have lots of data are still struggling to derive value from machine learning (ML) and data science investments. Why? Accessing the data may be difficult. Or maybe it’s poorly labeled. Or vital context is missing. Or there are questions around data integrity. Or standing up an ML service can be cumbersome and complex. At Nuuly, we offer an innovative clothing rental subscription model and are continually evolving our ML solutions to gain insight into the behaviors of our unique customer base as well as provide personalized services. In this session, I’ll share how we used event streaming with Apache Kafka® and Confluent Cloud to address many of the challenges that may be keeping your organization from maximizing the business value of machine learning and data science. First, you’ll see how we ensure that every customer interaction and its business context is collected. Next, I’ll explain how we can replay entire interaction histories using Kafka as a transport layer as well as a persistence layer and a business application processing layer. Order management, inventory management, logistics, subscription management – all of it integrates with Kafka as the common backbone. These data streams enable Nuuly to rapidly prototype and deploy dynamic ML models to support various domains, including pricing, recommendations, product similarity, and warehouse optimization. Join us and learn how Kafka can help improve machine learning and data science initiatives that may not be delivered to their full potential.

How Much Can You Connect? | Bhavesh Raheja, Disney + Hotstar

HostedbyConfluent

How many connects can you run in a single cluster? Disney + Hotstar runs over 10 different connect clusters with over 2000+ connectors. In this talk, we share our experience of running Kafka connect at scale. We will walk through our decisions of using one cluster vs many and how the improvements in the connect ecosystem like incremental rebalancing have allowed us to scale to thousands of connects. We will also discuss challenges with scaling up & down connect workers while keeping the ecosystem stable & present a wishlist of the missing features in this distributed task framework.

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

HostedbyConfluent

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. aFurthermore, it is financially critical to get up-to-date, accurate analytics over all records. Due to the changing nature of real time transactions, it is impossible to pre-compute the analytics as a fixed time series. We have overcome the challenge by creating a real time key-value store inside Pinot that can sustain half million QPS with all the financial transactions. We will talk about the details of our solution and the interesting technical challenges faced.

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...

confluent

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...

confluent

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

confluent

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...

HostedbyConfluent

Developing cloud native microservices introduced us to many new challenges. One of the most difficult is to build reliable microservices integrations and their data exchange patterns. In this session I will share my 10 years of experience with building microservices and application runtime platforms with some of the largest European organisations. I will introduce basic principles of developing Java Spring Boot with Apache Kafka. These patterns can be used for: microservices communication decoupling, implementing microservices state stores, avoiding dependencies on traditional database systems. This session is targeted for developers who are interested in learning new cloud native development practices and understanding how event streaming microservices improve their current work. Demo application code will be available to participants.

Easily Build a Smart Pulsar Stream Processor_Simon Crosby

StreamNative

For organizations with boundless data sources, it is important to analyze, learn, predict and even respond in real time – directly from streaming data. This is important when: •Data volumes are large, or moving raw data is expensive, •Data is generated by widely distributed assets (eg: mobile devices), •Data is of ephemeral value and analysis can’t wait, or •It is critical to always have the latest insight and extrapolation won’t do. Use cases include prediction of failures on assembly lines, prediction of traffic flows in cities, predicting demand placed in power grids, detection of hackers, and understanding connection quality in mobile networks. They are characterized by a need to know – now – and require real-time processing of streaming data. Our goal is to enable real-time stream processing for Apache Pulsar in which analysis, learning and prediction are done on-the-fly, with continuous insights streamed back to the broker.

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

HostedbyConfluent

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

How to use Standard SQL over Kafka: From the basics to advanced use cases | F...

HostedbyConfluent

Several different frameworks have been developed to draw data from Kafka and maintain standard SQL over continually changing data. This provides an easy way to query and transform data - now accessible by orders of magnitude more users. At the same time, using Standard SQL against changing data is a new pattern for many engineers and analysts. While the language hasn’t changed, we’re still in the early stages of understanding the power of SQL over Kafka - and in some interesting ways, this new pattern introduces some exciting new idioms. In this session, we’ll start with some basic use cases of how Standard SQL can be effectively used over events in Kafka- including how these SQL engines can help teams that are brand new to streaming data get started. From there, we’ll cover a series of more advanced functions and their implications, including: - WHERE clauses that contain time change the validity intervals of your data; you can programmatically introduce and retract records based on their payloads! - LATERAL joins turn streams of query arguments into query results; they will automatically share their query plans and resources! - GROUP BY aggregations can be applied to ever-growing data collections; reduce data that wouldn't even fit in a database in the first place. We'll review in-production examples where each of these cases make unmodified Standard SQL, run and maintain over data streams in Kafka, and provide the functionality of bespoke stream processors.

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

confluent

With 60+ products and over 24% of the US GDP flowing through it, system integration is a tough problem for Intuit. Seasonality, scale, and massive peaks in products like TurboTax, QuickBooks, and Mint.com add extra layers of difficulty when building shared data services around transaction and user graphs, clickstream processing, a/b testing, and personalization. To reduce complexity and latency, we’ve implemented Kafka as the backbone across these data services. This allows us to asynchronously trigger relevant processing, elegantly scaling up and down as needed around peaks, all without the need for point-to-point integrations. In this talk, we share what we’ve learned about Kafka at Intuit and describe our data services architecture. We found that Kafka is invaluable in achieving a scalable, clean architecture, allowing engineering teams to focus less on integration and more on product development.

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...

confluent

Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore: • Why cloud-native platforms and why run Kafka on Kubernetes? • What kind of workloads are best suited for this combination? • Tips to determine the path forward for legacy monoliths in your application portfolio • Running Kafka as a Streaming Platform on Container Orchestration

Design Microservice Architectures the Right Way

C4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2O7BN8T. Michael Bryzek highlights specific key decisions that very directly impact the quality and maintainability of a microservice architecture, covering infrastructure, continuous deployment, communication, event streaming, language choice and more, all to ensure that teams and systems remain productive and scale. Filmed at qconnewyork.com. Michael Bryzek is the CTO and co-founder of Flow Commerce, an enterprise SAAS platform that is the world’s most advanced solution for global ecommerce. Prior, he was the cofounder and CTO of Gilt Groupe, an innovative online shopping destination offering.

What's hot

Kafka Summit SF 2017 - Providing Reliability Guarantees in Kafka at One Trill...

confluent

Apache kafka-a distributed streaming platform

confluent

Kafka Summit NYC 2017 - Simplifying Omni-Channel Retail at Scale

confluent

Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer

confluent

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

confluent

How Much Can You Connect? | Bhavesh Raheja, Disney + Hotstar

HostedbyConfluent

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...

HostedbyConfluent

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...

confluent

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...

confluent

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

confluent

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...

HostedbyConfluent

Easily Build a Smart Pulsar Stream Processor_Simon Crosby

StreamNative

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

HostedbyConfluent

How to use Standard SQL over Kafka: From the basics to advanced use cases | F...

HostedbyConfluent

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

Kafka, Killer of Point-to-Point Integrations, Lucian Lita

confluent

Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...

HostedbyConfluent

What's hot (20)