Going from three nines to four nines using Kafka | Tejas Chopra, Netflix

•

0 likes•468 views

The document discusses using Apache Kafka to improve data upload availability from 99.9% to 99.99% when moving data between on-premise and cloud storage. It describes using Kafka to trigger uploads to the cloud from on-premise storage with 99.9% availability and using Kafka to split uploads between cloud and on-premise storage as well as rehydrating failed on-premise uploads from the cloud to achieve 99.99% availability. The presentation concludes that Kafka provides high throughput and persistence needed to design effective data rehydration strategies across cloud and on-premise storage for very high availability.

Technology

Moving from 99.9 to 99.99
availability using Kafka
Tejas Chopra (Netﬂix, Inc.)

Agenda
- Introduction
- Problems with Cloud Storage, and ways around it
- What is availability?
- Uploads with 99.9% availability
- Uploads with 99.99% availability
- Takeaways & Lessons

Introduction
- Senior Software Engineer, Netﬂix
- Keynote Speaker: Cloud, Distributed
Systems, Blockchain
- Senior Software Engineer, Box.
- Datrium, Samsung, Tensilica, Apple,
Inc.

Cloud Conundrums
- Cheap to put data into cloud
- Pay to store it, pay even more to read it
- Solution:
- What if we can store a copy of data on
premise?
- Saves on reads
- Hot data can be on-premise, archival on
cloud
- Security, latency,
- Save millions of dollars per year
- Box: petabytes, Netﬂix: exabytes

Availability
- What is it?
- For on-premise
- For Cloud
- Gartner: Avg cost of downtime:
$5600/min.
- 99.9% : $2.8M
- 99.99%: $291K

99.9% solution
- Upload to on-premise (availability =
99.9%)
- Use kafka events to trigger uploads to
cloud
- Reads served from on-premise if
present, else fetched from cloud

99.99% solution
- Split the incoming stream to both
cloud and on-premise
- Queue failed on-premise requests
using Kafka
- Use cloud to hydrate failed uploads
on-premise
- 99.99% availability

Takeaways and Lessons
- Millions of customers, and billions of ﬁles uploaded: Kafka scales without
downtime. Kafka throughput: thousands of messages per second
- Kafka persistence compared to Kinesis - very critical in designing rehydration
strategies
- Batch handling abilities of Kafka - very useful for non-critical data - thumbnails.
- Tracking of oﬀsets in a partition: left to the consumers.
- Kafka cluster management

Thank you
- Email: chopratejas@gmail.com
- LinkedIn: https://www.linkedin.com/in/chopratejas
- Twitter: @chopra_tejas

What's hot

Solving Hybrid Cloud Data Replication with Apache Cassandra

Aaron Ploetz

ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...

Altinity Ltd

Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber

confluent

(Kir Titievsky, Google) Kafka Summit SF 2018 In this talk we will share some stories and patterns from customers who have built streaming pipelines and event-driven systems using Confluent Cloud in combination with Google Cloud Platform-native analytics tools, such as BigQuery and Dataflow. We will discuss what Confluent Cloud enables for hybrid deployments and how and why to mix and match platform-native and platform-neutral tools.

Putting Kafka Together with the Best of Google Cloud Platform

confluent

Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective? In this talk we’ll give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons. To highlight the differences, we’ll dive into a project we’ve done, transitioning from reading Kafka in a stream to reading it in batch. By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we’ve achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution. We’ll explore the tradeoffs and give you the metrics and intuition you’ll need to make such decisions yourself. We’ll cover: Costs of processing in stream compared to batch Scaling up for bursts and reprocessing Making the tradeoff between wait times and costs Recovering from outages And much more…

Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...

HostedbyConfluent

"There's little talk about capacity planning Kafka clusters, it's very much learn as you go, every cluster is different. In this talk Kafka DevOps Engineer Jason Bell takes you through the things that will help you, from broker capacity, thinking about topics and how the other Confluent components can affect throughput and performance. With a number of production deployments under his watchful gaze for over six years Jason has plenty of experience, stories and useful information that will help you. By the end of the talk you'll have a good understanding of designing the cluster for various scenarios, where the points of latency are to watch and monitor. And also how to prevent teams breaking the cluster behind your back. This talk is designed for everyone, anyone who is just starting to those who are operating Kafka on a daily basis."

Capacity Planning Your Kafka Cluster | Jason Bell, Digitalis

HostedbyConfluent

Core banking systems are batch oriented: typically with heavy overnight batch cycles before business opens each morning. In this talk I will explain some of the common interface points between core-banking infrastructure and event streaming systems. Then I will focus on how to do stream processing using ksqlDB for core-banking shaped data: showing how to do common operation using various ksqlDB functions. The key features are avro-record keys and multi-key joins (ksqlDB 0.15), schema management and state store planning.

Use ksqlDB to migrate core-banking processing from batch to streaming | Mark ...

HostedbyConfluent

In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.

(BDT318) How Netflix Handles Up To 8 Million Events Per Second

Amazon Web Services

As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase. Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances. This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

HostedbyConfluent

Introduction to Data Engineer and Data Pipeline at Credit OK

Kriangkrai Chaonithi

Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka

confluent

At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

Contact: https://www.linkedin.com/in/brandonjobrien @hakczar Introducing streaming data concepts, Storm cluster architecture, Storm topology architecture, and demonstrate working example of a WordCount topology for SIGKDD Seattle chapter meetup. Presented by Brandon O'Brien Code example: https://github.com/OpenDataMining/brandonobrien Meetup: http://www.meetup.com/seattlesigkdd/events/222955114/

Introduction to Streaming Distributed Processing with Storm

Brandon O'Brien

Shift: Real World Migration from MongoDB to Cassandra

DataStax

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

Eventing and streaming open a world of compelling new possibilities to our software and platform designs. They can reduce time to decision and action while lowering total platform cost. But they are not a panacea. Understanding the edges and limits of these architectures can help you avoid painful missteps. This talk will focus on event-driven and streaming architectures and how Apache Kafka can help you implement these. It will also discuss key tradeoffs you will face along the way from partitioning schemes to the impact of availability vs. consistency (CAP Theorem). Finally, we’ll discuss some challenges of scale for patterns like Event Sourcing and how you can use other tools and even features of Kafka to work around them. This talk assumes a basic understanding of Kafka and distributed computing but will include brief refresher sections.

Hard Truths About Streaming and Eventing (Dan Rosanova, Microsoft) Kafka Summ...

confluent

(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018 When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project. The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge. After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform. Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.

Creating an Elastic Platform Using Kafka and Microservices in OpenShift

confluent

As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture? To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.

O'Reilly Media Webcast: Building Real-Time Data Pipelines

SingleStore

InfluxEnterprise Architectural Patterns by Dean Sheehan, Senior Director, Pre...

InfluxData

Jay Kreps, Open Source Visionary and Co Founder of Confluent and several open source projects will be visiting LA. I have asked him to come present at our group. He will present his vision and will answer questions regarding Kafka and other projects Bio:- Jay is the co-founder and CEO at Confluent a company built around realtime data streams and the open source messaging system Apache Kafka. He is the original author of several of open source projects including Apache Kafka, Apache Samza, Voldemort, and Azkaban.

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

Data Con LA

What's hot (20)