How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience

•

1 like•321 views

Ensuring a consistently great Netflix experience while continuously pushing innovative technology updates is no easy feat. We'll look at how Netflix turns log streams into real-time metrics to provide visibility into how devices are performing in the field. Including sharing some of the lessons learned around optimizing Druid to handle our load.

Technology

How Netflix Uses Druid in
Real-time to Ensure a High
Quality Streaming
Experience
September 2020
Ben Sykes, Sr Software Engineer, Netflix
1

Druid @
Netflix
How Netflix Uses Druid in Real-time to
Ensure a High Quality Streaming
Experience
Druid Summit II - Sep 2020

Druid Summit II
Sep 2020
Talking Points
Background
Quality of Experience
Metrics Pipeline
Using Druid
Druid Cluster
Data Ingestion
Managing Performance
Trade-offs
Tools

Druid Summit II
Sep 2020
Glossary
Measure
A value relating to an event. E.g. The presence of an error, size of a
buffer, or a duration.
Metric
Meaning and value derived from one or more measures. Counts,
Rates or Percentiles of measured values.
Dimension, Tag
An attribute of a metric that can be used to group or summarize
metrics by populations with shared properties.
Cardinality
The count of unique values of a given dimension.
Segment
An index file used by Druid to contain data for a given time block. A
time block may be formed of multiple segments.

Druid Summit II
Sep 2020
Quality of Experience

Druid Summit II
Sep 2020
Dashboards &
Monitoring

Druid Summit II
Sep 2020
Automated Alerting

Druid Summit II
Sep 2020
Automated Canary Analysis

Druid Summit II
Sep 2020
Metrics Pipeline

Druid Summit II
Sep 2020
Measure Extraction

Druid Summit II
Sep 2020
Rollup
Dimensions Metrics

Druid Summit II
Sep 2020
Cardinality
Dimensions
Device = { iPhone, Chrome, TV, iPhone, iPhone, TV, iPhone } |Device| = 3
Country = { US, CA, US, CA, CA, US, US } |Country| = 2

Druid Summit II
Sep 2020
Realtime Indexing
Per Kafka Topic Many Partitions
Many Indexers per
MiddleManager,
Many Partitions per
Indexer
Many Segments per
Indexer

Druid Summit II
Sep 2020
Managing Performance

Druid Summit II
Sep 2020
Compaction
2
4 35 3 4

Druid Summit II
Sep 2020
Trade-offs
● Restrict dimensionality
● Limit cardinality (TopN, include list)
● Increase query granularity
● Reduce concurrent query limit
● Over-provision cluster
● Education and context on effects
● Monitor cardinality increases
● Sufficient query granularity
● Fail-fast on overload
● Monitor resources, plan scale
Better Query Performance More Flexibility

Druid Summit II
Sep 2020
Measure and Monitor
Cluster Performance

Druid Summit II
Sep 2020
Frequent Benchmarks

Druid Summit II
Sep 2020
Atlas Lumen
Built on OSS

November 2-4, 2020
San Francisco, CA
druidsummit.org
28
Register Now for
Druid Summit

What's hot

Kafka Streams State Stores Being Persistent

confluent

Need for Time series Database

Pramit Choudhary

Kafka streams windowing behind the curtain

confluent

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Facebook, Airbnb, Netflix, Uber, Twitter, Bloomberg, and FINRA, Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments in the last few years. Inspired by the increasingly complex SQL queries run by the Presto user community, engineers at Facebook and Starburst have recently focused on cost-based query optimization. In this talk we will present the initial design and implementation of the CBO, support for connector-provided statistics, estimating selectivity, and choosing efficient query plans. Then, our detailed experimental evaluation will illustrate the performance gains for several classes of queries achieved thanks to the optimizer. Finally, we will discuss our future work enhancing the initial CBO and present the general Presto roadmap for 2018 and beyond. Speakers Kamil Bajda-Pawlikowski, Starburst Data, CTO & Co-Founder Martin Traverso

Presto query optimizer: pursuit of performance

DataWorks Summit

We went through the journey of deploying Apache Druid clusters on Kubernetes(K8s) and created a druid-operator (https://github.com/druid-io/druid-operator). This talk introduces the druid kubernetes operator, how to use it to deploy druid clusters and how it works under the hood. We will share how we use this operator to deploy Druid clusters at Splunk. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Druid is a complex stateful distributed system and a Druid cluster consists of multiple web services such as Broker, Historical, Coordinator, Overlord, MiddleManager etc each deployed with multiple replicas. Deploying a single web service on K8s requires creating few K8s resources via YAML files and it multiplies due to multiple services inside of a Druid cluster. Now doing it for multiple Druid clusters (dev, staging, production environments) makes it even more tedious and error prone. K8s enables creation of application (such as Druid) specific extension, called “Operator”, that combines kubernetes and application specific knowledge into a reusable K8s extension that makes deploying complex applications simple.

Splunk: Druid on Kubernetes with Druid-operator

Imply

Redpanda and ClickHouse

Altinity Ltd

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Paris Data Engineers !

How to Lock Down Apache Kafka and Keep Your Streams Safe

confluent

Cosco is an efficient shuffle-as-a-service that powers Spark (and Hive) jobs at Facebook warehouse scale. It is implemented as a scalable, reliable and maintainable distributed system. Cosco is based on the idea of partial in-memory aggregation across a shared pool of distributed memory. This provides vastly improved efficiency in disk usage compared to Spark's built-in shuffle. Long term, we believe the Cosco architecture will be key to efficiently supporting jobs at ever larger scale. In this talk we'll take a deep dive into the Cosco architecture and describe how it's deployed at Facebook. We will then describe how it's integrated to run shuffle for Spark, and contrast it with Spark's built-in sort-based shuffle mechanism and SOS (presented at Spark+AI Summit 2018).

Cosco: An Efficient Facebook-Scale Shuffle Service

Databricks

Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Databricks

Cloud providers like AWS allow free data transfers within an Availability Zone (AZ), but bill users when data moves between AZs. When the data volume streamed through Kafka reaches big data scale, (e.g. numeric data points or user activity tracking), the costs incurred by cross-AZ traffic can add significantly to your monthly cloud spend. Since Kafka serves reads and writes only from leader partitions, for a topic with a replication factor of 3, a message sent through Kafka can cross AZs up to 4 times. Once when a producer produces a message onto broker in a different AZ, two times during Kafka replication, and once more during message consumption. With careful design, we can eliminate the first and last part of the cross AZ traffic. We can also use message compression strategies provided by Kafka to reduce costs during replication. In this talk, we will discuss the architectural choices that allow us to ensure a Kafka message is produced and consumed within a single AZ, as well as an algorithm that lets consumers intelligently subscribe to partitions with leaders in the same AZ. We will also cover use cases in which cross-AZ message streaming is unavoidable due to design limitations. Talk outline: 1) A review of Kafka replication, 2) Cross-AZ traffic implications, 3) Architectural choices for AZ-aware message streaming, 4) Algorithms for AZ-aware producers and consumers, 5) Results, 6) Limitations, 7) Takeaways.

Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uday Sagar Si...

confluent

Cloud Monitoring tool Grafana

Dhrubaji Mandal ♛

Performant Streaming in Production: Preventing Common Pitfalls when Productio...

Databricks

In this talk, we will present Streams Replication Manager, a new open source Kafka mirroring solution designed specifically to provide disaster recovery and high availability for Kafka. We will describe and demo various replication topologies and recovery strategies using SRM and associated tooling. Finally, we will provide an update on the ongoing work to make this engine available for the Apache Kafka community as MirrorMaker2 (KIP-382).

Disaster Recovery and High Availability with Kafka, SRM and MM2

Abdelkrim Hadjidj

Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.

Apache Kafka at LinkedIn

Discover Pinterest

Apache Flink, AWS Kinesis, Analytics

Araf Karsh Hamid

Ceph Object Storage Reference Architecture Performance and Sizing Guide

Karan Singh

Amazon Aurora: Under the Hood

Amazon Web Services

Presto Summit 2018 - 09 - Netflix Iceberg

kbajda

Grafana introduction

Rico Chen

What's hot (20)

Kafka Streams State Stores Being Persistent

Need for Time series Database

Kafka streams windowing behind the curtain

Presto query optimizer: pursuit of performance

Splunk: Druid on Kubernetes with Druid-operator

Redpanda and ClickHouse

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

How to Lock Down Apache Kafka and Keep Your Streams Safe

Cosco: An Efficient Facebook-Scale Shuffle Service

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka (Uday Sagar Si...

Cloud Monitoring tool Grafana

Performant Streaming in Production: Preventing Common Pitfalls when Productio...

Disaster Recovery and High Availability with Kafka, SRM and MM2

Apache Kafka at LinkedIn

Apache Flink, AWS Kinesis, Analytics

Ceph Object Storage Reference Architecture Performance and Sizing Guide

Amazon Aurora: Under the Hood

Presto Summit 2018 - 09 - Netflix Iceberg

Grafana introduction

Similar to How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience

In this session, TrafficGuard’s Head of Data Science, Raigon Jolly, will discuss how TrafficGuard uses Druid and its partnership with Imply to: - Provide granular reporting to clients in near-real time - Monitor rules and concept drift - Staying ahead of the moving target that is ad fraud - Facilitate performance tuning and right-sizing infrastructure so our team can focus on innovation of our core product

How TrafficGuard uses Druid to Fight Ad Fraud and Bots

Imply

My past-3 yeas-developer-journey-at-linkedin-by-iantsai

Kim Kao

Druid at Strata Conf NY 2016.pdf

HimanshuGupta936

Game Analytics at London Apache Druid Meetup

Jelena Zanko

Real Time analytics with Druid, Apache Spark and Kafka

Daria Litvinov

Pivoting Spring XD to Spring Cloud Data Flow: A microservice based architecture for stream processing Microservice based architectures are not just for distributed web applications! They are also a powerful approach for creating distributed stream processing applications. Spring Cloud Data Flow enables you to create and orchestrate standalone executable applications that communicate over messaging middleware such as Kafka and RabbitMQ that when run together, form a distributed stream processing application. This allows you to scale, version and operationalize stream processing applications following microservice based patterns and practices on a variety of runtime platforms such as Cloud Foundry, Apache YARN and others. About Sabby Anandan Sabby Anandan is a Product Manager at Pivotal. Sabby is focused on building products that eliminate the barriers between application development, cloud, and big data.

Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan

PivotalOpenSourceHub

Saroj_Profile

saroj maharana

This presentation "Complex Infrastructure Made Easy - Learn About the GoGrid 3.0 Release & Our New East Coast Datacenter" is primarily for current GoGrid customers. The Webinar was recorded on 6/28/10. Presenters were Mario Olivarez & Justin Kitagawa of GoGrid and Ryan Holland of TrendMicro. The webinar is available here: http://www.vimeo.com/13335979 Topics: - Overview: Complex Infrastructure Made Easy - GoGrid's unique value proposition - GoGrid 3.0: The latest version of the GoGrid platform - East Coast demo - GoGrid Security: New services from Cisco, Fortinet, TrendMicro & Sentrigo - PCI Compliance: PCI compliance in the GoGrid Cloud - Demo with TrendMicro - Questions & Answers Switch to NEW Flash Player

GoGrid 3.0 Webinar: Complex Infrastructure Made Easy - Learn About the GoGrid...

GoGrid Cloud Hosting

https://ssimeetup.org/did-report-1-first-official-w3c-did-working-group-meeting-japan-drummond-reed-webinar-36/ The DID Report 1 about the First Meeting of the New W3C DID Working Group with Drummond Reed, co-author of the W3C DID specification, and Markus Sabadello from Danube Tech. Headline news in SSI land: this month W3C members approved forming a full W3C Working Group for Decentralized Identifiers (DIDs). DID spec co-author Drummond Reed has been in Fukuoka Japan for the first official meeting of this new Working Group and he will share highlights of the meeting and the roadmap for taking DIDs to a full Web standard.

The DID Report 1: The First Official W3C DID Working Group Meeting (Japan)- D...

SSIMeetup

https://fwdays.com/en/event/php-fwdays-2020 All of us think about many questions when we start a project, when we already have a product and when we release it. Here are some of them: which architecture and infrastructure to choose? what should be the repository structure? how to make the right evolution from one application to 100 microservices with success product release? how to distribute cross-stack commands as a whole? what development practices to use? This story will expose approaches to solving these and many other problems in PHP projects through: None-Breaking change development approach, Cross-stack contacts, Trunk Based development, evolution from Polyrepo to Monorepo with components on different technologies, Boilerplates for components, different Architecture Views, Continuous Testing & Quality, Infrastructure View, Infrastructure as a code as the main tool. This topic will appeal to everyone - from Software Developer to Architect, as many Tips & Tricks will be revealed.

PHPFrameworkDay 2020 - Different software evolutions from Start till Release ...

Alexandr Savchenko

Ця розповідь розкриє підходи для вирішення багатьох проблем в PHP проєктах через: None-Breaking change development підхід, cross-stack контракти, Trunk Based development, еволюція з Polyrepo до Monorepo з компонентами на різних технологіях, Boilerplat’и компонентів, різні Architecture View, Continuous Testing & Quality, Infrastructure View, Infrastructure as a code як основний інструмент.

"Different software evolutions from Start till Release in PHP product" Oleksa...

Fwdays

Paypal teradata gimel_thrift_server

Anisha Nainani

Big Data - Hadoop and MapReduce - new age tools for aid to testing and QA BigData with its slew of technologies and terms has been the most talked about area in last couple of years. This has evolved in Big Data Science, Analytics and now on the IoT and automation side. There is a need for testers and QA team to not only get used to this new age digital transformation area but at the same time embrace the technology to their own advantage. We have experimented and successfully used Big Data Technologies – Hadoop and MapReduce for a recent testing engagement. The actual application was implemented using classic technologies like CentOS and C++. Testing team implemented Hadoop and MapReduce to help in quick turnaround for the testing.

Ataas2016 - Big data hadoop and map reduce - new age tools for aid to test...

Agile Testing Alliance

Deploy multi-environment application with Azure DevOps

Andrea Tosato

At ING we needed a way to implement Data science models from exploration into production. I will do this talk from my experience on the exploration and production Hadoop environment as a senior Ops engineer. For this we are using OpenShift to run Docker containers that connect to the big data Hadoop environment. During this talk I will explain why we need this and how this is done at ING. Also how to set up a docker container running a data science model using Hive, Python, and Spark. I’ll explain how to use Docker files to build Docker images, add all the needed components inside the Docker image, and how to run different versions of software in different containers. In the end I will also give a demo of how it runs and is automated using Git with webhook connecting to Jenkins and start the docker service that will connect to a big data Hadoop environment. This is going to be a great technical talk for engineers and data scientist. Speaker Lennard Cornelis, Ops Engineer, ING

Docker data science pipeline

DataWorks Summit

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Using Telegraf, InfluxDB and Mist Mist.io are the creators of Mist, the open source multi-cloud management platform. Their solution helps clients manage the complexities of using a mix of public and private clouds, hypervisors, containers and bare metals. Mist streamlines its customers’ operations and optimizes costs by improving DevOps practices. Discover how Mist achieves that by building on top of Telegraf and InfluxDB. Join this webinar as Chris Psaltis and Dimitris Moraitis dive into: Mist’s approach to improving visibility across environments How a time series database is used to monitor infrastructures Mist’s ability to automate processes and reduce infrastructure costs

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...

InfluxData

Presented at OSCON 2018. Hyperledger Indy is a distributed ledger built for decentralized identity and is one of the open source frameworks hosted by Hyperledger. It provides tools, libraries, and reusable components for creating and using independent digital identities rooted on blockchains or other distributed ledgers. In this presentation, I introduce The Linux Foundation and Hyperledger. We look at Decentralized Identity Concepts -- identity models, decentralized identity, zero-knowledge proofs, and verifiable credentials. We look at a demo that utilizes Hyperledger Indy and these concepts. We then look at Hyperledger Indy's software stack and roadmap and touch on how you can get involved.

OSCON 2018 Getting Started with Hyperledger Indy

Tracy Kuhrt

Deploying software and controlling infrastructure quickly and safely is a hard task. In this talk, Brice Fernandes, Customer Success Engineer at Weaveworks, discusses GitOps, an operational model for Kubernetes and beyond to speed up development, while retaining extremely strong security guarantees. Brice describes and shows several open source tools developed at Weaveworks to support this approach. You will have a good idea of how to use the GitOps principles to create software pipelines that are fast, safe, and reproducible, while creating clear and high quality audit trails. Check out the full presentation on YouTube: https://youtu.be/QdCwUUtcj4I

Delivering Quality at Speed with GitOps

Weaveworks

The Right Tools for IoT Developers – Dan Gross @ Eclipse IoT Day ThingMonk 2016

Benjamin Cabé

Critical Breakthroughs and Challenges in Big Data and Analytics

Data Driven Innovation

Similar to How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience (20)

How TrafficGuard uses Druid to Fight Ad Fraud and Bots

My past-3 yeas-developer-journey-at-linkedin-by-iantsai

Druid at Strata Conf NY 2016.pdf

Game Analytics at London Apache Druid Meetup

Real Time analytics with Druid, Apache Spark and Kafka

Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan

Saroj_Profile

GoGrid 3.0 Webinar: Complex Infrastructure Made Easy - Learn About the GoGrid...

The DID Report 1: The First Official W3C DID Working Group Meeting (Japan)- D...

PHPFrameworkDay 2020 - Different software evolutions from Start till Release ...

"Different software evolutions from Start till Release in PHP product" Oleksa...

Paypal teradata gimel_thrift_server

Ataas2016 - Big data hadoop and map reduce - new age tools for aid to test...

Deploy multi-environment application with Azure DevOps

Docker data science pipeline

How to Gain Visibility into Containers, VM’s and Multi-Cloud Environments Usi...

OSCON 2018 Getting Started with Hyperledger Indy

Delivering Quality at Speed with GitOps

The Right Tools for IoT Developers – Dan Gross @ Eclipse IoT Day ThingMonk 2016

Critical Breakthroughs and Challenges in Big Data and Analytics

Pivot 2.0 - The next generation visualization tool for your streaming data

Imply

Peter Marshall, Technology Evangelist at Imply Abstract: Apache Druid® can revolutionise business decision-making with a view of the freshest of fresh data in web, mobile, desktop, and data science notebooks. In this talk, we look at key activities to integrate into Apache Druid POCs, discussing common hurdles and signposting to important information. Bio: Peter Marshall (https://petermarshall.io) is an Apache Druid Technology Evangelist at Imply (http://imply.io/), a company founded by original developers of Apache Druid. He has 20 years architecture experience in CRM, EDRM, ERP, EIP, Digital Services, Security, BI, Analytics, and MDM. He is TOGAF certified and has a BA degree in Theology and Computer Studies from the University of Birmingham in the United Kingdom.

Druid Adoption Tips and Tricks

Imply

Nicolas Trésegnie, Chief Architect at SuperAwesome Abstract: SuperAwesome's mission is to make the internet safer for kids. At the core of SuperAwesome's analytics is Druid. In this talk, we walk through how we run Druid on spot instances. We explain the consequences in terms of cost and reliability, how we managed to build a reliable system despite the risks, and how you could do the same. Nicolas works as Chief Architect at SuperAwesome, where is is looking after the overall architecture of the systems and the infrastructure. He is all about automation and how technology can be used to achieve business goals. Nicolas studied Computer Science and Bioinformatics, and he is now pursuing an MBA at Imperial.

Druid in Spot Instances

Imply

Apache Druid® is an open source analytics database powering fresh, fast analytics in companies from AirBnB to Zeotap on clickstream, telemetry, financial transactions, applications and more. In this talk, we open the box on the three distributed processes in Druid led by the coordinator, overlord, and broker, and the ways that these come together to deliver reliable, performant query, ingestion, and management services.

Apache Druid®: A Dance of Distributed Processes

Imply

Druid has been the production workhorse for the past 2+ years at Zeotap powering the core Audience planning across our Connect and Targeting products. Though Druid is best suited for data having time as a dimension as it partitions data based on time first, we have used Druid to serve ML powered enhanced insights and Estimation of potential dataset sizes, to assist us with our core business case of Audience planning. These are datasets without timestamp a.k.a non-temporal with high scale and having nested dimensions. These have been achieved using nuanced data modelling to store the data sets and achieve millisecond latency retrieval on top of the same. The core of the presentation would be on the data modelling journey to achieve these use cases detailing the query access patterns. We also delve upon the architecture - ingestion into druid sink and processing including ML. In the end we go over the production setup and configurations and provide the performance tunings applied. The presentation would have the following heads: The presentation would have the following heads * Business case in Ad-Tech and Mar-Tech vertical * Audience Planner Usecase 1 - Insights -Lambda Architecture and data flow -Deep dive on data model -Takeaways *Audience Planner Usecase 2 - Estimator -Architecture and data flow -Stratified sampling explained -Data model to solve nested data - deep dive -Takeaways *Audience Planner Usecase 3 - Skew correction -Skew correction model -Query Access -Data model in Druid to accommodate output from ML models -Takeaways *Production setup, config and Tunings *Production Operation experience takeaways

Zeotap: Data Modeling in Druid for Non temporal and Nested Data

Imply

At Nielsen Identity, we leverage Druid to provide our customers with real-time analytics tools for various use-cases, including in-flight analytics, reporting and building target audiences. The common challenge of these use-cases is counting distinct elements in real-time at scale. We’ve been using Druid to solve these problems for the past 4 years, and gained a lot of experience with it. In this talk, we will share some of the best practices and tips we’ve gathered over the years, including: *Data modeling *Ingestion *Retention and deletion *Query optimization

Nielsen: Casting the Spell - Druid in Practice

Imply

One of the most popular use cases for Apache Druid is building data applications. Data applications exist to deliver data into the hands of everyone on a team in a business, and are used by these teams to make faster, better decisions. To fulfill this role, they need to support granular drill down, because the devil is in the details, but also be extremely fast, because otherwise people won't use them! In this talk, Gian Merlino will cover: *The unique technical challenges of powering data-driven applications *What attributes of Druid make it a good platform for data applications *Some real-world data applications powered by Druid

Building Data Applications with Apache Druid

Imply

Druid is a powerful real-time database, and part of that power is the level of control you get over cluster configuration, allowing you to get maximum performance for your specific data and query types. In this talk, Gian Merlino, one of the original authors of Druid and CTO and co-founder of Imply, will walk you through some advanced techniques that can provide a multiplier to your Druid performance. Afterwards, he’ll take your questions about performance, or anything else Druid-related.

Maximizing Apache Druid performance: Beyond the basics

Imply

Target is one of the largest retailers in the United States, with brick-and-mortar stores in all 50 states and one of the most-visited ecommerce sites in the country. In addition to typical merchandising functions like assortment planning, pricing and inventory management, Target also operates a large supply chain, financial/banking operations and property management organizations. As a data-driven organization, we need a data analytics platform that can address the unique needs of each of these various business units, while scaling to hundreds of thousands of users and accommodating an ever-increasing amount of data. In this talk we’ll cover why Target chose to create our own analytics platform and specifically how Druid makes this platform successful. We’ll cover how we utilize key features in Druid, such as union datasources, arbitrary granularities, real-time ingestion, complex aggregation expressions and lightning-fast query response to provide analytics to users at all levels of the organization. We’ll also cover how Druid’s speed and flexibility allow us to provide interactive analytics to front-line, edge-of-business consumers to address hundreds of unique use-cases across several business units.

Building an Enterprise-Scale Dashboarding/Analytics Platform Powered by the C...

Imply

Talk abstract: Users are demanding access to large, multi-petabyte, multi-dimension, real-time datasets to answer business critical questions. Providing a self-service interface that meets the performance expectations of these users can be challenging. Enter Apache Druid: an open source analytics database powering real-time, ad hoc, lightning fast analytics. It is used for clickstream analytics, network telemetry, fraud detection, application monitoring and so much more by companies like Apple, Netflix, Twitter, and AirBnb. Druid can ingest millions of records per second and deliver sub-second response times on OLAP-style slice and dice queries. In this talk, we will start with an overview of Apache Druid followed by a look at several examples of how Druid is being used in the real-world. We'll finish up with Q&A and some virtual networking. Speaker Bio: Mike McLaughlin is a senior field engineer at Imply. He helps customers run and optimize Apache Druid in production. He has 20 years experience developing, architecting, and deploying software.

Apache Druid: Lightning Fast Analytics on Real-time and Historical Data (Atla...

Imply

August meetup - All about Apache Druid

Imply

Benchmarking Apache Druid

Imply

Apache Druid ingests and enables instant query on many billions of events in real-time. But how? In this talk, each of the components of an Apache Druid cluster is described – along with the data and query optimisations at its core – that unlock fresh, fast data for all. Bio: Peter Marshall (https://linkedin.com/in/amillionbytes/) leads outreach and engineering across Europe for Imply (http://imply.io/), a company founded by the original developers of Apache Druid. He has 20 years architecture experience in CRM, EDRM, ERP, EIP, Digital Services, Security, BI, Analytics, and MDM. He is TOGAF certified and has a BA (hons) degree in Theology and Computer Studies from the University of Birmingham in the United Kingdom.

Druid: Under the Covers (Virtual Meetup)

Imply

Check out the full webinar: https://imply.io/videos/why-data-warehouses-cannot-support-hot-analytics Today’s data warehouses - whether traditional, specialized or cloud-based - are good at supporting cold analytics, such as reporting, where query times can take minutes. But they cannot cost-effectively support hot analytics—interactive ad hoc analytics usually performed by larger groups of users against batch or streaming data. Examples of hot analytics include clickstream analytics; service, network and application performance monitoring; and risk analytics. Data warehouses struggle with hot analytics use cases because they are too slow, unable to scale, or too expensive. Learn how a new class of real-time data platforms overcome these limitations, and how companies implement a “temperature-based” approach to analytics.

Why data warehouses cannot support hot analytics

Imply

Check out the webinar: https://imply.io/videos/whats-new-imply-3-3-apache-druid-0-18 The most recent Imply 3.3 release, based on Apache 0.18 brings several major new features, including joins, query laning and Clarity Alerts. These new features deliver increased design flexibility during design, and provide improved ingestion performance, and sub-second response times to help accelerate data warehouse and data lake deployments, and add real-time analytics in general.

What’s New in Imply 3.3 & Apache Druid 0.18

Imply

Apache Druid Vision and Roadmap

Imply

MoPub, a Twitter company, provides monetization solutions for mobile app publishers and developers around the globe. MoPub receives over 33 Billion ad requests per day generating over 200TB of raw logs every day. We built MoPub Analytics as the analytics platform, using Druid + Imply for our end users who are Publishers, Demand side partners and Internal users. We will talk about the architecture of the analytics platform, our Druid cluster setup, hardware choices, monitoring, use cases, limiting factors, challenges with lookups and solutions we used. Watch video:https://imply.io/virtual-druid-summit/analytics-over-terabytes-of-data-at-twitter-apache-druid

Analytics over Terabytes of Data at Twitter

Imply

More from Imply (17)

Pivot 2.0 - The next generation visualization tool for your streaming data

Druid Adoption Tips and Tricks

Druid in Spot Instances

Apache Druid®: A Dance of Distributed Processes

Zeotap: Data Modeling in Druid for Non temporal and Nested Data

Nielsen: Casting the Spell - Druid in Practice

Building Data Applications with Apache Druid

Maximizing Apache Druid performance: Beyond the basics

Building an Enterprise-Scale Dashboarding/Analytics Platform Powered by the C...

Apache Druid: Lightning Fast Analytics on Real-time and Historical Data (Atla...

August meetup - All about Apache Druid

Benchmarking Apache Druid

Druid: Under the Covers (Virtual Meetup)

Why data warehouses cannot support hot analytics

What’s New in Imply 3.3 & Apache Druid 0.18

Apache Druid Vision and Roadmap

Analytics over Terabytes of Data at Twitter

Recently uploaded

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

Explore 'The Codex of Business: Writing Software for Real-World Solutions,' a compelling SlideShare presentation that delves into digital transformation in healthcare. Discover through a detailed case study how Agile methodologies empower healthcare providers to develop, iterate, and refine digital solutions that address real-world challenges. Learn how strategic planning, user feedback, and continuous improvement drive success in deploying technologies that enhance patient care and operational efficiency. Ideal for healthcare professionals, IT specialists, and digital transformation advocates seeking actionable insights and practical examples of technology making a real difference.

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Malak Abu Hammad

CNv6 Instructor Chapter 6 Quality of Service

giselly40

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Recently uploaded (20)

What Are The Drone Anti-jamming Systems Technology?

Finology Group – Insurtech Innovation Award 2024

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

🐬 The future of MySQL is Postgres 🐘

GenCyber Cyber Security Day Presentation

Automating Google Workspace (GWS) & more with Apps Script

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Handwritten Text Recognition for manuscripts and early printed texts

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Powerful Google developer tools for immediate impact! (2023-24 C)

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

08448380779 Call Girls In Friends Colony Women Seeking Men

How to Troubleshoot Apps for the Modern Connected Worker

08448380779 Call Girls In Civil Lines Women Seeking Men

The Codex of Business Writing Software for Real-World Solutions 2.pptx

CNv6 Instructor Chapter 6 Quality of Service

Presentation on how to chat with PDF using ChatGPT code interpreter

IAC 2024 - IA Fast Track to Search Focused AI Solutions

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience

1. How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience September 2020 Ben Sykes, Sr Software Engineer, Netflix 1

2. Druid @ Netflix How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience Druid Summit II - Sep 2020

3. Druid Summit II Sep 2020 Talking Points Background Quality of Experience Metrics Pipeline Using Druid Druid Cluster Data Ingestion Managing Performance Trade-offs Tools

4. Druid Summit II Sep 2020 Glossary Measure A value relating to an event. E.g. The presence of an error, size of a buffer, or a duration. Metric Meaning and value derived from one or more measures. Counts, Rates or Percentiles of measured values. Dimension, Tag An attribute of a metric that can be used to group or summarize metrics by populations with shared properties. Cardinality The count of unique values of a given dimension. Segment An index file used by Druid to contain data for a given time block. A time block may be formed of multiple segments.

5. Druid Summit II Sep 2020 Quality of Experience

6. Druid Summit II Sep 2020 Dashboards & Monitoring

7. Druid Summit II Sep 2020 Automated Alerting

8. Druid Summit II Sep 2020 Automated Canary Analysis

9. Druid Summit II Sep 2020 Metrics Pipeline

10. Druid Summit II Sep 2020 Measure Extraction

11. Using Druid Druid Summit II Sep 2020

12. Druid Summit II Sep 2020 Druid Cluster

13. Druid Summit II Sep 2020 Rollup Dimensions Metrics

14. Druid Summit II Sep 2020 Cardinality Dimensions Device = { iPhone, Chrome, TV, iPhone, iPhone, TV, iPhone } |Device| = 3 Country = { US, CA, US, CA, CA, US, US } |Country| = 2

15. Druid Summit II Sep 2020 Segments

16. Druid Summit II Sep 2020 Realtime Indexing Per Kafka Topic Many Partitions Many Indexers per MiddleManager, Many Partitions per Indexer Many Segments per Indexer

17. Druid Summit II Sep 2020 Realtime Indexing Per Kafka Topic Many Partitions Many Indexers per MiddleManager, Many Partitions per Indexer Many Segments per Indexer

18. Druid Summit II Sep 2020 Realtime Indexing Per Kafka Topic Many Partitions Many Indexers per MiddleManager, Many Partitions per Indexer Many Segments per Indexer

19. Druid Summit II Sep 2020 Realtime Indexing Per Kafka Topic Many Partitions Many Indexers per MiddleManager, Many Partitions per Indexer Many Segments per Indexer

20. Druid Summit II Sep 2020 Realtime Indexing Per Kafka Topic Many Partitions Many Indexers per MiddleManager, Many Partitions per Indexer Many Segments per Indexer 2

21. Druid Summit II Sep 2020 Managing Performance

22. Druid Summit II Sep 2020 Compaction 2 4 35 3 4

23. Druid Summit II Sep 2020 Trade-offs ● Restrict dimensionality ● Limit cardinality (TopN, include list) ● Increase query granularity ● Reduce concurrent query limit ● Over-provision cluster ● Education and context on effects ● Monitor cardinality increases ● Sufficient query granularity ● Fail-fast on overload ● Monitor resources, plan scale Better Query Performance More Flexibility

24. Druid Summit II Sep 2020 Measure and Monitor Cluster Performance

25. Druid Summit II Sep 2020 Frequent Benchmarks

26. Druid Summit II Sep 2020 Atlas Lumen Built on OSS

27. Time for questions 27 Thank you! Apache Druid is an independent project of The Apache Software Foundation. More information can be found at https://druid.apache.org. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

28. November 2-4, 2020 San Francisco, CA druidsummit.org 28 Register Now for Druid Summit

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience

Similar to How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience (20)

More from Imply

More from Imply (17)

Recently uploaded

Recently uploaded (20)

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experience