NoSQL for Netflix

•

9 likes•2,028 views

Cloudcamp SF slides summarizing the blog post that talks about a NoSQL use case with answers from several NoSQL suppliers.

Technology

What
Ne(lix
Needs
from
NoSQL

acockcro7@ne(lix.com

@adrianco
#ne(lixcloud

Basic
Requirements

•  Supports
running
on
Amazon
EC2

•  Supports
Amazon
Availability
Zones

•  Low
latency,
low
latency
variance

•  High
and
scalable
read
and
write
throughput

•  Large
and
scalable
capacity,
no
external
sharding

•  “AP”
Eventually
Consistent

•  Data
integrity
checks
and
repairs

•  Online
Snapshot
Backup,
Restore/Rollback

Scenario
–
Immediate
Read
a7er
Write

Q1:
Is
rouXng
and
replicaXon
zone
aware?

TV
Device

New
New

Favorite
Round
Robin
Favorites

Load
Balancer

List

API
API

(zone
A)
(Zone
B)

Append
New

New
Favorites

Favorite
List

Favorites
Favorites

(zone
A)
(Zone
B)

ReplicaXon

Network
ParXXon

Q2:
What
happens
next?

TV
Device

New
New

Favorite
Round
Robin
Favorites

Load
Balancer

List

API
API

(zone
A)
(Zone
B)

Append
New

New
Favorites

Favorite
List

Favorites
Favorites

(zone
A)
(Zone
B)

No
ReplicaXon

Network
ParXXon

Q4:
Append
vs.
Read/Modify/Write?

TV
Device

New
New

Favorite
Round
Robin
Favorites

Load
Balancer

List

RMW

API
API

(zone
A)
(Zone
B)

Old
New
New

Favorites
Favorites
Favorites

List
List
List

Favorites
Favorites

(zone
A)
(Zone
B)

ReplicaXon

Silent
Data
CorrupXon

Q4:
How
is
it
detected
and
corrected?

TV
Device

New
New

Favorite
Round
Robin
Favorites

Load
Balancer

List

API
API

(zone
A)
(Zone
B)

Append
New

New
Favorites

Favorite
List

Favorites
Favorites

(zone
A)
(Zone
B)

ReplicaXon
corrupted
on
disk
or
via
network

Backups

Q5:
Online
snapshot
and
restore?

•  Guard
against
applicaXon
bugs
&
corrupt
data

•  Point
in
Xme
(eventually)
consistent
backup

•  No
downXme
during
backup

•  No
downXme
during
restore

•  What
is
the
on-‐disk
format
for
archiving?

•  Business
ConXnuity
(non
app
speciﬁc
formats)

Answers
at

hfp://perfcap.blogspot.com/search?q=nosql

for
Cassandra
and
MongoDB
so
far

Adrian
Cockcro7

@adrianco
#ne(lixcloud

acockcro7@ne(lix.com

At a millions of operation/sec entropy is inevitable. How does Netflix deal with this data inconsistency? and What it takes Netflix to repair their fleet of C* systems for keeping Entropy to it’s minimum? In this talk, Vinay Chella, Cloud Data Architect @ Netflix will shed light on Netflix's AntiEntropy strategy and problems that arise at scale. He will discuss repair strategies and how Netflix repairs petabytes of data while maintaining a strict SLA for client applications and giving seamless movie experience to its customers . Come learn about different repair strategies, tools and techniques to repair your C* clusters.

Honest performance testing with NDBench

Vinay Kumar Chella

Apache Cassandra makes it possible to execute millions of operations per second in scalable fashion. Harnessing the power of C* leaves many developers pondering about the following: - Is my data model appropriate and not going to end up as wide partition(s) causing heap pressure and other issues? - How do I tune my connection pool configuration? What are the optimal settings for my environment ? - What is my C* cluster capacity in terms of number of IOPs for a given 95th and 99th latency? - How do I perf-test my data access layer? In this talk, Vinay Chella, Cloud Data Architect @ Netflix, will share open source tools, techniques and platform(NDBench) that Netflix uses to perf-test their C* fleet with simulations millions of operations per second.

Performance architecture for cloud connect

Adrian Cockcroft

Cassandra Operations at Netflix

greggulrich

Cassandra Performance and Scalability on AWS

Adrian Cockcroft

Cassandra EU 2012 - Netflix's Cassandra Architecture and Open Source Efforts

Acunu

Yow Conference Dec 2013 Netflix Workshop Slides with Notes

Adrian Cockcroft

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Recently uploaded

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Assure Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

By Design, not by Accident - Agile Venture Bolzano 2024

Pierluigi Pugliese

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Free Complete Python - A step towards Data Science

RinaMondal9

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Recently uploaded (20)

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Elevating Tactical DDD Patterns Through Object Calisthenics

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Assure Contact Center Experiences for Your Customers With ThousandEyes

By Design, not by Accident - Agile Venture Bolzano 2024

A tale of scale & speed: How the US Navy is enabling software delivery from l...

The Art of the Pitch: WordPress Relationships and Sales

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

GraphRAG is All You need? LLM & Knowledge Graph

Epistemic Interaction - tuning interfaces to provide information for AI support

Free Complete Python - A step towards Data Science

Introduction to CHERI technology - Cybersecurity

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Accelerate your Kubernetes clusters with Varnish Caching

Essentials of Automations: Optimizing FME Workflows with Parameters

NoSQL for Netflix

1. What Ne(lix Needs from NoSQL acockcro7@ne(lix.com @adrianco #ne(lixcloud

2. Basic Requirements •  Supports running on Amazon EC2 •  Supports Amazon Availability Zones •  Low latency, low latency variance •  High and scalable read and write throughput •  Large and scalable capacity, no external sharding •  “AP” Eventually Consistent •  Data integrity checks and repairs •  Online Snapshot Backup, Restore/Rollback

3. Scenario – Immediate Read a7er Write Q1: Is rouXng and replicaXon zone aware? TV Device New New Favorite Round Robin Favorites Load Balancer List API API (zone A) (Zone B) Append New New Favorites Favorite List Favorites Favorites (zone A) (Zone B) ReplicaXon

4. Network ParXXon Q2: What happens next? TV Device New New Favorite Round Robin Favorites Load Balancer List API API (zone A) (Zone B) Append New New Favorites Favorite List Favorites Favorites (zone A) (Zone B) No ReplicaXon

5. Network ParXXon Q4: Append vs. Read/Modify/Write? TV Device New New Favorite Round Robin Favorites Load Balancer List RMW API API (zone A) (Zone B) Old New New Favorites Favorites Favorites List List List Favorites Favorites (zone A) (Zone B) ReplicaXon

6. Silent Data CorrupXon Q4: How is it detected and corrected? TV Device New New Favorite Round Robin Favorites Load Balancer List API API (zone A) (Zone B) Append New New Favorites Favorite List Favorites Favorites (zone A) (Zone B) ReplicaXon corrupted on disk or via network

7. Backups Q5: Online snapshot and restore? •  Guard against applicaXon bugs & corrupt data •  Point in Xme (eventually) consistent backup •  No downXme during backup •  No downXme during restore •  What is the on-‐disk format for archiving? •  Business ConXnuity (non app speciﬁc formats)

8. Answers at hfp://perfcap.blogspot.com/search?q=nosql for Cassandra and MongoDB so far Adrian Cockcro7 @adrianco #ne(lixcloud acockcro7@ne(lix.com

NoSQL for Netflix

Recommended

Recommended

More Related Content

More from Adrian Cockcroft

More from Adrian Cockcroft (20)

Recently uploaded

Recently uploaded (20)

NoSQL for Netflix