Data Science with Elastic MapReduce (EMR) at Netflix

•

2 likes•2,309 views

Netflix uses a data warehouse called Aegisthus built on the DataStax Enterprise (DSE) platform. Aegisthus uses DSE, Cassandra, and S3. It provides a high SLA for querying and analyzing data for tasks like data science and metadata services. Data is stored in S3 for high availability and queried using services like Chukwa, Honu, Sting, and DSE.

What is Netflix’s data warehouse?
a) Cassandra
b) Teradata
c) Hive
d) S3

DSE Platform

Chukwa / Honu

Aegisthus
S3

DSE Platform
Sting
Chukwa / Honu

Aegisthus
S3

“Data Science as a Service”

• Execution Service / Genie

• Event Service

• Metadata Service

High SLA Cluster Job

High SLA

S3
Query Cluster Job

Query

Super SLA Cluster Job

Super SLA

High SLA Cluster Job
S3
High SLA

Query Cluster Job

Query

Super SLA Cluster Job

High SLA Cluster Job
High SLA

S3
Query Cluster Job

Query

Questions?

http://jobs.netflix.com
kurtbrown@netflix.com

For a service like Netflix, data is crucial. In this session, the Director of Cloud Platform Engineering at Netflix details how they chose and leveraged Cassandra, a highly-available and scalable open source key/value store. In this presentation they discuss why they chose Cassandra, the tools and processes they developed to quickly and safely move data into AWS without sacrificing availability or performance, and best practices that help Cassandra work well in AWS.

Netflix - Elevating Your Data Platform - TDWI Keynote - San Diego 2015

Kurt Brown

Are you getting the most out of your data platform? The technologies you choose are important, but even more so is how you put them into practice. Part philosophy and part pragmatic reality, I'll dive into our thinking at Netflix on technology selection and trade-offs, challenging everything (constructively), providing building blocks and paved paths, staffing, and more. I'll also talk through our tech stack, which includes many big data technologies (e.g. Hadoop, Spark, and Presto), traditional BI tools (e.g. Teradata, MicroStrategy, and Tableau), and custom tools / services (e.g. our big data portal and API). My hope (and expectation!) is that you'll leave with an arsenal of new ideas on the best way to get things done.

ENT101 Embracing the Cloud - AWS re: Invent 2012

Amazon Web Services

re:Invent 2012 Optimizing Cassandra

Ruslan Meshenberg

Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale

Sriram Krishnan

The Data Platform at Twitter supports engineers and data scientists running batch jobs on Hadoop clusters that are several 1000s of nodes, and real-time jobs on top of systems such as Storm. In this presentation, I discuss the overall Data Platform stack at Twitter. In particular, I talk about enabling real-time and batch analytics at scale with the help of Scalding, which is a Scala DSL for batch jobs using MapReduce, Summingbird, which is a framework for combined real-time and batch processing, and Tsar, which is a framework for real-time time-series aggregations.

BDT303 Data Science with Elastic MapReduce - AWS re: Invent 2012

Amazon Web Services

In this talk, we dive into the Netflix Data Science & Engineering architecture. Not just the what, but also the why. Some key topics include the big data technologies we leverage (Cassandra, Hadoop, Pig + Python, and Hive), our use of Amazon S3 as our central data hub, our use of multiple persistent Amazon Elastic MapReduce (EMR) clusters, how we leverage the elasticity of AWS, our data science as a service approach, how we make our hybrid AWS / data center setup work well, and more.

A Tool for Practical Garbage Collection Analysis In the CloudArun Kejariwal

A few years ago, Netflix had a fairly classic business intelligence tech stack. Now, things have changed. Netflix is a heavy user of AWS for much of its ongoing operations, and Data Science & Engineering (DSE) is no exception. In this talk, we dive into the Netflix DSE architecture: what and why. Key topics include their use of Big Data technologies (Cassandra, Hadoop, Pig + Python, and Hive); their Amazon S3 central data hub; their multiple persistent Amazon EMR clusters; how they benefit from AWS elasticity; their data science-as-a-service approach, how they made a hybrid AWS/data center setup work well, their open-source Hadoop-related software, and more.

NetflixOSS meetup lightning talks and roadmapRuslan Meshenberg

AWS Re:Invent - Optimizing Costs with AWS

Coburn Watson

Airflow at WePay

Chris Riccomini

Netflix Data Engineering @ Uber Engineering Meetup

Blake Irvine

MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012

Amazon Web Services

Netflix designed a massive scale cloud based media transcoding system from scratch for processing professionally produced studio content(to meet the unique scale and time constraints of our business). We bucked the common industry trend of vertical scaling and, instead, designed a horizontally scaled elastic system using AWS to meet the unique scale and time constraints of our business. Come hear how we designed this system, how it continues to get less expensive for Netflix, and how AWS represents a transformative opportunity in the wider media owning industry.

Introduction to Apache Airflow - Data Day Seattle 2016

Sid Anand

RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...

Amazon Web Services

Devops at Netflix (re:Invent)

Jeremy Edberg

Airflow - An Open Source Platform to Author and Monitor Data PipelinesDataWorks Summit

AWS Re:Invent - High Availability Architecture at Netflix

Adrian Cockcroft

Netflix oss season 2 episode 1 - meetup Lightning talksRuslan Meshenberg

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

Amazon Web Services

In this session, we discuss how Spark and Presto complement the Netflix big data platform stack that started with Hadoop, and the use cases that Spark and Presto address. Also, we discuss how we run Spark and Presto on top of the Amazon EMR infrastructure; specifically, how we use Amazon S3 as our data warehouse and how we leverage Amazon EMR as a generic framework for data-processing cluster management.

ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012

Amazon Web Services

ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012

Amazon Web Services

Linux Profiling at Netflix

Brendan Gregg

Talk for SCaLE13x. Video: https://www.youtube.com/watch?v=_Ik8oiQvWgo . Profiling can show what your Linux kernel and appliacations are doing in detail, across all software stack layers. This talk shows how we are using Linux perf_events (aka "perf") and flame graphs at Netflix to understand CPU usage in detail, to optimize our cloud usage, solve performance issues, and identify regressions. This will be more than just an intro: profiling difficult targets, including Java and Node.js, will be covered, which includes ways to resolve JITed symbols and broken stacks. Included are the easy examples, the hard, and the cutting edge.

Culture

Reed Hastings

2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개

Amazon Web Services Korea

이 강연에서는 AWS Big Data 분석 아키텍처 모범 사례를 살펴보고 표준 SQL을 사용해 Amazon S3에 저장된 데이터를 간편하게 분석할 수 있는 대화식 쿼리 서비스인 Amazon Athena의 특징과 최신 기능들에 대하여 고객 사례와 함께 소개드립니다. 연사: Greg Khairallah, 아마존 웹서비스 Amazon Big Data 및 Athena 총괄 사업 개발 매니저

Getting Buzzed on Buzzwords: Using Cloud & Big Data to Pentest at Scale

Bishop Fox

You’ve heard about cloud, big data, server-less infrastructure, web scale, and other buzzwords that cause VCs to throw money at people - but how does this help you? If you’re getting bored going over the same checklist in your pentests then you’re missing out on what some of these new technologies can offer you. Using some of the newer cloud technologies not only can you automate all of your workflows, but you can do so with almost zero maintenance at a low cost with almost infinite scalability! This talk will show you how to blow conventional pentesters out of the water using some cool new technologies along with a little bit of trickery. Some of the topics we’ll go over include: * Cheap and scalable rainbow tables with BigQuery, 5TB in 10 seconds * SQS & Lambda, like Burp Intruder but 10K QPS * Scalable GPU Clusters on the cheap with Spot Instances and Elastic Beanstalk * Cloud exit nodes, rotating IPs via Elastic Beanstalk and nano instances * Cost effective fuzzing with Elastic Beanstalk and Spot Instances (This was originally presented on November 16, 2018 at Kiwicon 2038).

Deep Dive on Amazon S3

Adrian Hornsby

Running Presto and Spark on the Netflix Big Data Platform

Eva Tse

Viewers also liked

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

Bolke de Bruin

Data Science at Netflix with Amazon EMR (BDT306) | AWS re:Invent 2013

Amazon Web Services

NetflixOSS meetup lightning talks and roadmapRuslan Meshenberg

AWS Re:Invent - Optimizing Costs with AWS

Coburn Watson

Airflow at WePay

Chris Riccomini

Netflix Data Engineering @ Uber Engineering Meetup

Blake Irvine

MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012

Amazon Web Services

Introduction to Apache Airflow - Data Day Seattle 2016

Sid Anand

RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...

Amazon Web Services

Devops at Netflix (re:Invent)

Jeremy Edberg

Airflow - An Open Source Platform to Author and Monitor Data PipelinesDataWorks Summit

AWS Re:Invent - High Availability Architecture at Netflix

Adrian Cockcroft

Netflix oss season 2 episode 1 - meetup Lightning talksRuslan Meshenberg

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

Amazon Web Services

ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012

Amazon Web Services

ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012

Amazon Web Services

Linux Profiling at Netflix

Brendan Gregg

Culture

Reed Hastings

Viewers also liked (18)

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

Data Science at Netflix with Amazon EMR (BDT306) | AWS re:Invent 2013

NetflixOSS meetup lightning talks and roadmap

AWS Re:Invent - Optimizing Costs with AWS

Airflow at WePay

Netflix Data Engineering @ Uber Engineering Meetup

MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012

Introduction to Apache Airflow - Data Day Seattle 2016

RMG202 Rainmakers: How Netflix Operates Clouds for Maximum Freedom and Agilit...

Devops at Netflix (re:Invent)

Airflow - An Open Source Platform to Author and Monitor Data Pipelines

AWS Re:Invent - High Availability Architecture at Netflix

Netflix oss season 2 episode 1 - meetup Lightning talks

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012

ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012

Linux Profiling at Netflix

Culture

Similar to Data Science with Elastic MapReduce (EMR) at Netflix

2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개

Amazon Web Services Korea

Getting Buzzed on Buzzwords: Using Cloud & Big Data to Pentest at Scale

Bishop Fox

Deep Dive on Amazon S3

Adrian Hornsby

Running Presto and Spark on the Netflix Big Data Platform

Eva Tse

AWS re:Invent 2016: Case Study: How Startups Like Smartsheet and Quantcast Ac...

Amazon Web Services

Startups around the world use AWS services to access the power of the cloud to grow faster and more cost effectively. In this session, Smartsheet talks about how they were able to cost-effectively build their prototype for scale and avoid replatforming at different points in the adoption curve, and Quantcast discusses how they are running a high-performance analytics solution on AWS. They provide several tips and tricks for S3, and show how they removed a traditional MySQL data store from a distributed-image hosting application so that the only required data store is S3. They also show how to avoid common, cumbersome database practices by working with the eventually consistent nature of S3 objects and the fact that objects and directories share the same namespace.

AWS re:Invent 2016: Workshop: AWS S3 Deep-Dive Hands-On Workshop: Deploying a...

Amazon Web Services

In this session, we’ll expand on the S3 re:Invent deep-dive session with a hands-on workshop on advanced S3 features and storage management capabilities. We’ll have AWS S3 and Glacier experts on-hand to deep-dive on S3 architecture, performance & scalability optimization, how to analyze your content and leverage storage tiers (S3 Standard, S3 Standard Infrequent Access, Glacier) to balance cost and SLAs, security considerations, replication with Cross Region Replication (CRR), versioning for data protection and more. In the hands-on lab, we’ll walk through a customer scenario: architecting a high-performance infrastructure for consumer applications. In the scenario, we’ll use sample data sets on S3, analyze object retrieval patterns and design a complete solution using many of the features S3 offers including migrating objects to an appropriate tier. Prerequisites: - Participants should have an AWS account established and available for use during the workshop. - Please bring your own laptop.

Scaling on AWS for the First 10 Million Users

Amazon Web Services

Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.

AWS re:Invent 2016: Netflix: Using Amazon S3 as the fabric of our big data ec...

Amazon Web Services

Amazon S3 is the central data hub for Netflix's big data ecosystem. We currently have over 1.5 billion objects and 60+ PB of data stored in S3. As we ingest, transform, transport, and visualize data, we find this data naturally weaving in and out of S3. Amazon S3 provides us the flexibility to use an interoperable set of big data processing tools like Spark, Presto, Hive, and Pig. It serves as the hub for transporting data to additional data stores / engines like Teradata, Redshift, and Druid, as well as exporting data to reporting tools like Microstrategy and Tableau. Over time, we have built an ecosystem of services and tools to manage our data on S3. We have a federated metadata catalog service that keeps track of all our data. We have a set of data lifecycle management tools that expire data based on business rules and compliance. We also have a portal that allows users to see the cost and size of their data footprint. In this talk, we’ll dive into these major uses of S3, as well as many smaller cases, where S3 smoothly addresses an important data infrastructure need. We will also provide solutions and methodologies on how you can build your own S3 big data hub.

Cassandra Data Modeling - Practical Considerations @ Netflix

nkorla1share

Escalando hasta sus primeros 10 millones de usuarios

Amazon Web Services LATAM

Escalando hasta sus primeros 10 millones de usuarios

Amazon Web Services LATAM

Log Analysis At Scale

Amazon Web Services

As the amount of data being created by applications increases, the requirement to keep pace in this space becomes increasingly difficult. This session covers how to properly collect, manage and present the data usefully using service offerings from Amazon Web Services such as Amazon RedShift and Amazon Kinesis. At this session we will include live demo on parsing the data generated and management of the data. Alex Smith, Solutions Architect, Amazon Web Services, ASEAN

Clojure at BackType

nathanmarz

ITSubbotik - как скрестить ежа с ужом или подводные камни внедрения функциона...

Vyacheslav Lapin

Protect & Manage Amazon S3 & Amazon Glacier Objects at Scale (STG316-R1) - AW...

Amazon Web Services

As your data repository grows on AWS using the object storage services Amazon S3 and Amazon Glacier, it becomes increasingly helpful to use particular features to help protect and manage your objects. In this chalk talk, you have the opportunity to speak directly with the AWS engineering team that builds and maintains features like Cross-Region Replication, S3 Storage Class Analysis, S3 Inventory, S3 Lifecycle, Amazon Glacier Vault Lock, and others. Bring your feedback, questions, and expertise to discuss innovative ways to protect data from corruption or malicious and accidental deletion, managing the data lifecycle to reduce costs, identifying wasted storage, and much more.

AWS Melbourne CO Interest Group meetup - Lightning talks - 20 Nov 2017

Peter Shi

Introduction to Amazon Athena

Amazon Web Services

SRV301 Latest Updates & Best Practices for Amazon S3

Amazon Web Services

Get the latest on what we've been developing in Amazon S3. In this session, we share best practices for performance optimization, security, data protection, storage management, and much more. We discuss ways to optimize key naming to increase throughput, apply the appropriate AWS Identity and Access Management (IAM) and encryption configurations, and take advantage of object tagging and other features to enhance security.

2018512 AWS上での機械学習システムの構築とSageMaker

Yasuhiro Matsuo

Querying and Analyzing Data in Amazon S3

Amazon Web Services

Similar to Data Science with Elastic MapReduce (EMR) at Netflix (20)

2017 AWS DB Day | Amazon Athena 서비스 최신 기능 소개

Getting Buzzed on Buzzwords: Using Cloud & Big Data to Pentest at Scale

Deep Dive on Amazon S3

Running Presto and Spark on the Netflix Big Data Platform

AWS re:Invent 2016: Case Study: How Startups Like Smartsheet and Quantcast Ac...

AWS re:Invent 2016: Workshop: AWS S3 Deep-Dive Hands-On Workshop: Deploying a...

Scaling on AWS for the First 10 Million Users

AWS re:Invent 2016: Netflix: Using Amazon S3 as the fabric of our big data ec...

Cassandra Data Modeling - Practical Considerations @ Netflix

Escalando hasta sus primeros 10 millones de usuarios

Log Analysis At Scale

Clojure at BackType

ITSubbotik - как скрестить ежа с ужом или подводные камни внедрения функциона...

Protect & Manage Amazon S3 & Amazon Glacier Objects at Scale (STG316-R1) - AW...

AWS Melbourne CO Interest Group meetup - Lightning talks - 20 Nov 2017

Introduction to Amazon Athena

SRV301 Latest Updates & Best Practices for Amazon S3

2018512 AWS上での機械学習システムの構築とSageMaker

Querying and Analyzing Data in Amazon S3

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

National Security Agency - NSA mobile device best practices

Quotidiano Piemontese

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Neo4j

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

DevOps and Testing slides at DASA Connect

Kari Kakkonen

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

Free Complete Python - A step towards Data Science

RinaMondal9

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance