NoSQL at Twitter (NoSQL EU 2010)

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

NoSQL at Twitter
Kevin Weil -- @kevinweil
Analytics Lead, Twitter

April 21, 2010

TM

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

My Background
‣ Studied Mathematics and Physics at Harvard, Physics at
Stanford
‣ Tropos Networks (city-wide wirel...

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Data, Data Everywhere
‣ Twitter users generate a lot of data
‣ Anybody want to guess?
‣ 7 TB/day (2+ PB/yr)
‣ ...

Syslog?
‣ Started with syslog-ng
‣ As our volume grew, it didn’t scale

Syslog?
‣ Started with syslog-ng
‣ As our volume grew, it didn’t scale
‣ Resources overwhelmed
‣ Lost data

Scribe
‣ Surprise! FB had same problem, built and open-sourced Scribe
‣ Log collection framework over Thrift
‣ You w...

Scribe
‣ Runs locally; reliable in network outage

FE FE FE

Scribe
‣ Runs locally; reliable in network outage
‣ Nodes only know downstream
...

Scribe
‣ Runs locally; reliable in network outage
‣ Nodes only know downstream
...

Scribe at Twitter
‣ Solved our problem, opened new vistas
‣ Currently 30 different categories logged from multiple sou...

Scribe at Twitter
‣ We’ve contributed to it as we’ve used it
‣ Improved logging, monitoring, writing to HDFS, comp...

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?
‣ ~80 MB/s

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?
‣ ~80 MB/s
‣ 24.3 hours to write 7 TB

How do you store 7TB/day?
‣ Single machine?
‣ What’s HD write speed?
‣ ~80 MB/s
‣ 24.3 hours to write 7 TB
‣ U...

Where do I put 7TB/day?
‣ Need a cluster of machines

Where do I put 7TB/day?
‣ Need a cluster of machines

‣ ... which adds new layers of complexity

Hadoop
‣ Distributed file system
‣ Automatic replication, fault tolerance

Hadoop
‣ Distributed file system
‣ Automatic replication, fault tolerance
‣ MapReduce-based parallel computation
‣...

Hadoop
‣ Open source: top-level Apache project
‣ Scalable: Y! has a 4000 node cluster
‣ Powerful: sorted 1TB of rand...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

MapReduce Workflow
Inputs

Map
Shufﬂe/Sort ‣ Challenge: how many twe...

Two Analysis Challenges
‣ 1. Compute friendships in Twitter’s social graph
‣ grep, awk? No way.
‣ Data is in MyS...

Two Analysis Challenges
‣ 1. Compute friendships in Twitter’s social graph
‣ grep, awk? No way.
‣ Data is in MyS...

Two Analysis Challenges
‣ 2. Large-scale grouping and counting?
‣ select count(*) from users? Maybe...
‣ select co...

Back to Hadoop
‣ Didn’t we have a cluster of machines?

Back to Hadoop
‣ Didn’t we have a cluster of machines?

Back to Hadoop
‣ Didn’t we have a cluster of machines?
‣ Hadoop makes it easy to distribute the
calculation
‣ Pu...

Back to Hadoop
‣ Didn’t we have a cluster of machines?
‣ Hadoop makes it easy to distribute the
calculation
‣ Pu...

Analysis at scale
‣ Now we’re rolling
‣ Count all tweets: 12 billion, 5 minutes
‣ Hit FlockDB in parallel to assembl...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

But...
‣ Analysis typically in Java
‣ “I need less Java in my life, not more.”
‣ Single-input, two-stage data flow...

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

Pig
‣ High-level language
‣ Transformations on sets of records
‣ Process data one step at a time
‣ Easier than SQL?

Why Pig?
‣ Because I bet you can read the following script.

A Real Pig Script

A Real Pig Script

‣ Now, just for fun... the same calculation in vanilla Hadoop MapReduce.

Pig Democratizes Large-scale Data
Analysis
‣ The Pig version is:
‣ 5% of the code

Pig Democratizes Large-scale Data
Analysis
‣ The Pig version is:
‣ 5% of the code
‣ 5% of the time

Pig Democratizes Large-scale Data
Analysis
‣ The Pig version is:
‣ 5% of the code
‣ 5% of the time
‣ Within ...

One Thing I’ve Learned
‣ It’s easy to answer questions
‣ It’s hard to ask the right questions

One Thing I’ve Learned
‣ It’s easy to answer questions
‣ It’s hard to ask the right questions

‣ Value the system t...

One Thing I’ve Learned
‣ It’s easy to answer questions
‣ It’s hard to ask the right questions

‣ Value the system t...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

The Hadoop Ecosystem at Twitter
‣ Running Cloudera’s free distro, CDH2 and Hadoop 0.20.1
‣ Heavily modified Scribe wri...

Data?
‣ Semi-structured: apache logs
(search, .com, mobile), search query
logs, RoR logs, mysql query logs, A/B
...

Data?
‣ Semi-structured: apache logs
(search, .com, mobile), search query
logs, RoR logs, mysql query logs, A/B
...

Data?
‣ Semi-structured: apache logs
(search, .com, mobile), search query
logs, RoR logs, mysql query logs, A/B
...

So what do we do with it?

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What ...

Counting Big Data
‣ standard counts, min, max, std dev
‣ How many requests do we serve in a day?
‣ What...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ probabilities, covariance, influence
‣ How does usage differ for mobile users?
‣ ...

Correlating Big Data
‣ What is the correlation between users with registered phones
and users that tweet?

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ What can we tell about a user from their...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features get ...

Research on Big Data
‣ prediction, graph analysis, natural language
‣ Sentiment analysis
‣ What features ge...

Research on Big Data
‣ How well can we detect bots and other non-human tweeters?

Introduction
‣ How We Arrived at NoSQL: A Crash Course
‣ Collecting Data (Scribe)
‣ Storing and Analyzing Data (...

HBase
‣ BigTable clone on top of HDFS
‣ Distributed, column-oriented, no datatypes
‣ Unlike the rest of HDFS, design...

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search
‣ Old version: offline process ...

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search
‣ Old version: offline process ...

HBase at Twitter
‣ We began building real products based on Hadoop
‣ People search
‣ Old version: offline process ...

People Search
‣ Import user data into HBase

People Search
‣ Import user data into HBase
‣ Periodic MapReduce job reading from HBase
‣ Hits FlockDB, multiple o...

People Search
‣ Import user data into HBase
‣ Periodic MapReduce job reading from HBase
‣ Hits FlockDB, multiple o...

People Search
‣ Import user data into HBase
‣ Periodic MapReduce job reading from HBase
‣ Hits FlockDB, multiple o...

HBase
‣ More products now being built on top of it
‣ Flexible, easy to connect to MapReduce/Pig

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ “Their origins reveal their strengths and weaknesses”
‣ HBase built on top of batch-oriented syst...

HBase vs Cassandra
‣ Your mileage may vary
‣ At Twitter: HBase for analytics, analysis, dataset generation
‣ Cas...

HBase vs Cassandra
‣ Your mileage may vary
‣ At Twitter: HBase for analytics, analysis, dataset generation
‣ Cas...

FlockDB
‣ Realtime, distributed
social graph store
‣ NOT optimized for data mining

‣ Note: the following slide...

FlockDB
‣ Realtime, distributed Intersection
Temporal

soci...

Set operations?
‣ This tweet needs to
be delivered to people
who follow both
@aplusk (4.7M
followers) an...

Original solution
‣ MySQL table source_id destination-id

‣ Indices on source_id
...

Next Try
‣ MySQL still
‣ Denormalized
‣ Byte-packed
‣ Chunked
‣ Still temporally ordered

Next Try
‣ Problems
‣ O(n) deletes
‣ Data consistency challenges
‣ Inefficient intersections
‣ All of thes...

FlockDB
‣ MySQL underneath still (like PNUTS from Y!)
‣ Partitioned by user_id, gizzard handles sharding/partitioning
...

FlockDB Timings
‣ Counts: 1ms

FlockDB Timings
‣ Counts: 1ms
‣ Temporal Query: 2ms

FlockDB Timings
‣ Counts: 1ms
‣ Temporal Query: 2ms
‣ Writes: 1ms for journal, 16ms for durability

FlockDB Timings
‣ Counts: 1ms
‣ Temporal Query: 2ms
‣ Writes: 1ms for journal, 16ms for durability
‣ Full walks: 1...

FlockDB is Open Source
‣ We will maintain a community at
‣ http://www.github.com/twitter/flockdb
‣ http://...

Cassandra
‣ Why Cassandra, for Twitter?

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Old/current: vertically, horizontally partitioned MySQL
‣ All kinds of...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Cassandra
‣ Why Cassandra, for Twitter?
‣ Decentralized, fault-tolerant
‣ All kinds of caching layers, all appli...

Eventually Consistent?
‣ Twitter is already eventually consistent

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse
‣ Ryan’s new ter...

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse
‣ Ryan’s new ter...

Eventually Consistent?
‣ Twitter is already eventually consistent
‣ Your system may be even worse
‣ Ryan’s new ter...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches f...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches f...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches f...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches...

Rolling out Cassandra
‣ 1. Integrate Cassandra alongside MySQL
‣ 100% reads/writes to MySQL
‣ Dynamic switches...

Cassandra for Realtime Analytics
‣ Starting a project around realtime analytics
‣ Cassandra as the backing store
‣ ...

That was a lot of slides
‣ Thanks for sticking with me.

Questions? Follow me at
twitter.com/kevinweil

TM

You are reading a preview.

Activate your 30 day free trial to continue reading.

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 156 Ad

1 of 156 Ad