Hadoop and Pig at Twitter__HadoopSummit2010

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Hadoop at Twitter <ul><li>Kevin Weil -- @kevinweil </li></ul><ul><li>Analytics Lead, Twitter </li></ul>

The Twitter Data Lifecycle <ul><li>Data Input </li></ul><ul><li>Data Storage </li></ul><ul><li>Data Analysis </li></ul><ul...

The Twitter Data Lifecycle <ul><li>Data Input: Scribe , Crane </li></ul><ul><li>Data Storage: Elephant Bird , HBase </...

My Background <ul><li>Studied Mathematics and Physics at Harvard, Physics at Stanford </li></ul><ul><li>Tropos Networks (...

The Twitter Data Lifecycle <ul><li>Data Input: Scribe , Crane </li></ul><ul><li>Data Storage </li></ul><ul><li>Data Anal...

What Data? <ul><li>Two main kinds of raw data </li></ul><ul><li>Logs </li></ul><ul><li>Tabular data </li></ul>

Logs <ul><li>Started with syslog-ng </li></ul><ul><li>As our volume grew, it didn’t scale </li></ul>

Logs <ul><li>Started with syslog-ng </li></ul><ul><li>As our volume grew, it didn’t scale </li></ul><ul><li>Resources over...

Scribe <ul><li>Scribe daemon runs locally; reliable in network outage </li></ul><ul><li>Nodes only know downstream </li></...

Scribe at Twitter <ul><li>Solved our problem, opened new vistas </li></ul><ul><li>Currently 57 different categories logged...

Scribe at Twitter <ul><li>We’ve contributed to it as we’ve used it 1 </li></ul><ul><li>Improved logging, monitoring, writi...

Tabular Data <ul><li>Most site data is in MySQL </li></ul><ul><li>Tweets, users, devices, client applications, etc </li></...

Crane Driver Configuration/Batch Management Extract Load Transform Protobuf P1 Protobuf P2 Source Sink ZooKeeper Registrat...

Crane <ul><li>Extract </li></ul><ul><li>MySQL, HDFS, HBase, Flock, GA, Facebook Insights </li></ul>

Crane <ul><li>Extract </li></ul><ul><li>MySQL, HDFS, HBase, Flock, GA, Facebook Insights </li></ul><ul><li>Transform </li>...

Crane <ul><li>Extract </li></ul><ul><li>MySQL, HDFS, HBase, Flock, GA, Facebook Insights </li></ul><ul><li>Transform </li>...

Crane <ul><li>Extract </li></ul><ul><li>MySQL, HDFS, HBase, Flock, GA, Facebook Insights </li></ul><ul><li>Transform </li>...

The Twitter Data Lifecycle <ul><li>Data Input </li></ul><ul><li>Data Storage: Elephant Bird , HBase </li></ul><ul><li>Da...

Storage Basics <ul><li>Incoming data: 7 TB/day </li></ul><ul><li>LZO encode everything </li></ul><ul><li>Save 3-4x on stor...

Elephant Bird http://www.flickr.com/photos/jagadish/3072134867/ 1 http://github.com/kevinweil/elephant-bird

Elephant Bird <ul><li>We have data coming in as protocol buffers via Crane... </li></ul>

Elephant Bird <ul><li>We have data coming in as protocol buffers via Crane... </li></ul><ul><li>Protobufs: codegen for eff...

Elephant Bird <ul><li>We have data coming in as protocol buffers via Crane... </li></ul><ul><li>Protobufs: codegen for eff...

Elephant Bird <ul><li>We have data coming in as protocol buffers via Crane... </li></ul><ul><li>Protobufs: codegen for eff...

Elephant Bird <ul><li>We have data coming in as protocol buffers via Crane... </li></ul><ul><li>Protobufs: codegen for eff...

Challenge: Mutable Data <ul><li>HDFS is write-once: no seek on write, no append (yet) </li></ul><ul><li>Logs are easy. </l...

Challenge: Mutable Data <ul><li>HDFS is write-once: no seek on write, no append (yet) </li></ul><ul><li>Logs are easy. </l...

Challenge: Mutable Data <ul><li>HDFS is write-once: no seek on write, no append (yet) </li></ul><ul><li>Logs are easy. </l...

HBase <ul><li>Has already solved the update problem </li></ul><ul><li>Bonus: low-latency query API </li></ul><ul><li>Bonus...

HBase at Twitter <ul><li>Crane loads data directly into HBase </li></ul><ul><li>One CF for protobuf bytes, one CF to denor...

HBase at Twitter <ul><li>Crane loads data directly into HBase </li></ul><ul><li>One CF for protobuf bytes, one CF to denor...

The Twitter Data Lifecycle <ul><li>Data Input </li></ul><ul><li>Data Storage </li></ul><ul><li>Data Analysis: Pig , Oink ...

Enter Pig <ul><li>High level language </li></ul><ul><li>Transformations on sets of records </li></ul><ul><li>Process data ...

Why Pig? <ul><li>Because I bet you can read the following script. </li></ul>

A Real Pig Script <ul><li>Now, just for fun... the same calculation in vanilla Hadoop MapReduce. </li></ul>

No, seriously.

Pig Democratizes Large-scale Data Analysis <ul><li>The Pig version is: </li></ul><ul><ul><li>5% of the code </li></ul></ul...

Pig Examples <ul><li>Using the HBase Loader </li></ul><ul><li>Using the protobuf loaders </li></ul>

Pig Workflow <ul><li>Oink: framework around Pig for loading, combining, running, post-processing </li></ul><ul><li>Everyon...

Counting Big Data <ul><li>standard counts, min, max, std dev </li></ul><ul><li>How many requests do we serve in a day? </l...

Correlating Big Data <ul><li>How does usage differ for mobile users? </li></ul><ul><li>How about for users with 3rd party ...

Research on Big Data <ul><li>What can we tell about a user from their tweets? </li></ul><ul><li>From the tweets of those t...

Research on Big Data <ul><li>Sentiment analysis </li></ul><ul><li>What features get a tweet retweeted? </li></ul><ul><li>H...

The Twitter Data Lifecycle <ul><li>Data Input </li></ul><ul><li>Data Storage </li></ul><ul><li>Data Analysis </li></ul><ul...

Data Products <ul><li>Ad Hoc Analyses </li></ul><ul><li>Answer questions to keep the business agile, do research </li></ul...

Questions ? Follow me at twitter.com/kevinweil TM <ul><li>P.S. We’re hiring. Help us build the next step: realtime big da...

You are reading a preview.

Activate your 30 day free trial to continue reading.

Share SlideShare

Facebook
Twitter
LinkedIn

Embed

Size (px)

Show related SlideShares at end

WordPress Shortcode

Link

LinkedIn
Facebook
Twitter
Email

Replay

Top clipped slide

1 of 46 Ad

1 of 46 Ad