SlideShare a Scribd company logo
1 of 35
Download to read offline
Hadoop and Pig @Twitter
              Kevin Weil -- @kevinweil
              Analytics Lead, Twitter




                                         TM




Friday, July 23, 2010
Agenda
           ‣     Hadoop Overview
           ‣     Pig: Rapid Learning Over Big Data
           ‣     Data-Driven Products
           ‣     Hadoop/Pig and Analytics




Friday, July 23, 2010
My Background
           ‣     Mathematics and Physics at Harvard, Physics at
                 Stanford
           ‣     Tropos Networks (city-wide wireless): mesh
                 routing algorithms, GBs of data
           ‣     Cooliris (web media): Hadoop and Pig for
                 analytics, TBs of data
           ‣     Twitter: Hadoop, Pig, HBase, Cassandra,
                 machine learning, visualization, social graph
                 analysis, soon to be PBs data



Friday, July 23, 2010
Agenda
           ‣     Hadoop Overview
           ‣     Pig: Rapid Learning Over Big Data
           ‣     Data-Driven Products
           ‣     Hadoop/Pig and Analytics




Friday, July 23, 2010
Data is Getting Big
           ‣     NYSE: 1 TB/day
           ‣     Facebook: 20+ TB
                 compressed/day
           ‣     CERN/LHC: 40 TB/day
                 (15 PB/year)
           ‣     And growth is
                 accelerating
           ‣     Need multiple machines,
                 horizontal scalability


Friday, July 23, 2010
Hadoop
           ‣      Distributed file system (hard to store a PB)
           ‣      Fault-tolerant, handles replication, node failure,
                  etc
           ‣      MapReduce-based parallel computation
                  (even harder to process a PB)
           ‣      Generic key-value based computation interface
                  allows for wide applicability




Friday, July 23, 2010
Hadoop
           ‣      Open source: top-level Apache project
           ‣      Scalable: Y! has a 4000-node cluster
           ‣      Powerful: sorted a TB of random integers in 62
                  seconds


           ‣      Easy Packaging: Cloudera RPMs, DEBs




Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
MapReduce Workflow
    Inputs
                                                           ‣   Challenge: how many tweets per
                        Map
                              Shuffle/Sort                      user, given tweets table?
                        Map
                                                           ‣   Input: key=row, value=tweet info
                                                 Outputs
                        Map             Reduce             ‣   Map: output key=user_id,
                        Map             Reduce
                                                               value=1
                        Map             Reduce             ‣   Shuffle: sort by user_id
                        Map                                ‣   Reduce: for each user_id, sum
                        Map
                                                           ‣   Output: user_id, tweet count
                                                           ‣   With 2x machines, runs 2x faster

Friday, July 23, 2010
But...
           ‣     Analysis typically in Java
           ‣     Single-input, two-stage
                 data flow is rigid
           ‣     Projections, filters:
                 custom code
           ‣     Joins are lengthy, error-prone
           ‣     Hard to manage n-stage jobs
           ‣     Exploration requires compilation!



Friday, July 23, 2010
Agenda
           ‣     Hadoop Overview
           ‣     Pig: Rapid Learning Over Big Data
           ‣     Data-Driven Products
           ‣     Hadoop/Pig and Analytics




Friday, July 23, 2010
Enter Pig
          ‣      High level language
          ‣      Transformations on
                 sets of records
          ‣      Process data one step at a time
          ‣      Easier than SQL?


          ‣      Top-level Apache project



Friday, July 23, 2010
Why Pig?
             ‣      Because I bet you can read the following script.




Friday, July 23, 2010
A Real Pig Script




Friday, July 23, 2010
Now, just for fun...
             ‣      The same calculation in vanilla MapReduce




Friday, July 23, 2010
No, seriously.




Friday, July 23, 2010
Pig Democratizes Large-scale
           Data Analysis
           ‣     The Pig version is:
           ‣            5% of the code
           ‣            5% of the development time
           ‣            Within 25% of the execution time
           ‣            Readable, reusable




Friday, July 23, 2010
One Thing I’ve Learned
           ‣     It’s easy to answer questions
           ‣     It’s hard to ask the right questions


           ‣     Value the system that promotes innovation and
                 iteration




Friday, July 23, 2010
Agenda
           ‣     Hadoop Overview
           ‣     Pig: Rapid Learning Over Big Data
           ‣     Data-Driven Products
           ‣     Hadoop/Pig and Analytics




Friday, July 23, 2010
MySQL, MySQL, MySQL
           ‣     We all start there.
           ‣     But MySQL is not built for analysis.
           ‣     select count(*) from users? Maybe.
           ‣     select count(*) from tweets? Uh...
           ‣     Imagine joining them.
           ‣     And grouping.
           ‣     Then sorting.



Friday, July 23, 2010
Non-Pig Hadoop at Twitter
           ‣     Data Sink via Scribe
           ‣     Distributed Grep
           ‣     A few performance-critical, simple jobs
           ‣     People Search




Friday, July 23, 2010
People Search?
           ‣     First real product built with Hadoop
           ‣     “Find People”
           ‣     Old version: offline process on
                 a single node
           ‣     New version: complex graph
                 calculations, hit internal network
                 services, custom indexing
           ‣     	      Faster, more reliable,
                 more observable
Friday, July 23, 2010
People Search
           ‣     Import user data into HBase
           ‣     Periodic MapReduce job reading from HBase
           ‣      Hits FlockDB, other internal services in
                 mapper
           ‣            Custom partitioning
           ‣     Data sucked across to sharded, replicated,
                 horizontally scalable, in-memory, low-latency
                 Scala service
           ‣       Build a trie, do case folding/normalization,
                 suggestions, etc
Friday, July 23, 2010
Agenda
           ‣     Hadoop Overview
           ‣     Pig: Rapid Learning Over Big Data
           ‣     Data-Driven Products
           ‣     Hadoop/Pig and Analytics




Friday, July 23, 2010
Order of Operations

          ‣      Counting



          ‣      Correlating



          ‣      Research/
                 Algorithmic
                 Learning

Friday, July 23, 2010
Counting
           ‣     How many requests per day?
           ‣     What’s the average latency? 95% latency?
           ‣     What’s the response code distribution?
           ‣     How many searches per day? Unique users?
           ‣     What’s the geographic breakdown of requests?
           ‣     How many tweets? From what clients?
           ‣     How many signups? Profile completeness?
           ‣     How many SMS notifications did we send?


Friday, July 23, 2010
Correlating
           ‣     How does usage differ for mobile users?
           ‣     ... for desktop client users (Tweetdeck, etc)?
           ‣     Cohort analyses
           ‣     What services fail at the same time?
           ‣     What features get users hooked?
           ‣     What do successful users do often?
           ‣     How does tweet volume change over time?



Friday, July 23, 2010
Research
           ‣     What can we infer from a user’s tweets?
           ‣     ... from the tweets of their followers? followees?
           ‣     What features tend to get a tweet retweeted?
           ‣     ... and what influences the retweet tree depth?
           ‣     Duplicate detection, language detection
           ‣     What graph structures lead to increased usage?
           ‣     Sentiment analysis, entity extraction
           ‣     User reputation


Friday, July 23, 2010
If We Had More Time...
           ‣     HBase
           ‣     LZO compression and Hadoop
           ‣     Protocol buffers
           ‣     Our open source: hadoop-lzo, elephant-bird
           ‣     Analytics and Cassandra




Friday, July 23, 2010
Questions?
                   Follow me at
                   twitter.com/kevinweil



                                           TM




Friday, July 23, 2010

More Related Content

Viewers also liked

Hadoop and Pig at Twitter__HadoopSummit2010
Hadoop and Pig at Twitter__HadoopSummit2010Hadoop and Pig at Twitter__HadoopSummit2010
Hadoop and Pig at Twitter__HadoopSummit2010Yahoo Developer Network
 
Practical Hadoop using Pig
Practical Hadoop using PigPractical Hadoop using Pig
Practical Hadoop using PigDavid Wellman
 
Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!Nathan Bijnens
 
Introduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUGIntroduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUGAdam Kawa
 
Practical Problem Solving with Apache Hadoop & Pig
Practical Problem Solving with Apache Hadoop & PigPractical Problem Solving with Apache Hadoop & Pig
Practical Problem Solving with Apache Hadoop & PigMilind Bhandarkar
 
Extending Hadoop for Fun & Profit
Extending Hadoop for Fun & ProfitExtending Hadoop for Fun & Profit
Extending Hadoop for Fun & ProfitMilind Bhandarkar
 
Hadoop: The Default Machine Learning Platform ?
Hadoop: The Default Machine Learning Platform ?Hadoop: The Default Machine Learning Platform ?
Hadoop: The Default Machine Learning Platform ?Milind Bhandarkar
 
Future of Data Intensive Applicaitons
Future of Data Intensive ApplicaitonsFuture of Data Intensive Applicaitons
Future of Data Intensive ApplicaitonsMilind Bhandarkar
 
The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
The Zoo Expands: Labrador *Loves* Elephant, Thanks to HamsterThe Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
The Zoo Expands: Labrador *Loves* Elephant, Thanks to HamsterMilind Bhandarkar
 
Measuring CDN performance and why you're doing it wrong
Measuring CDN performance and why you're doing it wrongMeasuring CDN performance and why you're doing it wrong
Measuring CDN performance and why you're doing it wrongFastly
 
Modeling with Hadoop kdd2011
Modeling with Hadoop kdd2011Modeling with Hadoop kdd2011
Modeling with Hadoop kdd2011Milind Bhandarkar
 
Hadoop, Pig, and Python (PyData NYC 2012)
Hadoop, Pig, and Python (PyData NYC 2012)Hadoop, Pig, and Python (PyData NYC 2012)
Hadoop, Pig, and Python (PyData NYC 2012)mortardata
 
Hive vs Pig for HadoopSourceCodeReading
Hive vs Pig for HadoopSourceCodeReadingHive vs Pig for HadoopSourceCodeReading
Hive vs Pig for HadoopSourceCodeReadingMitsuharu Hamba
 

Viewers also liked (20)

Hadoop and Pig at Twitter__HadoopSummit2010
Hadoop and Pig at Twitter__HadoopSummit2010Hadoop and Pig at Twitter__HadoopSummit2010
Hadoop and Pig at Twitter__HadoopSummit2010
 
Practical Hadoop using Pig
Practical Hadoop using PigPractical Hadoop using Pig
Practical Hadoop using Pig
 
Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!Hadoop Pig: MapReduce the easy way!
Hadoop Pig: MapReduce the easy way!
 
Introduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUGIntroduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUG
 
Practical Problem Solving with Apache Hadoop & Pig
Practical Problem Solving with Apache Hadoop & PigPractical Problem Solving with Apache Hadoop & Pig
Practical Problem Solving with Apache Hadoop & Pig
 
Un introduction à Pig
Un introduction à PigUn introduction à Pig
Un introduction à Pig
 
RainBird
RainBirdRainBird
RainBird
 
Pig workshop
Pig workshopPig workshop
Pig workshop
 
Scaling hadoopapplications
Scaling hadoopapplicationsScaling hadoopapplications
Scaling hadoopapplications
 
Hadoop Overview kdd2011
Hadoop Overview kdd2011Hadoop Overview kdd2011
Hadoop Overview kdd2011
 
Extending Hadoop for Fun & Profit
Extending Hadoop for Fun & ProfitExtending Hadoop for Fun & Profit
Extending Hadoop for Fun & Profit
 
Hadoop: The Default Machine Learning Platform ?
Hadoop: The Default Machine Learning Platform ?Hadoop: The Default Machine Learning Platform ?
Hadoop: The Default Machine Learning Platform ?
 
Future of Data Intensive Applicaitons
Future of Data Intensive ApplicaitonsFuture of Data Intensive Applicaitons
Future of Data Intensive Applicaitons
 
The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
The Zoo Expands: Labrador *Loves* Elephant, Thanks to HamsterThe Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster
 
Measuring CDN performance and why you're doing it wrong
Measuring CDN performance and why you're doing it wrongMeasuring CDN performance and why you're doing it wrong
Measuring CDN performance and why you're doing it wrong
 
Modeling with Hadoop kdd2011
Modeling with Hadoop kdd2011Modeling with Hadoop kdd2011
Modeling with Hadoop kdd2011
 
Pig statements
Pig statementsPig statements
Pig statements
 
Apache pig
Apache pigApache pig
Apache pig
 
Hadoop, Pig, and Python (PyData NYC 2012)
Hadoop, Pig, and Python (PyData NYC 2012)Hadoop, Pig, and Python (PyData NYC 2012)
Hadoop, Pig, and Python (PyData NYC 2012)
 
Hive vs Pig for HadoopSourceCodeReading
Hive vs Pig for HadoopSourceCodeReadingHive vs Pig for HadoopSourceCodeReading
Hive vs Pig for HadoopSourceCodeReading
 

Recently uploaded

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfUK Journal
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...CzechDreamin
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityScyllaDB
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 

Recently uploaded (20)

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 

Hadoop and pig at twitter (oscon 2010)

  • 1. Hadoop and Pig @Twitter Kevin Weil -- @kevinweil Analytics Lead, Twitter TM Friday, July 23, 2010
  • 2. Agenda ‣ Hadoop Overview ‣ Pig: Rapid Learning Over Big Data ‣ Data-Driven Products ‣ Hadoop/Pig and Analytics Friday, July 23, 2010
  • 3. My Background ‣ Mathematics and Physics at Harvard, Physics at Stanford ‣ Tropos Networks (city-wide wireless): mesh routing algorithms, GBs of data ‣ Cooliris (web media): Hadoop and Pig for analytics, TBs of data ‣ Twitter: Hadoop, Pig, HBase, Cassandra, machine learning, visualization, social graph analysis, soon to be PBs data Friday, July 23, 2010
  • 4. Agenda ‣ Hadoop Overview ‣ Pig: Rapid Learning Over Big Data ‣ Data-Driven Products ‣ Hadoop/Pig and Analytics Friday, July 23, 2010
  • 5. Data is Getting Big ‣ NYSE: 1 TB/day ‣ Facebook: 20+ TB compressed/day ‣ CERN/LHC: 40 TB/day (15 PB/year) ‣ And growth is accelerating ‣ Need multiple machines, horizontal scalability Friday, July 23, 2010
  • 6. Hadoop ‣ Distributed file system (hard to store a PB) ‣ Fault-tolerant, handles replication, node failure, etc ‣ MapReduce-based parallel computation (even harder to process a PB) ‣ Generic key-value based computation interface allows for wide applicability Friday, July 23, 2010
  • 7. Hadoop ‣ Open source: top-level Apache project ‣ Scalable: Y! has a 4000-node cluster ‣ Powerful: sorted a TB of random integers in 62 seconds ‣ Easy Packaging: Cloudera RPMs, DEBs Friday, July 23, 2010
  • 8. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 9. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 10. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 11. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 12. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 13. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 14. MapReduce Workflow Inputs ‣ Challenge: how many tweets per Map Shuffle/Sort user, given tweets table? Map ‣ Input: key=row, value=tweet info Outputs Map Reduce ‣ Map: output key=user_id, Map Reduce value=1 Map Reduce ‣ Shuffle: sort by user_id Map ‣ Reduce: for each user_id, sum Map ‣ Output: user_id, tweet count ‣ With 2x machines, runs 2x faster Friday, July 23, 2010
  • 15. But... ‣ Analysis typically in Java ‣ Single-input, two-stage data flow is rigid ‣ Projections, filters: custom code ‣ Joins are lengthy, error-prone ‣ Hard to manage n-stage jobs ‣ Exploration requires compilation! Friday, July 23, 2010
  • 16. Agenda ‣ Hadoop Overview ‣ Pig: Rapid Learning Over Big Data ‣ Data-Driven Products ‣ Hadoop/Pig and Analytics Friday, July 23, 2010
  • 17. Enter Pig ‣ High level language ‣ Transformations on sets of records ‣ Process data one step at a time ‣ Easier than SQL? ‣ Top-level Apache project Friday, July 23, 2010
  • 18. Why Pig? ‣ Because I bet you can read the following script. Friday, July 23, 2010
  • 19. A Real Pig Script Friday, July 23, 2010
  • 20. Now, just for fun... ‣ The same calculation in vanilla MapReduce Friday, July 23, 2010
  • 22. Pig Democratizes Large-scale Data Analysis ‣ The Pig version is: ‣ 5% of the code ‣ 5% of the development time ‣ Within 25% of the execution time ‣ Readable, reusable Friday, July 23, 2010
  • 23. One Thing I’ve Learned ‣ It’s easy to answer questions ‣ It’s hard to ask the right questions ‣ Value the system that promotes innovation and iteration Friday, July 23, 2010
  • 24. Agenda ‣ Hadoop Overview ‣ Pig: Rapid Learning Over Big Data ‣ Data-Driven Products ‣ Hadoop/Pig and Analytics Friday, July 23, 2010
  • 25. MySQL, MySQL, MySQL ‣ We all start there. ‣ But MySQL is not built for analysis. ‣ select count(*) from users? Maybe. ‣ select count(*) from tweets? Uh... ‣ Imagine joining them. ‣ And grouping. ‣ Then sorting. Friday, July 23, 2010
  • 26. Non-Pig Hadoop at Twitter ‣ Data Sink via Scribe ‣ Distributed Grep ‣ A few performance-critical, simple jobs ‣ People Search Friday, July 23, 2010
  • 27. People Search? ‣ First real product built with Hadoop ‣ “Find People” ‣ Old version: offline process on a single node ‣ New version: complex graph calculations, hit internal network services, custom indexing ‣ Faster, more reliable, more observable Friday, July 23, 2010
  • 28. People Search ‣ Import user data into HBase ‣ Periodic MapReduce job reading from HBase ‣ Hits FlockDB, other internal services in mapper ‣ Custom partitioning ‣ Data sucked across to sharded, replicated, horizontally scalable, in-memory, low-latency Scala service ‣ Build a trie, do case folding/normalization, suggestions, etc Friday, July 23, 2010
  • 29. Agenda ‣ Hadoop Overview ‣ Pig: Rapid Learning Over Big Data ‣ Data-Driven Products ‣ Hadoop/Pig and Analytics Friday, July 23, 2010
  • 30. Order of Operations ‣ Counting ‣ Correlating ‣ Research/ Algorithmic Learning Friday, July 23, 2010
  • 31. Counting ‣ How many requests per day? ‣ What’s the average latency? 95% latency? ‣ What’s the response code distribution? ‣ How many searches per day? Unique users? ‣ What’s the geographic breakdown of requests? ‣ How many tweets? From what clients? ‣ How many signups? Profile completeness? ‣ How many SMS notifications did we send? Friday, July 23, 2010
  • 32. Correlating ‣ How does usage differ for mobile users? ‣ ... for desktop client users (Tweetdeck, etc)? ‣ Cohort analyses ‣ What services fail at the same time? ‣ What features get users hooked? ‣ What do successful users do often? ‣ How does tweet volume change over time? Friday, July 23, 2010
  • 33. Research ‣ What can we infer from a user’s tweets? ‣ ... from the tweets of their followers? followees? ‣ What features tend to get a tweet retweeted? ‣ ... and what influences the retweet tree depth? ‣ Duplicate detection, language detection ‣ What graph structures lead to increased usage? ‣ Sentiment analysis, entity extraction ‣ User reputation Friday, July 23, 2010
  • 34. If We Had More Time... ‣ HBase ‣ LZO compression and Hadoop ‣ Protocol buffers ‣ Our open source: hadoop-lzo, elephant-bird ‣ Analytics and Cassandra Friday, July 23, 2010
  • 35. Questions? Follow me at twitter.com/kevinweil TM Friday, July 23, 2010