Hadoop, Pig, and Twitter (NoSQL East 2009)

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

9 comments

Comments 1 - 9 of 9 previous next Post a comment

  • + satishgaude satishgaude 3 days ago
    That was great. Cleared some of my doubts
  • + kevinweil Kevin Weil 2 months ago
    Hitesh,

    We don’t read or write data directly from/to MySQL with Pig. The interfaces for load/store aren’t quite right yet, although they’re getting better (follow along in realtime at https://issues.apache.org/jira/browse/PIG-966). Rather, we export in batch from mysql into hdfs at regular intervals, which is good because then non-pig jobs can access this data too. And we have a little wrapper around pig that can optionally push the results of pig jobs back into databases etc after the job has run. This is something we plan on open sourcing once it hardens a bit.
  • + guest15726d Hitesh 2 months ago
    Kevin, great read. Thanks a lot for sharing. How does Pig interface with the MySQL based environment? Is it through flat file exports data dump or realtime? How often do you that process?
  • + kevinweil Kevin Weil 2 months ago
    Matt, we use a lot of Thrift for interop between services in the online part of the site, and we make heavy use of protobufs as an on-disk efficient, versioned storage format in HDFS. Protobufs are great as a serialization format because you never have to write parsing code again, yet you retain the ability to modify your schema, add fields, etc. The protobuf generated code (at least in Java/C++/Python) is great too, with codegen’d reflection interfaces that help you write generic code.
  • + shino Shunichi Shinohara 3 months ago
    It has real Pig script, where 'co-group' is used. Cool.
  • + kcmani Mani Kumar 3 months ago
    kevin,

    well said. apart from that Hive still need love of open source community until then its not stable to be used in production. pig seems to be stable and being used by other people production as well.

    thanks!
    mani
  • + kevinweil Kevin Weil 3 months ago
    Mani, my personal preference is Pig because it allows analysis to be a little more free form. UDFs are more of a first class citizen, there’s no prescription for how data is stored, etc. Hive is flexible, but requires setup. I like the idea of being able to analyze any data sitting in our cluster without first having to wrap a create table statetement around it. Also, I tried to write a custom serde for protobuf once, and found the docs extremely lacking, which was worrisome. Doing the equivalent in Pig took less than a day.

    That said, Hive is a great tool, and if it fits your needs, that’s all that matters.
  • + kcmani Mani Kumar 3 months ago
    hi kevin,

    all well but just one question: why you guys choose Pig over Hive? Hive is more like a data warehouse and you can run ad-hoc sql queries. (so not much of a learning curve).

    thanks for sharing!

    mani kumar
  • + mokeefe Matt O'Keefe 3 months ago
    Thanks for posting. How widespread is the use of protobuf at Twitter?
Post a comment
Embed Video
Edit your comment Cancel

91 Favorites

Hadoop, Pig, and Twitter (NoSQL East 2009) - Presentation Transcript

  1. Hadoop, Pig, and Twitter Kevin Weil -- @kevinweil Analytics Lead, Twitter TM
  2. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
  3. My Background ‣ Studied Mathematics and Physics at Harvard, Physics at Stanford ‣ Tropos Networks (city-wide wireless): mesh routing algorithms, GBs of data ‣ Cooliris (web media): Hadoop and Pig for analytics, TBs of data ‣ Twitter: Hadoop, Pig, machine learning, visualization, social graph analysis, ??? of data
  4. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
  5. Data is Getting Big ‣ NYSE: 1 TB/day ‣ Facebook: 20+ TB compressed/day ‣ CERN/LHC: 40 TB/day (15 PB/year!) ‣ And growth is accelerating ‣ Need multiple machines, horizontal scalability
  6. Hadoop ‣ Distributed file system (hard to store a PB) ‣ Fault-tolerant, handles replication, node failure, etc ‣ MapReduce-based parallel computation (even harder to process a PB) ‣ Generic key-value based computation interface allows for wide applicability ‣ Open source, top-level Apache project ‣ Scalable: Y! has a 4000-node cluster ‣ Powerful: sorted a TB of random integers in 62 seconds
  7. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  8. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  9. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  10. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  11. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  12. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  13. MapReduce? ‣ Challenge: how many tweets per user, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=user_id, value=1 ‣ Shuffle: sort by user_id ‣ Reduce: for each user_id, sum ‣ Output: user_id, tweet count ‣ With 2x machines, runs close to 2x faster.
  14. But... ‣ Analysis typically done in Java ‣ Single-input, two-stage data flow is rigid ‣ Projections, filters: custom code ‣ Joins: lengthy, error-prone ‣ n-stage jobs: Hard to manage ‣ Prototyping/exploration requires ‣ analytics in Eclipse? compilation ur doin it wrong...
  15. Enter Pig ‣ High level language ‣ Transformations on sets of records ‣ Process data one step at a time ‣ Easier than SQL?
  16. Why Pig? ‣ Because I bet you can read the following script.
  17. A Real Pig Script ‣ Now, just for fun... the same calculation in vanilla Hadoop MapReduce.
  18. No, seriously.
  19. Pig Democratizes Large-scale Data Analysis ‣ The Pig version is: ‣ 5% of the code ‣ 5% of the time ‣ Within 50% of the execution time. ‣ Innovation increasingly driven from large-scale data analysis ‣ Need fast iteration to understand the right questions ‣ More minds contributing = more value from your data
  20. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
  21. MySQL, MySQL, MySQL ‣ We all start there. ‣ But MySQL is not built for analysis. ‣ select count(*) from users? Maybe. ‣ select count(*) from tweets? Uh... ‣ Imagine joining them. ‣ And grouping. ‣ Then sorting.
  22. The Hadoop Ecosystem at Twitter ‣ Cloudera’s free distribution, running Hadoop 0.20.1 ‣ Heavily modified Facebook Scribe for log collection -> HDFS* ‣ Heavily modified LZO code for fast, splittable data compression** ‣ Data stored either as LZO-compressed flat files (logs, etc) or serialized, LZO-compressed protocol buffers (structured data). ‣ Custom InputFormats, Pig LoadFuncs for the above* ‣ Some Java-based MapReduce, some Hadoop Streaming ‣ Most analysis, and most interesting analyses, done in Pig. ‣ * Open sourced, or on the way. Please come talk afterwards if you’re interested. ‣ ** http://www.github.com/kevinweil/hadoop-lzo
  23. Data? ‣ Semi-structured: apache logs, search logs, RoR logs, mysql query logs, rate limiter logs, per-application logs ‣ Structured: tweets, users, block notifications, phones, favorites, saved searches, retweets, authentications, sms usage, third party clients, followings ‣ Entangled: the social graph
  24. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
  25. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣
  26. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣
  27. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣
  28. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣ How many searches happen each day on Twitter? ‣
  29. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣ How many searches happen each day on Twitter? ‣ How many unique queries, how many unique users? ‣
  30. Counting Big Data ‣ standard counts, min, max, std dev ‣ How many requests do we serve in a day? ‣ What is the average latency? 95% latency? ‣ Group by response code. What is the hourly distribution? ‣ How many searches happen each day on Twitter? ‣ How many unique queries, how many unique users? ‣ What is their geographic distribution?
  31. Counting Big Data ‣ Where are users querying from? The API, the front page, their profile page, etc? ‣
  32. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users?
  33. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients?
  34. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses
  35. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time?
  36. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked?
  37. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked? ‣ Which features do successful users use often?
  38. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked? ‣ Which features do successful users use often? ‣ Search corrections, search suggestions
  39. Correlating Big Data ‣ probabilities, covariance, influence ‣ How does usage differ for mobile users? ‣ How about for users with 3rd party desktop clients? ‣ Cohort analyses ‣ Site problems: what goes wrong at the same time? ‣ Which features get users hooked? ‣ Which features do successful users use often? ‣ Search corrections, search suggestions ‣ A/B testing
  40. Correlating Big Data ‣ What is the correlation between users with registered phones and users that tweet?
  41. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets?
  42. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow?
  43. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers?
  44. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers? ‣ From the ratio of followers/following?
  45. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers? ‣ From the ratio of followers/following? ‣ What graph structures lead to successful networks?
  46. Research on Big Data ‣ prediction, graph analysis, natural language ‣ What can we tell about a user from their tweets? ‣ From the tweets of those they follow? ‣ From the tweets of their followers? ‣ From the ratio of followers/following? ‣ What graph structures lead to successful networks? ‣ User reputation
  47. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis
  48. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted?
  49. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree?
  50. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection
  51. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection ‣ Machine learning
  52. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection ‣ Machine learning ‣ Language detection
  53. Research on Big Data ‣ prediction, graph analysis, natural language ‣ Sentiment analysis ‣ What features get a tweet retweeted? ‣ How deep is the corresponding retweet tree? ‣ Long-term duplicate detection ‣ Machine learning ‣ Language detection ‣ ... the list goes on.
  54. Research on Big Data ‣ How well can we detect bots and other non-human tweeters?
  55. Introduction ‣ Hadoop Overview ‣ Why Pig? ‣ Evolution of Data Processing at Twitter ‣ Pig for Counting ‣ Pig for Correlating ‣ Pig for Research and Data Mining ‣ Conclusions and Next Steps
  56. Why Hadoop? ‣ Data is growing rapidly; need horizontally scalable computation ‣ Fault tolerant; gracefully handles machine failure ‣ ... but writing MapReduce jobs in Java is harder than it should be.
  57. Why Pig? ‣ Pig makes Hadoop accessible ‣ Pig chains together complex job flows ‣ User-defined functions are first class citizens ‣ Vibrant OS community, dedicated team at Y! improving it daily ‣ At Twitter, Pig helps us understand our business faster.
  58. Questions? Follow me at twitter.com/kevinweil ‣ If this sounded interesting to you -- that’s because it is. And we’re hiring. TM

+ Kevin WeilKevin Weil, 3 months ago

custom

14005 views, 91 favs, 4 embeds more stats

A talk on the use of Hadoop and Pig inside Twitter, more

More info about this presentation

© All Rights Reserved

  • Total Views 14005
    • 13966 on SlideShare
    • 39 from embeds
  • Comments 9
  • Favorites 91
  • Downloads 412
Most viewed embeds
  • 14 views on http://www.adambreckler.com
  • 11 views on http://juepner-bpm.com
  • 7 views on http://www.iweb34.com
  • 7 views on http://marcelmaatkamp.blogspot.com

more

All embeds
  • 14 views on http://www.adambreckler.com
  • 11 views on http://juepner-bpm.com
  • 7 views on http://www.iweb34.com
  • 7 views on http://marcelmaatkamp.blogspot.com

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories