Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

Keynote at Accel's Datacenter 2.0 conference

Published in: Technology
  • Be the first to comment


  1. 1. Data Management at Facebook Jeff Hammerbacher Manager, Data June 12, 2008
  2. 2. My Background Thanks for Asking ▪ Studied Mathematics at Harvard ▪ Worked as a Quant on Wall Street ▪ Came to Facebook in early 2006 as a Research Scientist ▪ Now manage the Facebook Data Team ▪ 25 amazing data engineers and scientists with more on the way ▪ Skills span databases, distributed systems, statistics, machine ▪ learning, data visualization, social network analysis, and more
  3. 3. Serving Data Retrieval and Hardware GET /index.php HTTP/1.1 Host: Three main server profiles: ▪ Web ▪ Memcached ▪ Web Tier MySQL ▪ (more than 10,000 Servers) Simplified away: ▪ AJAX ▪ Memcached Tier (around 1,000 servers) MySQL Tier Photo and Video ▪ (around 2,000 servers) Services ▪
  4. 4. Serving Request Volume per Second Web 10M requests 15TB RAM Memcache 500K requests 25TB RAM MySQL
  5. 5. Services Infrastructure Thrift, Mainly Developing a Thrift service: ▪ Define your data structures ▪ JSON-like data model ▪ Define your service endpoints ▪ Select your languages ▪ Generate stub code ▪ Write service logic ▪ Write client ▪ Configure and deploy ▪ Monitor, provision, and upgrade ▪
  6. 6. Services Infrastructure What’s an SOA? Almost all services written in Thrift ▪ Networks Type-ahead, Search, Ads, SMS Gateway, Chat, Notes Import, Scribe ▪ Batteries included ▪ Network transport libraries ▪ Serialization libraries ▪ Code generation ▪ Robust server implementations (multithreaded, nonblocking, etc.) ▪ Now an Apache Incubator project ▪ For more information, read the whitepaper ▪ Related projects: Sun-RPC, CORBA, RMI, ICE, XML-RPC, JSON, Cisco’s Etch ▪
  7. 7. Data Infrastructure Offline Batch Processing Scribe Tier MySQL Tier “Data Warehousing” ▪ Began with Oracle database ▪ Schedule data collection via cron ▪ Collect data every 24 hours ▪ “ETL” scripts: hand-coded Python ▪ Data Collection Server Data volumes quickly grew ▪ Started at tens of GB in early 2006 ▪ Oracle Database Server Up to about 1 TB per day by mid-2007 ▪ Log files largest source of data growth ▪
  8. 8. Data Infrastructure Distributed Processing with Cheetah Goal: summarize log files outside of the database ▪ Solution: Cheetah, a distributed log file processing system ▪ distribute binaries to processing nodes ▪ C++ Binaries: parse, agg, load ▪ Partitioned Log File Cheetah Master Filer Processing Tier
  9. 9. Data Infrastructure Moving from Cheetah to Hadoop Cheetah limitations ▪ Limited filer bandwidth ▪ No centralized logfile metadata ▪ Writing a new Cheetah job requires writing C++ binaries ▪ Jobs are difficult to monitor and debug ▪ No support for ad hoc querying ▪ Not open source ▪
  10. 10. Data Infrastructure Hadoop as Enterprise Data Warehouse Scribe Tier MySQL Tier Hadoop Tier Oracle RAC Servers
  11. 11. Anatomy of the Facebook Cluster Hardware Individual nodes ▪ CPU: Intel Xeon dual socket quad cores (8 cores per box) ▪ Memory: 16 GB ECC DRAM ▪ Disk: 4 x 1 TB 7200 RPM SATA ▪ Network: 1 gE ▪ Topology ▪ 320 nodes arranged into 8 racks of 40 nodes each ▪ 8 x 1 Gbps links out to the core switch ▪
  12. 12. Anatomy of the Facebook Cluster Recent Cluster Statistics From May 2nd to May 21st: ▪ Total jobs: 8,794 ▪ Total map tasks: 1,362,429 ▪ Total reduce tasks: 86,806 ▪ Average duration of a successful job: 296 s ▪ Average duration of a successful map: 81 s ▪ Average duration of a successful reduce: 678 s ▪
  13. 13. Initial Hadoop Applications Hadoop Streaming Almost all applications at Facebook use Hadoop Streaming ▪ Mapper and Reducer take inputs from a pipe and write outputs to a pipe ▪ Facebook users write in Python, PHP, C++ ▪ Allows for library reuse, faster development ▪ Eats way too much CPU ▪ More info: ▪
  14. 14. Initial Hadoop Applications Unstructured text analysis Intern asked to understand brand sentiment and influence ▪ Many tools for supporting his project had to be built ▪ Understanding serialization format of wall post logs ▪ Common data operations: project, filter, join, group by ▪ Developed using Hadoop streaming for rapid prototyping in Python ▪ Scheduling regular processing and recovering from failures ▪ Making it easy to regularly load new data ▪
  15. 15. Lexicon
  16. 16. Initial Hadoop Applications Ensemble Learning Build a lot of Decision Trees and average them ▪ Random Forests are a combination of tree predictors such that each ▪ tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest Can be used for regression or classification ▪ See “Random Forests” by Leo Breiman ▪
  17. 17. More Hadoop Applications Insights Monitor performance of your Facebook Ad, Page, Application ▪ Regular aggregation of high volumes of log file data ▪ First hourly pipelines ▪ Publish data back to a MySQL tier ▪ System currently only running partially on Hadoop ▪
  18. 18. Insights
  19. 19. More Hadoop Applications Platform Application Reputation Scoring Users complaining about being spammed by Platform applications ▪ Now, every Platform Application has a set of quotas ▪ Notifications ▪ News Feed story insertion ▪ Invitations ▪ Emails ▪ Quotas determined by calculating a “reputation score” for the ▪ application
  20. 20. More Hadoop Applications Miscellaneous Experimentation Platform back end ▪ A/B Testing ▪ Champion/Challenger Testing ▪ Lots of internal analyses ▪ Export smaller data sets to R ▪ Ads: targeting optimization, fraud detection ▪ Search: index building, ranking optimization ▪ Load testing for new storage systems ▪ Language prediction for translation targeting ▪
  21. 21. Hive Structured Data Management with Hadoop Hadoop: ▪ HDFS ▪ MapReduce ▪ Resource Manager ▪ Job Scheduler ▪ Hive: ▪ Logical data partitioning ▪ Metadata store ▪ Query Operators ▪ Query Language ▪
  22. 22. Hive
  23. 23. Hive Data Model
  24. 24. Hive Sample Queries ▪ CREATE TABLE page_view(viewTime DATETIME, userid MEDIUMINT, page_url STRING, referrer_url STRING, ip STRING) COMMENT 'This is the page view table' PARTITIONED BY(date DATETIME, country STRING) BUCKETED ON (userid) INTO 32 BUCKETS ROW FORMAT DELIMITED FIELD DELIMITER 001 ROW DELIMITER 013 STORED AS COMPRESSED LOCATION '/user/facebook/warehouse/page_view'; ▪ FROM pv_users INSERT INTO TABLE pv_gender_sum SELECT pv_users.gender, count_distinct(pv_users.userid) GROUP BY(pv_users.gender) INSERT INTO FILE /user/facebook/tmp/pv_age_sum.txt SELECT pv_users.age, count_distinct(pv_users.userid) GROUP BY(pv_users.age);
  25. 25. Hive The Team Joydeep Sen Sarma ▪ Ashish Thusoo ▪ Pete Wyckoff ▪ Suresh Anthony ▪ Zheng Shao ▪ Venky Iyer ▪ Dhruba Borthakur ▪ Namit Jain ▪ Raghu Murthy ▪
  26. 26. Cassandra Structured Storage over a P2P Network Conceptually: BigTable data model on Dynamo infrastructure ▪ Design Goals: ▪ High availability ▪ Incremental scalability ▪ Eventual consistency (trade consistency for availability) ▪ Optimistic replication ▪ Low total cost of ownership ▪ Minimal administrative overhead ▪ Tunable tradeoffs between consistency, durability, and latency ▪
  27. 27. Cassandra Architecture
  28. 28. Cassandra The Team Avinash Lakshman ▪ Prashant Malik ▪ Karthik Ranganathan ▪
  29. 29. (c) 2008 Facebook, Inc. or its licensors.  quot;Facebookquot; is a registered trademark of Facebook, Inc.. All rights reserved. 1.0