Datacenter@Night: How Big Data Technologies Power Facebook

1,071 views

Published on

Karthik Ranganathan, ehemaliger Lead-Ingenieur bei Facebook, und heute bei NUTANIX beschäftigt erklärt in seiner Präsentation moderne Datencenter anhand des Business Use Case Facebook.

Published in: Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,071
On SlideShare
0
From Embeds
0
Number of Embeds
242
Actions
Shares
0
Downloads
45
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Datacenter@Night: How Big Data Technologies Power Facebook

  1. 1. 1 Kannan Muthukkaruppan & Karthik Ranganathan Jun/20/2013 How Big Data Technologies Power Facebook How Big Data Technologies Power Facebook Karthik Ranganathan September, 2013
  2. 2. 2 Introduction Email: karthik@nutanix.com Twitter: @KarthikR Current: Member of Technical Staff, Nutanix Background: Technical Engineering Lead at Facebook. Co- built Cassandra for Facebook Inbox Search and improved performance and resiliency of Hbase for Facebook Messages and Search Indexing.
  3. 3. 3 Agenda  Big data at Facebook  HBase use cases • OLTP • Analytics  Operating at scale  The Nutanix solution
  4. 4. 4 Big Data at Facebook  OLTP • User databases (MySQL) • Photos (Haystack) • Facebook Messages, Operational Data Store (HBase)  Warehouse • Hive Analytics • Graph Search Indexing
  5. 5. 5 HBase in a nutshell  Apache project, modeled after BigTable  Distributed, large scale data store  Built on top of Hadoop DFS (HDFS)  Efficient at random reads and writes
  6. 6. 6 FB’s Largest Hbase Application Facebook Messages
  7. 7. 7 The New Facebook Messages
  8. 8. 8 Why HBase?  Evaluated a bunch of different options • MySQL, Cassandra, building a custom storage system for messages  Horizontal Scalability  Automatic failover and load balancing  Optimized for write-heavy workloads  HDFS already battle-tested at Facebook  HBase’s strong consistency model
  9. 9. 9 Quick stats (as of Nov 2011)  Traffic to HBase • Billions of messages per day • 75B+ rpc’s per day  Usage pattern • 55% reads, 45% writes • Average write: 16 KV’s to multiple CF’s
  10. 10. 10 Data Sizes  7PB+ online data • ~21PB with replication • LZO compressed • Excludes backups  Growth rate • 500TB+ per month • ~20PB of raw disk per year!
  11. 11. 11 Growing with size  Constant need of features with growth  Read and write path improvements • Performance optimizations • IOPS reduction • New database file format  Intelligent data and compute placement • Shard level block placement • Locality based load-balancing
  12. 12. 12 Other OLTP use cases of HBase  Operational Data Store  Multi-tenant KeyValue store  Site integrity – fighting spam
  13. 13. 13 Warehouse use cases of HBase  Graph Search Indexing • Complex application logic • Multiple verticals  Hive over HBase • Realtime data ingest • Enables real-time analytics
  14. 14. 14 Real-time monitoring and anomaly detection Operational Data Store
  15. 15. 15 ODS: Facebook’s #1 Debugging Tool  Collects metrics from production servers  Supports complex aggregations and transformations  Really well-designed UI
  16. 16. 16 Quick stats  Traffic to HBase • 150B+ ops per day  Usage pattern • Heavy reads of recent data • Frequent MR jobs for rollups • TTL to expire older data
  17. 17. 17 Real-time Analytics Facebook Insights
  18. 18. 18 Real-time URL/Domain Insights  Deep analytics for websites • Facebook widgets  Massive scale • Billions of URL’s • Millions of increments/sec
  19. 19. 19 Detailed Insights  Tracks many metrics • Clicks, likes, shares, impr essions • Referral traffic  Detailed breakdown • Age buckets, gender, location
  20. 20. 20 Controlled Multi-tenancy Generic KeyValue Store
  21. 21. 21 A Multi-tenant solution on HBase  Generic Key-Value store • Multiple apps on the same cluster • Transparent schema design • Simple API put(appid, key, value) value = get(appid, key)
  22. 22. 22 Architecture HBase put(appid, key, value) Memcache get(appid, key) Read Write
  23. 23. 23 Multi-tenancy Issues  Not a self-service model • Each app is reviewed  Global and per-app metrics • Monitor RPCs by type, latencies, errors • Friendly names for apps  If things went wrong • Per-app kill switch
  24. 24. 24 Powering FB’s Semantic Search Engine Graph Search Indexing
  25. 25. 25 Framework to build search indexes  Multiple, independent input sources  HBase stores document info  Output is the search index image rowKey = document id value = terms, document data
  26. 26. 26 Architecture HBase cluster Document source 2 Document source 1 MR cluster … Image files …
  27. 27. 27 Do’s and Do-Not’s From Experience Operating at Scale
  28. 28. 28 Design for failures(!)  Architect for failures and manageability  No single point of failure • Killing any process is legit  Minimize manual intervention • Especially for frequent failures  Uptime is important • Rolling upgrades are the norm • Need to survive rack failures
  29. 29. 29 Dashboard and Metrics  Single place to graph/report everything  RPC calls  SLA misses • Latencies, p99, Errors • Per-request profiling  Cluster and node health  Network Utilization
  30. 30. 30 Health Checks  Constantly monitor nodes  Auto-exclude nodes on failure • Machine not ssh-able • Hardware failures (HDD failure, etc) • Do NOT exclude on rack failures  Auto-include nodes once repaired  Rate limit remediation of nodes
  31. 31. 31 In a nutshell…  Use commodity hardware  Scaling out is #1  Efficiency is #2 • though pretty close behind scale-out  Design for failures • Frequent failures must be auto handled  Metrics, Metrics, Metrics!
  32. 32. 32 Overview through comparison The Nutanix Solution
  33. 33. 33 Nutanix compared with HBase  Evaluated a bunch of different options • MySQL, Cassandra, building a custom storage system for messages  Horizontal Scalability  Just add more nodes to scale out  Automatic failover and load balancing  When a node goes down, others take its place automatically  Load of node that went down is distributed to many others
  34. 34. 34 Nutanix compared with HBase philosophy  Optimized for write-heavy workloads  Optimized for virtualized environments  Read and write heavy workloads  Transparent use of flash to boost perf  HDFS already battle-tested at Facebook  Nutanix is also quite battle-tested  HBase’s strong consistency model  Nutanix is also strongly consistent
  35. 35. 35 Other aspects of Nutanix  Architected for failures and manageability  No single point of failure  Minimal manual intervention for frequent failures  Uptime is important  Rolling upgrades are the norm • Need to survive rack failures  Single place to graph/report everything  Prism UI to report and manage the entire cluster  Constantly monitor nodes  Auto-exclude nodes on failure
  36. 36. 36 In a nutshell about Nutanix…  Runs on commodity hardware  Scaling out is #1  Drop in scale out for nodes  Efficiency is #2  Constant work on perf improvements  Design for failures  Frequent failures auto handled  Alerts in UI for many other states  Metrics, Metrics, Metrics!  Prism UI gives insights into the cluster health
  37. 37. 37 Questions?
  38. 38. 38NUTANIX INC. – CONFIDENTIAL AND PROPRIETARY

×