LCA13: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013

1,438 views
1,201 views

Published on

Resource: LCA13
Name: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013
Date: 06-03-2013
Speaker: Jason Taylor

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,438
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
48
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

LCA13: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013

  1. 1. ARM & Disaggregated Rack: Facebook’s approach to smaller processors Jason Taylor, PhD Director, Capacity Engineering & Analysis
  2. 2. Agenda 1 Facebook Scale & Infrastructure 2 Mobile Processors 3 Disaggregated Rack
  3. 3. 82 % of users are outside of the U.S 4 domestic regions today. Europe region will come online later this year. Facebook Scale
  4. 4. Facebook Stats • 1 billion users • 350+ million photos added per day • 4.2 billion likes, posts and comments per day • 140+ billion friend connections • 240+ billion photos • 17 billion check-ins
  5. 5. Cost and Efficiency •From our 10-Q filed with the SEC in October 2012: •“The first nine months of 2012 ... $1.0 billion for capital expenditures related to the purchase of servers, networking equipment, storage infrastructure, and the construction of data centers.” •At this size, we spend a lot of time thinking about efficiency and costs.
  6. 6. Architecture Service Cluster Back-End Cluster Front-End Cluster Web 250 racks Ads 30 racks Cache (~144TB) Search Photos Msg Others UDB ADS-DB Tao Leader Multifeed 9 racks Other small services
  7. 7. Lots of “vanity free” servers.
  8. 8. Multifeed rack • The rack is our unit of capacity • All 40 servers work together • Leaf + agg code runs on all servers • Leaf has most of the the RAM • Aggregator uses most of the CPU • Lots of network BW within the rack Leaf Aggregator AL AL AL . . . .
  9. 9. Life of a “hit” Front-End Back-End Web MC MC MC MC Ads Database L Feed agg request starts Time request completes L L L L L
  10. 10. Standard Systems I Web III Database IV Hadoop V Haystack VI Feed CPU High 2 x E5-2670 Med 2 x X5650 Low 1 x L5630 High 2 x E5-2660 Memory Low 16GB High 144GB Medium 48GB Low 18GB High 144GB Disk Low 250GB High IOPS 3.2 TB Flash High 12 x 3TB SATA High 12 x 3TB SATA Medium 2TB SATA Services Web, Chat Database Hadoop Photos, Video Multifeed, Search, Ads Five Standard Servers
  11. 11. Five Server Types Advantages: • Volume pricing • Re-purposing • Easier operations - simpler repairs, drivers, DC headcount • New servers allocated in hours rather than months Drawbacks: • 40 major services; 200 minor ones - not all fit perfectly • Service needs change over time.
  12. 12. Agenda 1 Facebook Scale & Infrastructure 2 Mobile Processors 3 Disaggregated Rack
  13. 13. Server Processors • Servers in datacenters use processors that were designed for desktop computers. •Intel and AMD have dominated this market with big x86 processors.
  14. 14. Mobile Processors • Smaller processors for smart phones will pass two criteria by 2014: • 64 bit instructions • High clock speed - ~2.4 GHz •It is now reasonable to consider ARM, Atom and even MIPS processors for big compute jobs.
  15. 15. Compute Power
  16. 16. Cores Required
  17. 17. Watts Required
  18. 18. The Problem • Big processors provide a cost advantage by amortizing fixed costs in the servers. •If all other costs remain the same then wimpy cores (ARM, MIPS, Atom) will effectively triple the price of fixed resources: • Rack, chassis, disk, RAM, NIC, etc.
  19. 19. Our Solution: Group Hug •Facebook is driving a solutions through the Open Compute initiative: • Group Hug server board: • Allows up to 10 individual compute boards. • Single Processor PCIE-like cards • A 1GB interfaces mux’ed up to a 10GB NIC • No drives, flash, or prehephrials • ==> 3 to 5x the processors compared to a dual-socket system • ==> About the same throughput and power.
  20. 20. Agenda 1 Facebook Scale & Infrastructure 2 Mobile Processors 3 Disaggregated Rack
  21. 21. Disaggregated Rack Challenge • Can we build hardware that will fit more services and still do well in terms of serviceability and cost? • Can we build hardware that will grow with services over time? • What might it look like to support Group Hug?
  22. 22. Server/Service Fit - across services TYPE-6 server CPU Other Service A RAM MultiFeed CPU RAM WASTED CPU RESOURCE TYPE-6 server
  23. 23. Server/Service Fit - over time TYPE-6 server CPU Year 2 - more CPU needed RAM Year 1 CPU RAM NOT ENOUGH CPU TYPE-6 server
  24. 24. Building blocks: • CPU • RAM (key/value pairs) • Disk IOPS • Disk space • Flash IOPS • Flash space Common resource pairs: In-Rack Resources
  25. 25. Disaggregated Rack How can we build hardware that is highly configurable and re-configurable but still cost effective?
  26. 26. A rack of multifeed servers... COMPUTE RAM STORAGE Type-6 Server Network Switch Type-6 Server Type-6 Server Type-6 Server = > 40 Feed servers per rack each server with: 2 x E5-2660 144GB RAM 2TB hard drives 760GB of flash * We assume full line-rate network within the rack. 5.8 TB 80 TB . . . FLASH30 TB Type-6 Server 80 processors 640 cores
  27. 27. Compute • Standard Server • 2 processors • 8 or 16 DIMM slots • no hard drive - small flash boot partition. • big NIC - 10 Gbps or more • Group Hug • 10 individual single-proc servers • A few DIMMS • no hard drive - small flash boot partition. • smaller NICs to 10 GBps
  28. 28. Ram Sled •Hardware • 128GB to 512GB • compute: FPGA, ASIC, mobile processor or desktop processor •Performance • 450k to 1 million key/value gets/sec •Cost • Excluding RAM cost: $500 to $700 or a few dollars per GB
  29. 29. Storage Sled (Knox) •Hardware • 15 drives • Replace SAS expander w/ small server •Performance • 3k IOPS •Cost • Excluding drives: $500 to $700 or less than $0.01 per GB
  30. 30. Flash Sled •Hardware • 175GB to 18TB of flash •Performance • 600k IOPS •Cost • Excluding flash cost: $500 to $700 NIC at 70% utilization IOPS Capacity 1 Gbps 21k 175 GB 10 Gb 210k 1.75 TB 25 Gb 525k 4.4 TB 40 Gb 840k 7.7 TB 50 Gb 1.05M 8.8 TB 100 Gb 2.1M 17.5 TB
  31. 31. A disaggregated rack for graph search... Compute Network Switch Compute Storage Sled RAM Sled = > . . Flash Sled . . COMPUTE RAM STORAGE 3.1 TB 60 TB FLASH30 TB 40 processors 320 cores 20 Compute Servers 8 Flash Sleds 2 RAM Sleds 1 Storage Sled => 1:10 RAM:Flash ratio * Add 4 more flash sleds in 2014 to get to a 1:15 RAM:Flash ratio *
  32. 32. Disaggregated Rack Strengths: • Volume pricing, serviceability, etc. • Custom Configurations • Hardware evolves with service • Smarter Technology Refreshes • Speed of Innovation Potential issues: • Physical changes required • Interface overhead
  33. 33. Questions?

×