• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
LCA13: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013
 

LCA13: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013

on

  • 266 views

Resource: LCA13

Resource: LCA13
Name: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013
Date: 06-03-2013
Speaker: Jason Taylor

Statistics

Views

Total Views
266
Views on SlideShare
266
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    LCA13: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013 LCA13: Jason Taylor Keynote - ARM & Disaggregated Rack - LCA13-Hong - 6 March 2013 Presentation Transcript

    • ARM & Disaggregated Rack: Facebook’s approach to smaller processors Jason Taylor, PhD Director, Capacity Engineering & Analysis
    • Agenda 1 Facebook Scale & Infrastructure 2 Mobile Processors 3 Disaggregated Rack
    • 82 % of users are outside of the U.S 4 domestic regions today. Europe region will come online later this year. Facebook Scale
    • Facebook Stats • 1 billion users • 350+ million photos added per day • 4.2 billion likes, posts and comments per day • 140+ billion friend connections • 240+ billion photos • 17 billion check-ins
    • Cost and Efficiency •From our 10-Q filed with the SEC in October 2012: •“The first nine months of 2012 ... $1.0 billion for capital expenditures related to the purchase of servers, networking equipment, storage infrastructure, and the construction of data centers.” •At this size, we spend a lot of time thinking about efficiency and costs.
    • Architecture Service Cluster Back-End Cluster Front-End Cluster Web 250 racks Ads 30 racks Cache (~144TB) Search Photos Msg Others UDB ADS-DB Tao Leader Multifeed 9 racks Other small services
    • Lots of “vanity free” servers.
    • Multifeed rack • The rack is our unit of capacity • All 40 servers work together • Leaf + agg code runs on all servers • Leaf has most of the the RAM • Aggregator uses most of the CPU • Lots of network BW within the rack Leaf Aggregator AL AL AL . . . .
    • Life of a “hit” Front-End Back-End Web MC MC MC MC Ads Database L Feed agg request starts Time request completes L L L L L
    • Standard Systems I Web III Database IV Hadoop V Haystack VI Feed CPU High 2 x E5-2670 Med 2 x X5650 Low 1 x L5630 High 2 x E5-2660 Memory Low 16GB High 144GB Medium 48GB Low 18GB High 144GB Disk Low 250GB High IOPS 3.2 TB Flash High 12 x 3TB SATA High 12 x 3TB SATA Medium 2TB SATA Services Web, Chat Database Hadoop Photos, Video Multifeed, Search, Ads Five Standard Servers
    • Five Server Types Advantages: • Volume pricing • Re-purposing • Easier operations - simpler repairs, drivers, DC headcount • New servers allocated in hours rather than months Drawbacks: • 40 major services; 200 minor ones - not all fit perfectly • Service needs change over time.
    • Agenda 1 Facebook Scale & Infrastructure 2 Mobile Processors 3 Disaggregated Rack
    • Server Processors • Servers in datacenters use processors that were designed for desktop computers. •Intel and AMD have dominated this market with big x86 processors.
    • Mobile Processors • Smaller processors for smart phones will pass two criteria by 2014: • 64 bit instructions • High clock speed - ~2.4 GHz •It is now reasonable to consider ARM, Atom and even MIPS processors for big compute jobs.
    • Compute Power
    • Cores Required
    • Watts Required
    • The Problem • Big processors provide a cost advantage by amortizing fixed costs in the servers. •If all other costs remain the same then wimpy cores (ARM, MIPS, Atom) will effectively triple the price of fixed resources: • Rack, chassis, disk, RAM, NIC, etc.
    • Our Solution: Group Hug •Facebook is driving a solutions through the Open Compute initiative: • Group Hug server board: • Allows up to 10 individual compute boards. • Single Processor PCIE-like cards • A 1GB interfaces mux’ed up to a 10GB NIC • No drives, flash, or prehephrials • ==> 3 to 5x the processors compared to a dual-socket system • ==> About the same throughput and power.
    • Agenda 1 Facebook Scale & Infrastructure 2 Mobile Processors 3 Disaggregated Rack
    • Disaggregated Rack Challenge • Can we build hardware that will fit more services and still do well in terms of serviceability and cost? • Can we build hardware that will grow with services over time? • What might it look like to support Group Hug?
    • Server/Service Fit - across services TYPE-6 server CPU Other Service A RAM MultiFeed CPU RAM WASTED CPU RESOURCE TYPE-6 server
    • Server/Service Fit - over time TYPE-6 server CPU Year 2 - more CPU needed RAM Year 1 CPU RAM NOT ENOUGH CPU TYPE-6 server
    • Building blocks: • CPU • RAM (key/value pairs) • Disk IOPS • Disk space • Flash IOPS • Flash space Common resource pairs: In-Rack Resources
    • Disaggregated Rack How can we build hardware that is highly configurable and re-configurable but still cost effective?
    • A rack of multifeed servers... COMPUTE RAM STORAGE Type-6 Server Network Switch Type-6 Server Type-6 Server Type-6 Server = > 40 Feed servers per rack each server with: 2 x E5-2660 144GB RAM 2TB hard drives 760GB of flash * We assume full line-rate network within the rack. 5.8 TB 80 TB . . . FLASH30 TB Type-6 Server 80 processors 640 cores
    • Compute • Standard Server • 2 processors • 8 or 16 DIMM slots • no hard drive - small flash boot partition. • big NIC - 10 Gbps or more • Group Hug • 10 individual single-proc servers • A few DIMMS • no hard drive - small flash boot partition. • smaller NICs to 10 GBps
    • Ram Sled •Hardware • 128GB to 512GB • compute: FPGA, ASIC, mobile processor or desktop processor •Performance • 450k to 1 million key/value gets/sec •Cost • Excluding RAM cost: $500 to $700 or a few dollars per GB
    • Storage Sled (Knox) •Hardware • 15 drives • Replace SAS expander w/ small server •Performance • 3k IOPS •Cost • Excluding drives: $500 to $700 or less than $0.01 per GB
    • Flash Sled •Hardware • 175GB to 18TB of flash •Performance • 600k IOPS •Cost • Excluding flash cost: $500 to $700 NIC at 70% utilization IOPS Capacity 1 Gbps 21k 175 GB 10 Gb 210k 1.75 TB 25 Gb 525k 4.4 TB 40 Gb 840k 7.7 TB 50 Gb 1.05M 8.8 TB 100 Gb 2.1M 17.5 TB
    • A disaggregated rack for graph search... Compute Network Switch Compute Storage Sled RAM Sled = > . . Flash Sled . . COMPUTE RAM STORAGE 3.1 TB 60 TB FLASH30 TB 40 processors 320 cores 20 Compute Servers 8 Flash Sleds 2 RAM Sleds 1 Storage Sled => 1:10 RAM:Flash ratio * Add 4 more flash sleds in 2014 to get to a 1:15 RAM:Flash ratio *
    • Disaggregated Rack Strengths: • Volume pricing, serviceability, etc. • Custom Configurations • Hardware evolves with service • Smarter Technology Refreshes • Speed of Innovation Potential issues: • Physical changes required • Interface overhead
    • Questions?