Neal Sample<br />VP Architecture, Platform Products, Developers Program<br />
eBay’s global commerce platform <br />
eBay marketplace = economy of scale <br />9<br />23<br />2<br />2Bpage views/day<br />23Mlines of code<br />9Petabytesof d...
Cloud bursting: ↓ costs by ↑ efficiency<br />2000<br />1800<br />1600<br />1400<br />1200<br />1000<br />800<br />600<br /...
Datacenter efficiency<br />TIER II<br />2(N+1)<br />TIER IV<br />N+1<br />Datacenter tier = level of redundancy<br />Blend...
Energy-proportional computing<br />Traffic<br />KWh cost / incentives / forecasts<br />Meta <br />Scheduler<br />BMS metri...
Financial model structure<br />Hourly demand(in average TPS)<br />Hourly demand(in average TPS)<br />Historical (from ODW)...
Cost components<br />Datacenter<br />People<br />Total cost per CU/h : $1.07<br />
TSP ↑ = more attractive options for cloud bursts<br />All self operated CUs<br />TPS and FE Compute Units (CU) 24-Month Pr...
Cost-benefit analysis<br />DC CUs Impact on Project Annual Cost 9/2010 – 8/2011<br />DC CU Hr Cost = 1.07, 30 TPS per CU<b...
Focus for the future<br />Business Intelligence<br /><ul><li>Commerce expertise
Customer experience
Upcoming SlideShare
Loading in …5
×

Openstack and eBay

5,584 views

Published on

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,584
On SlideShare
0
From Embeds
0
Number of Embeds
114
Actions
Shares
0
Downloads
326
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • eBay Inc. has two strong core businesses … that offer significant potential for growth and profitability. And we have a consistent track record of delivering on our commitments.We are a more customer-focused company.We are accelerating innovation across the board.And we have a unique combination of assets that position us well to capitalize aggressively on the inflection point we are seeing in the external environment.eBay/marketplaces around the world: North America: eBay.com and eBay.ca EU: eBay.co.uk eBay.de eBay.fr Etc. Asia: eBay.in eBay.au Korea: GMarketCore adjacency businesses: PayPalStubHub
  • Top 8 pools represent 1900 machines (Nov 2009). By defining a cutoff at 800 machines , it means that we would need to cloud burst 36800 VM/hours to cover the peak.And using $1/h on Azure(excluding bandwidth) = $36,800 / month1900 – 800 = 1100 machines x $1000 = $1.1M /monthImprove power efficiencyServer Consolidation (Sharing)Reduce DR capacity by implementing pre-emptionClicks 1, 2, 3: Align costs to utilizationData center efficiencyEnergy-proportional computingClick 4: “Fill out the valleys”Reuse spare capacity for experimentation, …Offer Infrastructure as a serviceClick 5: Chop off peaksClick 6: Move the peaks into the valleys (smooth out)Cloudburst (borrow capacity to address peaks)
  • Tiered approach more efficient, cost-effective, and reliable.Equate to flying airplane with 8 engines when 2 will do. Overcompensates, costs too much, reduces efficiency.Can allocate power/cost of other 6 engines to other applications at tiered levels, based on business need. Avoid spending the capital up front when you build lower-level tiers/Tier IIEvaluate systems based on level of business-critical applications/requirements.Place applications based on their resilience to failure. EXAMPLE: Search – a business-critical function – points immediately at another center if there is an issue, as opposed to a more time-consuming process to migrate a DB from one location to another. The search nodes can be in Tier II, the DB needs to stay in Tier IV (more sensitive to failures). We believe that 70%+ of our applications can reside in Tier II (half the capex, half the opex). Today, all of them live in Tier IV.  Assign top tier operations – redundancy, cost, reliability – to mission-critical applications.Assign lower-level operational structure to less-critical applications.
  • Future state : based on utility cost, datacenter and infrastructure metrics, traffic will be directed to the most economical location while achieving SLA. consider utility companies incentives to shift traffic and power down or idle down infrastructure.
  • VISUALLY CONSISTENT – UPDATE LABELS – ADD AXESBalance: all in-house; all outsourceInflection points to choose some distributionLines rep various choice; dots rep various modelseBay’s method: Adopt an hybrid cloud approach (part on premise, part off premise) Use the on premise infrastructure for core business critical applications and infrastructure. Convert site to a cloud infrastructure to enable agility and increase average utilization Use the off premise cloud to offload capacity for peak demand, and new/experimental projects. Reduce on premise footprint.Anticipating double-digit transaction per second (TPS) increases YoY.Clarify TPS definition: Combination of internal and external factors, including processing, analytics, as well as site traffic and commerce transactions.As TPS increases, need options for addressing cloud bursting:Advantage: Adjust computing power to applications’ varying demandEnables higher resource utilization due to statistical multiplexing of loadEnables short time to market for new app to capture opportunities when they are identifiedGreen line: Servers operated by eBay Estimated ~ 5228 servers / 13130 CU’s Variety of HW types and virtualization status (Nov 2010)Cost based on 2 yr leaseComplete control; tried and known environmentParameters: Server cost per hourChange in the number of servers over timeBlue line: Internal cloud: Resources dedicated to a single operator who incurs constant cost per server regardless of actual loadTan line: External cloud: use and pay for server only when neededUsually pay by hour of reserved operationLight blue line: Hybrid cloud: A combination of internal and external clouds (part on-premise; part off-premise)Use the on premise infrastructure for core business critical applications and infrastructure.Convert site to a cloud infrastructure to enable agility and increase average utilizationUse the off premise cloud to offload capacity for peak demand, and new/experimental projects. Reduce on premise footprint.Internal cloud “stamps” 2 yr lease. Fixed cost independent of utilizationQuickly deploy and change appsShare load dynamically with self-operated serversParameters: number of stamps and installation/operation datesExternal cloud Paid for by the hour.Quickly turned on to carry traffic when exceeds capacity of DC-based serversParameters: hourly cost of server unit
  • As infrastructure is commoditized, eBay can decrease cloud costs as much as possible by working with external industry leaders.Allows eBay to focus on strengths of running world’s largest online marketplace: Business intelligence Commerce expertise Customer focus Reliability Security Global presence
  • Openstack and eBay

    1. 1. Neal Sample<br />VP Architecture, Platform Products, Developers Program<br />
    2. 2. eBay’s global commerce platform <br />
    3. 3. eBay marketplace = economy of scale <br />9<br />23<br />2<br />2Bpage views/day<br />23Mlines of code<br />9Petabytesof data storage<br />94<br />200<br />$62<br />94M active users<br />200Mlive listings<br />$62B2010 gross merchandise volume<br />6,000<br />75<br />6,000 application servers<br />75B database calls/day<br />Commercial data warehouse 100x larger than Library of Congress<br />
    4. 4. Cloud bursting: ↓ costs by ↑ efficiency<br />2000<br />1800<br />1600<br />1400<br />1200<br />1000<br />800<br />600<br />400<br />200<br />0<br />Fixed Cost<br />Datacenter efficiency<br />Cloud Burst<br />Energy-proportional computing<br />Excess Capacity<br />Ideal<br />Remove the peaks<br />Request for Data<br />Current Traffic<br />text<br />text<br />text<br />text<br />text<br />text<br />text<br />text<br />Timeline<br />
    5. 5. Datacenter efficiency<br />TIER II<br />2(N+1)<br />TIER IV<br />N+1<br />Datacenter tier = level of redundancy<br />Blend of Tier IV & Tier II<br />Minimum 30% reduction in OpEx cost per MW enables cost effective operations/expansion.<br />50% reduction in CapEx for new lower-tier datacenters (Tier IV vs. Tier II)<br />Burst Capacity (DC and Cloud)<br />
    6. 6. Energy-proportional computing<br />Traffic<br />KWh cost / incentives / forecasts<br />Meta <br />Scheduler<br />BMS metrics<br />SLA<br />Models<br />Utilization (headroom)<br />Availability<br />Cloud <br />Bursting<br />SLC<br />PHX (Tier 4)<br />PHX (Tier2)<br />Direct traffic to the most economical location / optimize placement within DC<br />
    7. 7. Financial model structure<br />Hourly demand(in average TPS)<br />Hourly demand(in average TPS)<br />Historical (from ODW) or predicted<br />Historical (from ODW) or predicted<br />Define a standard CU<br />Define a standard CU<br />Hourly computing units (CU) required<br />Max TPS per CU<br />Hourly computing units (CU) required<br />Max TPS per CU<br />Based on server inventoryand growth forecast<br />Based on server inventoryand growth forecast<br />Hourly CUs neededin external cloud<br />Number of CU in internal cloud<br />Hourly CUs neededin external cloud<br />Number of CU in internal cloud<br />Negotiated price with public cloud vendor<br />Negotiated price with public cloud vendor<br />Hourly cost of CUin external cloud<br />Hourly total cost<br />Hourly cost of CU in internal cloud<br />Hourly cost of CUin external cloud<br />Hourly total cost<br />Hourly cost of CU in internal cloud<br />HW/SW lease, networking, space, power, laborPredict future costs trends<br />HW/SW lease, networking, space, power, laborPredict future costs trends<br />Sum cost over period<br />Sum cost over period<br />Evaluate tradeoffs by varying size of internal cloud, and costs<br />Evaluate tradeoffs by varying size of internal cloud, and costs<br />
    8. 8. Cost components<br />Datacenter<br />People<br />Total cost per CU/h : $1.07<br />
    9. 9. TSP ↑ = more attractive options for cloud bursts<br />All self operated CUs<br />TPS and FE Compute Units (CU) 24-Month Projects<br />Load served by external cloud<br />Historical TPS hourly data from ODW<br />CU decreasing by<br />Lease expiration<br />Load served by self operated CU <br />
    10. 10. Cost-benefit analysis<br />DC CUs Impact on Project Annual Cost 9/2010 – 8/2011<br />DC CU Hr Cost = 1.07, 30 TPS per CU<br />60,000,000<br />Cloud cost ($/hour)<br />Total Cost ($)<br />0.5<br />0.8<br />1<br />40,000,000<br />1.1<br />1.2<br />1.5<br />2<br />3<br />4<br />20,000,000<br />0 DCs<br />$20M<br />0 DCs<br />$30M<br />3K DCs<br />$40M<br />3.5K DCs<br />$45M<br />4K DCs<br />$47M<br />4.8K DCs<br />$52M<br />5K DCs<br />$55M<br />0<br />2000<br />4000<br />6000<br />8000<br />DC CUs<br />
    11. 11. Focus for the future<br />Business Intelligence<br /><ul><li>Commerce expertise
    12. 12. Customer experience
    13. 13. Reliability
    14. 14. Security
    15. 15. Etc.</li></ul>Business Intelligence<br />Increased Focus<br />Infrastructureas Commodity<br />Infrastructureas Commodity<br />LowerCost<br />
    16. 16. Financial Cost Model for the Hybrid Cloud<br />

    ×