Capacity Planning

1,143 views
983 views

Published on

Deploying MongoDB can be a challenge if you don't understand how resources are used nor how to plan for the capacity of your systems. If you need to deploy, or grow, a MongoDB single instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment. This talk will cover what resources MongoDB uses, and how to plan for their use in your deployment. Topics covered will include understanding how to model and plan capacity needs from the perspective of a new deployment, growing an existing one, and defining where the steps along scalability on your path to the top. The goal of this presentation will be to provide you with the tools needed to be successful in managing your MongoDB capacity planning tasks.

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,143
On SlideShare
0
From Embeds
0
Number of Embeds
315
Actions
Shares
0
Downloads
35
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Capacity Planning

  1. 1. Capacity Planning:Deploying MongoDB
  2. 2. This deck is a work in progress!•  Written by Asya (and presented by her a few times)•  …but it’s still new, so you may want to rehearse it a few extra times before presenting
  3. 3. Capacity Planning: Why, What, When•  Dont be the "goat" – spent too much $ or caused failure of site, etc.•  Users frequently ask about HW they need for their application. What does 10gen "recommend"?•  No right answer in a vacuum.•  Why we need to plan to meet expectations, etc. –  future planning •  data increases, dont want performance drop-off
  4. 4. Capacity Planning: Why, What, When Why? What are the consequences of not planning?
  5. 5. Why•  Once we launch, we dont want to have avoidable down time due to poorly selected HW•  As our success grows we want to stay in front of the demand curve•  We want to meet business and users expectations•  We want to keep our jobs J•  and get big raises! ;)
  6. 6. Capacity Planning: Why, What, When Why? What are the consequences of not planning?
  7. 7. Why•  We want to keep our jobs J•  and get big raises! ;)•  so we should stay within reasonable budget
  8. 8. Capacity Planning: Why, What, When What? Why? Requirements
  9. 9. What•  There is one thing that is absolutely mandatory to have in order to succeed in capacity planning•  Without it, you will not be successful•  We must have REQUIREMENTS from business –  without requirements, were building a roadmap without knowing the desired destination Imagine building a car without knowing what its top speed should be, acceleration, MPH, and cost?
  10. 10. Capacity Planning: Why, What, When What? •  Availability •  Throughput •  Responsiveness
  11. 11. What•  Availability: what is uptime requirement?•  Throughput –  average read/write/users –  peak throughput? –  OPS (operations per second)? per hour? per day?•  Responsiveness –  what is acceptable latency? –  is higher during peak times acceptable?
  12. 12. Capacity Planning: Why, What, When What?•  Availability•  Throughput•  Responsiveness
  13. 13. Capacity Planning: Why, What, When When? Before its too late! Start Launch Version 2
  14. 14. Capacity Planning: Why?•  Capacity –  Under –  Over –  Just right?•  Prediction Models –  User/Load –  System(s) Behavior•  Change Velocity (reaction time) –  Data/Resource-Allocation/Provisioning
  15. 15. Capacity Planning: What?•  Understand Resources –  Storage –  Memory –  CPU –  Network •  Understand Your Application –  Monitor and Collect Metrics –  Model to Predict Change –  Allocate and Deploy –  (repeat process)
  16. 16. Resource Usage•  Storage •  CPU –  IOPS –  Speed –  Size –  Cores –  Data & Loading Patterns•  Memory •  Network –  Latency –  Working Set –  Throughput
  17. 17. Storage•  Active•  Archival•  Loading Patterns•  Integration (BI/DW)
  18. 18. Storage Example IOPS•  Active•  Archival•  Loading Patterns•  Integration (BI/DW)
  19. 19. Storage Capability Example IOPS7,200 rpm SATA ~ 75-100 IOPS15,000 rpm SAS ~ 175-210 IOPSAmazon EBS/Provisioned ~ 100 IOPS "up to" 2,000 IOPSAmazon SSD 9,000 – 120,000 IOPS
  20. 20. Storage Capability Example IOPS7,200 rpm SATA ~ 75-100 IOPS15,000 rpm SAS ~ 175-210 IOPSAmazon EBS/Provisioned ~ 100 IOPS "up to" 2,000 IOPSAmazon SSD 9,000 – 120,000 IOPSIntel X25-E (SLC) ~ 5,000 IOPSFusion IO ~ 135,000 IOPSViolin Memory 6000 ~ 1,000,000 IOPS
  21. 21. Storage Costs Cost of IOPS7,200 rpm SATA ~ 75-100 IOPS15,000 rpm SAS ~ 175-210 IOPSAmazon EBS/Provisioned ~ 100 IOPS "up to" 2,000 IOPSAmazon SSD 9,000 – 120,000 IOPSIntel X25-E (SLC) ~ 5,000 IOPSFusion IO ~ 135,000 IOPSViolin Memory 6000 ~ 1,000,000 IOPS
  22. 22. Memory•  Working Set –  Active Data in Memory –  Measured Over Periods
  23. 23. Memory•  Work: SORTS – Sorting Connections – Aggregation – Connections Aggregations
  24. 24. Memory & Storage ? > <
  25. 25. Memory & Storage MOPs PFs
  26. 26. Memory & Storage % Disk UtilMOPS: MongoDB Ops/sec MOPS
  27. 27. Memory & Storage % Disk Util MOPS
  28. 28. Memory & Storage % Disk Util MOPS
  29. 29. CPU•  Non-indexed Data•  Sorting•  Aggregation –  Map/Reduce –  Framework•  Data –  Fields –  Nesting –  Arrays/Embedded-Docs
  30. 30. Network•  Latency –  WriteConcern –  ReadPreference –  Batching –  Documents (and Collections)•  Throughput –  Update/Write Patterns –  Reads/Queries
  31. 31. Starter Questions•  What is the working set? –  How does that equate to memory –  How much disk access will that require•  How efficient are the queries?•  What is the rate of data change?•  How big are the highs and lows?
  32. 32. Deployment TypesAll of these use the same resources:•  Single Instance•  Multiple Instances (Replica Set)•  Cluster (Sharding)•  Data Centers
  33. 33. Capacity Planning: When?Monitoring§  Storage§  Memory§  CPU§  Network§  Application Metrics
  34. 34. Tools•  MMS (MongoDB Monitoring Service)•  MongoDB: mongotop, mongostat•  Linux: iostat, vmstat, sar, etc•  Windows: PerfmonMeasure realistic loads (generated by Load testing)
  35. 35. Models•  Load/Users –  Response Time/TTFB•  System Performance –  Peak Usage –  Min Usage
  36. 36. Velocity of Change•  Limitations -> takes time –  Data Movement –  Allocation/Provisioning (servers/mem/disk)•  Improvement –  Limit Size of Change (if you can) –  Increase Frequency –  MEASURE its effect –  Practice
  37. 37. Repeat (continuously)•  Repeat Testing•  Repeat Evaluations•  Repeat Deployment
  38. 38. Capacity Planning: What If...What if I skip capacity planning?You will be featured ...
  39. 39. Thank You

×