0
NoSQL Infrastructure
~3333 write ops/s 0.07 - 0.05 ms response
David Mytton
Woop Japan!
MongoDB at Server Density
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs
•27 nodes
MongoDB at Server Density
•MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs
•MySQL -> MongoDB
•25TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs
Why?
Why?
• Replication
Why?
• Replication
• Official drivers
Why?
• Replication
• Official drivers
• Easy deployment
Why?
• Replication
• Official drivers
• Easy deployment
• Fast out of the box
It’s a little different.
Picture is unrelated! Mmm, ice cream.
• Fast network
Performance
• Fast network
Performance
EC2 10 Gigabit Ethernet
- Cluster Compute
- High Memory Cluster
- Cluster GPU
- High I/O
- High...
• Fast network
Performance
Workload: Read/Write?
Result set size
What is being stored?
• Fast network
Performance
Use Network Throughput
Normal 0-100Mb/s
Replication (Initial Sync) Burst +100Mb/s
Replication (...
• Fast network
Performance
Inter-DC LAN
• Fast network
Performance
Inter-DC LAN
Cross USA Washington, DC - San Jose, CA
• Fast network
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe - J...
Failover
•Replica sets
Failover
•Replica sets
•Master/slave
Failover
•Replica sets
•Min 3 nodes
•Master/slave
Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover
•Replication lag
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Pacific 150ms
Europe -...
Replication Lag
1. Reads: eventual consistency
Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind
Eventual Consistency
Stale data
Eventual Consistency
Stale data
Inconsistent data
Eventual Consistency
Stale data
Inconsistent data
Changing data
Eventual Consistency
Use Case Needs consistency?
Graphs No
User profile Yes
Statistics Depends
Alert config Yes
Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind
Slave behind
Failover: out of date master
• Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Pri...
Tags
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}}...
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_i...
{
_id : "someSet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_i...
Picture is unrelated! Mmm, ice cream.
• Fast network
•More RAM
Performance
http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
More RAM = expensive
Performance
x2 4GB RAM 12 month Prices
RAM
SSDs
Spinning disk
Cost Speed
Softlayer disk pricing
Performance
EC2 disk/RAM pricing
Performance
$2232/m
$2520/m
$43/m
$295/m
SSD vs Spinning
Performance
SSD vs Spinning
Performance
SSD vs Spinning
Performance
Tips: rand()
•Field names
Tips: rand()
•Field names
•Covered indexes
Tips: rand()
•Field names
•Covered indexes
•Collections / databases
Backups
What is the use case?
Backups
What is the use case?
Fixing user errors?
Point in time restore?
Disaster recovery?
Backups
•Disaster recovery
Offsite
Backups
•Disaster recovery
Age
Offsite
Backups
•Disaster recovery
Age
Offsite
Restore time
Backups
Frequency
Consistency
Verification
www.flickr.com/photos/daddo83/3406962115/
Monitoring
•System
Disk i/o
Disk use
www.flickr.com/photos/daddo83/3406962115/
Monitoring
Disk i/o
Disk use
•System
Swap
www.flickr.com/photos/daddo83/3406962115/
Monitoring
Optime
State
•Replication
mongostat
Monitoring tools
Run yourself
Ganglia
Monitoring tools
www.serverdensity.com
David Mytton
david@serverdensity.com
@davidmytton
Woop Japan!
blog.serverdensity.com/mongodb
NoSQL Infrastructure - Late 2013
NoSQL Infrastructure - Late 2013
Upcoming SlideShare
Loading in...5
×

NoSQL Infrastructure - Late 2013

1,000

Published on

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment.

This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,000
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "NoSQL Infrastructure - Late 2013"

  1. 1. NoSQL Infrastructure
  2. 2. ~3333 write ops/s 0.07 - 0.05 ms response
  3. 3. David Mytton Woop Japan!
  4. 4. MongoDB at Server Density
  5. 5. •27 nodes MongoDB at Server Density
  6. 6. • June 2009 - 4yrs •27 nodes MongoDB at Server Density
  7. 7. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - 4yrs
  8. 8. •MySQL -> MongoDB •25TB data per month •27 nodes MongoDB at Server Density • June 2009 - 4yrs
  9. 9. Why?
  10. 10. Why? • Replication
  11. 11. Why? • Replication • Official drivers
  12. 12. Why? • Replication • Official drivers • Easy deployment
  13. 13. Why? • Replication • Official drivers • Easy deployment • Fast out of the box
  14. 14. It’s a little different.
  15. 15. Picture is unrelated! Mmm, ice cream. • Fast network Performance
  16. 16. • Fast network Performance EC2 10 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage
  17. 17. • Fast network Performance Workload: Read/Write? Result set size What is being stored?
  18. 18. • Fast network Performance Use Network Throughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog
  19. 19. • Fast network Performance Inter-DC LAN
  20. 20. • Fast network Performance Inter-DC LAN Cross USA Washington, DC - San Jose, CA
  21. 21. • Fast network Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms
  22. 22. Failover •Replica sets
  23. 23. Failover •Replica sets •Master/slave
  24. 24. Failover •Replica sets •Min 3 nodes •Master/slave
  25. 25. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover
  26. 26. •Replication lag Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms
  27. 27. Replication Lag 1. Reads: eventual consistency
  28. 28. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
  29. 29. Eventual Consistency Stale data
  30. 30. Eventual Consistency Stale data Inconsistent data
  31. 31. Eventual Consistency Stale data Inconsistent data Changing data
  32. 32. Eventual Consistency Use Case Needs consistency? Graphs No User profile Yes Statistics Depends Alert config Yes
  33. 33. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
  34. 34. Slave behind Failover: out of date master
  35. 35. • Safe by default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries MongoDB WriteConcern
  36. 36. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
  37. 37. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
  38. 38. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
  39. 39. Picture is unrelated! Mmm, ice cream. • Fast network •More RAM Performance
  40. 40. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
  41. 41. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  42. 42. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  43. 43. More RAM = expensive Performance x2 4GB RAM 12 month Prices
  44. 44. RAM SSDs Spinning disk Cost Speed
  45. 45. Softlayer disk pricing Performance
  46. 46. EC2 disk/RAM pricing Performance $2232/m $2520/m $43/m $295/m
  47. 47. SSD vs Spinning Performance
  48. 48. SSD vs Spinning Performance
  49. 49. SSD vs Spinning Performance
  50. 50. Tips: rand() •Field names
  51. 51. Tips: rand() •Field names •Covered indexes
  52. 52. Tips: rand() •Field names •Covered indexes •Collections / databases
  53. 53. Backups What is the use case?
  54. 54. Backups What is the use case? Fixing user errors? Point in time restore? Disaster recovery?
  55. 55. Backups •Disaster recovery Offsite
  56. 56. Backups •Disaster recovery Age Offsite
  57. 57. Backups •Disaster recovery Age Offsite Restore time
  58. 58. Backups Frequency Consistency Verification
  59. 59. www.flickr.com/photos/daddo83/3406962115/ Monitoring •System Disk i/o Disk use
  60. 60. www.flickr.com/photos/daddo83/3406962115/ Monitoring Disk i/o Disk use •System Swap
  61. 61. www.flickr.com/photos/daddo83/3406962115/ Monitoring Optime State •Replication
  62. 62. mongostat
  63. 63. Monitoring tools Run yourself Ganglia
  64. 64. Monitoring tools www.serverdensity.com
  65. 65. David Mytton david@serverdensity.com @davidmytton Woop Japan! blog.serverdensity.com/mongodb
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×