NoSQL Infrastructure - Late 2013

1,584 views

Published on

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment.

This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,584
On SlideShare
0
From Embeds
0
Number of Embeds
570
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

NoSQL Infrastructure - Late 2013

  1. 1. NoSQL Infrastructure
  2. 2. ~3333 write ops/s 0.07 - 0.05 ms response
  3. 3. David Mytton Woop Japan!
  4. 4. MongoDB at Server Density
  5. 5. •27 nodes MongoDB at Server Density
  6. 6. • June 2009 - 4yrs •27 nodes MongoDB at Server Density
  7. 7. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - 4yrs
  8. 8. •MySQL -> MongoDB •25TB data per month •27 nodes MongoDB at Server Density • June 2009 - 4yrs
  9. 9. Why?
  10. 10. Why? • Replication
  11. 11. Why? • Replication • Official drivers
  12. 12. Why? • Replication • Official drivers • Easy deployment
  13. 13. Why? • Replication • Official drivers • Easy deployment • Fast out of the box
  14. 14. It’s a little different.
  15. 15. Picture is unrelated! Mmm, ice cream. • Fast network Performance
  16. 16. • Fast network Performance EC2 10 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage
  17. 17. • Fast network Performance Workload: Read/Write? Result set size What is being stored?
  18. 18. • Fast network Performance Use Network Throughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog
  19. 19. • Fast network Performance Inter-DC LAN
  20. 20. • Fast network Performance Inter-DC LAN Cross USA Washington, DC - San Jose, CA
  21. 21. • Fast network Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms
  22. 22. Failover •Replica sets
  23. 23. Failover •Replica sets •Master/slave
  24. 24. Failover •Replica sets •Min 3 nodes •Master/slave
  25. 25. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover
  26. 26. •Replication lag Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Pacific 150ms Europe - Japan 300ms
  27. 27. Replication Lag 1. Reads: eventual consistency
  28. 28. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
  29. 29. Eventual Consistency Stale data
  30. 30. Eventual Consistency Stale data Inconsistent data
  31. 31. Eventual Consistency Stale data Inconsistent data Changing data
  32. 32. Eventual Consistency Use Case Needs consistency? Graphs No User profile Yes Statistics Depends Alert config Yes
  33. 33. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind
  34. 34. Slave behind Failover: out of date master
  35. 35. • Safe by default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries MongoDB WriteConcern
  36. 36. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})
  37. 37. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags
  38. 38. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags
  39. 39. Picture is unrelated! Mmm, ice cream. • Fast network •More RAM Performance
  40. 40. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
  41. 41. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  42. 42. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  43. 43. More RAM = expensive Performance x2 4GB RAM 12 month Prices
  44. 44. RAM SSDs Spinning disk Cost Speed
  45. 45. Softlayer disk pricing Performance
  46. 46. EC2 disk/RAM pricing Performance $2232/m $2520/m $43/m $295/m
  47. 47. SSD vs Spinning Performance
  48. 48. SSD vs Spinning Performance
  49. 49. SSD vs Spinning Performance
  50. 50. Tips: rand() •Field names
  51. 51. Tips: rand() •Field names •Covered indexes
  52. 52. Tips: rand() •Field names •Covered indexes •Collections / databases
  53. 53. Backups What is the use case?
  54. 54. Backups What is the use case? Fixing user errors? Point in time restore? Disaster recovery?
  55. 55. Backups •Disaster recovery Offsite
  56. 56. Backups •Disaster recovery Age Offsite
  57. 57. Backups •Disaster recovery Age Offsite Restore time
  58. 58. Backups Frequency Consistency Verification
  59. 59. www.flickr.com/photos/daddo83/3406962115/ Monitoring •System Disk i/o Disk use
  60. 60. www.flickr.com/photos/daddo83/3406962115/ Monitoring Disk i/o Disk use •System Swap
  61. 61. www.flickr.com/photos/daddo83/3406962115/ Monitoring Optime State •Replication
  62. 62. mongostat
  63. 63. Monitoring tools Run yourself Ganglia
  64. 64. Monitoring tools www.serverdensity.com
  65. 65. David Mytton david@serverdensity.com @davidmytton Woop Japan! blog.serverdensity.com/mongodb

×