NoSQL Infrastructure

620 views

Published on

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment.

This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

Presented at NoSQL Roadshow Berlin 2013 by David Mytton.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
620
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

NoSQL Infrastructure

  1. 1. NoSQL Infrastructure
  2. 2. ~3333 write ops/s 0.07 - 0.05 ms response
  3. 3. David MyttonWoop Japan!
  4. 4. MongoDB at Server Density
  5. 5. •27 nodesMongoDB at Server Density
  6. 6. • June 2009 - 4yrs•27 nodesMongoDB at Server Density
  7. 7. •MySQL -> MongoDB•27 nodesMongoDB at Server Density• June 2009 - 4yrs
  8. 8. •MySQL -> MongoDB•20TB data per month•27 nodesMongoDB at Server Density• June 2009 - 4yrs
  9. 9. Why?
  10. 10. Why?• Replication
  11. 11. Why?• Replication• Official drivers
  12. 12. Why?• Replication• Official drivers• Easy deployment
  13. 13. Why?• Replication• Official drivers• Easy deployment• Fast out of the box
  14. 14. It’s a little different.
  15. 15. Picture is unrelated! Mmm, ice cream.• Fast networkPerformance
  16. 16. • Fast networkPerformanceEC2 10 Gigabit Ethernet- Cluster Compute- High Memory Cluster- Cluster GPU- High I/O- High Storage
  17. 17. • Fast networkPerformanceWorkload: Read/Write?Result set sizeWhat is being stored?
  18. 18. • Fast networkPerformanceUse Network ThroughputNormal 0-100Mb/sReplication (Initial Sync) Burst +100Mb/sReplication (Oplog) 0-100Mb/sBackup Initial Sync + Oplog
  19. 19. • Fast networkPerformanceInter-DC LAN
  20. 20. • Fast networkPerformanceInter-DC LANCross USA Washington, DC - San Jose, CA
  21. 21. • Fast networkPerformanceLocation Ping RTT LatencyWithin USA 40-80msTrans-Atlantic 100msTrans-Pacific 150msEurope - Japan 300ms
  22. 22. Failover•Replica sets
  23. 23. Failover•Replica sets•Master/slave
  24. 24. Failover•Replica sets•Min 3 nodes•Master/slave
  25. 25. Failover•Replica sets•Min 3 nodes•Master/slave•Automatic failover
  26. 26. • Replication lagPerformanceLocation Ping RTT LatencyWithin USA 40-80msTrans-Atlantic 100msTrans-Pacific 150msEurope - Japan 300ms
  27. 27. Replication Lag1. Reads: eventual consistency
  28. 28. Replication Lag1. Reads: eventual consistency2. Failover: slave behind
  29. 29. Eventual ConsistencyStale data
  30. 30. Eventual ConsistencyStale dataInconsistent data
  31. 31. Eventual ConsistencyStale dataInconsistent dataChanging data
  32. 32. Eventual ConsistencyUse Case Needs consistency?Graphs NoUser profile YesStatistics DependsAlert config Yes
  33. 33. Replication Lag1. Reads: eventual consistency2. Failover: slave behind
  34. 34. Slave behindFailover: out of date master
  35. 35. MongoDB WriteConcernChanged Nov 27 2012
  36. 36. • Safe by default>>> from pymongo import MongoClient>>> connection = MongoClient(w=int/str)Value Meaning0 Unsafe1 Primary2 Primary + x1 secondary3 Primary + x2 secondariesMongoDB WriteConcern
  37. 37. Tags{_id : "someSet",members : [{_id : 0, host : "A", tags : {"dc": "ny"}},{_id : 1, host : "B", tags : {"dc": "ny"}},{_id : 2, host : "C", tags : {"dc": "sf"}},{_id : 3, host : "D", tags : {"dc": "sf"}},{_id : 4, host : "E", tags : {"dc": "cloud"}}]settings : {getLastErrorModes : {veryImportant : {"dc" : 3},sortOfImportant : {"dc" : 2}}}}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
  38. 38. {_id : "someSet",members : [{_id : 0, host : "A", tags : {"dc": "ny"}},{_id : 1, host : "B", tags : {"dc": "ny"}},{_id : 2, host : "C", tags : {"dc": "sf"}},{_id : 3, host : "D", tags : {"dc": "sf"}},{_id : 4, host : "E", tags : {"dc": "cloud"}}]settings : {getLastErrorModes : {veryImportant : {"dc" : 3},sortOfImportant : {"dc" : 2}}}}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})(A or B) + (C or D) + ETags
  39. 39. {_id : "someSet",members : [{_id : 0, host : "A", tags : {"dc": "ny"}},{_id : 1, host : "B", tags : {"dc": "ny"}},{_id : 2, host : "C", tags : {"dc": "sf"}},{_id : 3, host : "D", tags : {"dc": "sf"}},{_id : 4, host : "E", tags : {"dc": "cloud"}}]settings : {getLastErrorModes : {veryImportant : {"dc" : 3},sortOfImportant : {"dc" : 2}}}}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "sortOfImportant"})(A + C) or (D + E) ...Tags
  40. 40. Picture is unrelated! Mmm, ice cream.• Fast network•More RAMPerformance
  41. 41. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
  42. 42. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  43. 43. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  44. 44. More RAM = expensivePerformancex2 4GB RAM 12 month Prices
  45. 45. RAMSSDsSpinning diskCost Speed
  46. 46. Softlayer disk pricingPerformance
  47. 47. EC2 disk/RAM pricingPerformance$2232/m$2520/m$43/m$295/m
  48. 48. SSD vs SpinningPerformance
  49. 49. SSD vs SpinningPerformance
  50. 50. SSD vs SpinningPerformance
  51. 51. Tips: rand()•Field names
  52. 52. Tips: rand()•Field names•Covered indexes
  53. 53. Tips: rand()•Field names•Covered indexes•Collections / databases
  54. 54. BackupsWhat is the use case?
  55. 55. BackupsWhat is the use case?Fixing user errors?Point in time restore?Disaster recovery?
  56. 56. Backups•Disaster recoveryOffsite
  57. 57. Backups•Disaster recoveryAgeOffsite
  58. 58. Backups•Disaster recoveryAgeOffsiteRestore time
  59. 59. BackupsFrequencyConsistencyVerification
  60. 60. www.flickr.com/photos/daddo83/3406962115/Monitoring•SystemDisk i/oDisk use
  61. 61. www.flickr.com/photos/daddo83/3406962115/MonitoringDisk i/oDisk use•SystemSwap
  62. 62. www.flickr.com/photos/daddo83/3406962115/MonitoringOptimeState•Replication
  63. 63. mongostat
  64. 64. Monitoring toolsRun yourselfGanglia
  65. 65. Monitoring toolswww.serverdensity.com/nosql
  66. 66. www.serverdensity.com/nosql
  67. 67. David Myttondavid@serverdensity.com@davidmyttonWoop Japan!www.serverdensity.com/nosql

×