Your SlideShare is downloading. ×
NoSQL Infrastructure
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

NoSQL Infrastructure

193
views

Published on

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. …

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment.

This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

Presented at NoSQL Roadshow Berlin 2013 by David Mytton.

Published in: Technology, Business

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
193
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. NoSQL Infrastructure
  • 2. ~3333 write ops/s 0.07 - 0.05 ms response
  • 3. David MyttonWoop Japan!
  • 4. MongoDB at Server Density
  • 5. •27 nodesMongoDB at Server Density
  • 6. • June 2009 - 4yrs•27 nodesMongoDB at Server Density
  • 7. •MySQL -> MongoDB•27 nodesMongoDB at Server Density• June 2009 - 4yrs
  • 8. •MySQL -> MongoDB•20TB data per month•27 nodesMongoDB at Server Density• June 2009 - 4yrs
  • 9. Why?
  • 10. Why?• Replication
  • 11. Why?• Replication• Official drivers
  • 12. Why?• Replication• Official drivers• Easy deployment
  • 13. Why?• Replication• Official drivers• Easy deployment• Fast out of the box
  • 14. It’s a little different.
  • 15. Picture is unrelated! Mmm, ice cream.• Fast networkPerformance
  • 16. • Fast networkPerformanceEC2 10 Gigabit Ethernet- Cluster Compute- High Memory Cluster- Cluster GPU- High I/O- High Storage
  • 17. • Fast networkPerformanceWorkload: Read/Write?Result set sizeWhat is being stored?
  • 18. • Fast networkPerformanceUse Network ThroughputNormal 0-100Mb/sReplication (Initial Sync) Burst +100Mb/sReplication (Oplog) 0-100Mb/sBackup Initial Sync + Oplog
  • 19. • Fast networkPerformanceInter-DC LAN
  • 20. • Fast networkPerformanceInter-DC LANCross USA Washington, DC - San Jose, CA
  • 21. • Fast networkPerformanceLocation Ping RTT LatencyWithin USA 40-80msTrans-Atlantic 100msTrans-Pacific 150msEurope - Japan 300ms
  • 22. Failover•Replica sets
  • 23. Failover•Replica sets•Master/slave
  • 24. Failover•Replica sets•Min 3 nodes•Master/slave
  • 25. Failover•Replica sets•Min 3 nodes•Master/slave•Automatic failover
  • 26. • Replication lagPerformanceLocation Ping RTT LatencyWithin USA 40-80msTrans-Atlantic 100msTrans-Pacific 150msEurope - Japan 300ms
  • 27. Replication Lag1. Reads: eventual consistency
  • 28. Replication Lag1. Reads: eventual consistency2. Failover: slave behind
  • 29. Eventual ConsistencyStale data
  • 30. Eventual ConsistencyStale dataInconsistent data
  • 31. Eventual ConsistencyStale dataInconsistent dataChanging data
  • 32. Eventual ConsistencyUse Case Needs consistency?Graphs NoUser profile YesStatistics DependsAlert config Yes
  • 33. Replication Lag1. Reads: eventual consistency2. Failover: slave behind
  • 34. Slave behindFailover: out of date master
  • 35. MongoDB WriteConcernChanged Nov 27 2012
  • 36. • Safe by default>>> from pymongo import MongoClient>>> connection = MongoClient(w=int/str)Value Meaning0 Unsafe1 Primary2 Primary + x1 secondary3 Primary + x2 secondariesMongoDB WriteConcern
  • 37. Tags{_id : "someSet",members : [{_id : 0, host : "A", tags : {"dc": "ny"}},{_id : 1, host : "B", tags : {"dc": "ny"}},{_id : 2, host : "C", tags : {"dc": "sf"}},{_id : 3, host : "D", tags : {"dc": "sf"}},{_id : 4, host : "E", tags : {"dc": "cloud"}}]settings : {getLastErrorModes : {veryImportant : {"dc" : 3},sortOfImportant : {"dc" : 2}}}}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
  • 38. {_id : "someSet",members : [{_id : 0, host : "A", tags : {"dc": "ny"}},{_id : 1, host : "B", tags : {"dc": "ny"}},{_id : 2, host : "C", tags : {"dc": "sf"}},{_id : 3, host : "D", tags : {"dc": "sf"}},{_id : 4, host : "E", tags : {"dc": "cloud"}}]settings : {getLastErrorModes : {veryImportant : {"dc" : 3},sortOfImportant : {"dc" : 2}}}}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})(A or B) + (C or D) + ETags
  • 39. {_id : "someSet",members : [{_id : 0, host : "A", tags : {"dc": "ny"}},{_id : 1, host : "B", tags : {"dc": "ny"}},{_id : 2, host : "C", tags : {"dc": "sf"}},{_id : 3, host : "D", tags : {"dc": "sf"}},{_id : 4, host : "E", tags : {"dc": "cloud"}}]settings : {getLastErrorModes : {veryImportant : {"dc" : 3},sortOfImportant : {"dc" : 2}}}}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "sortOfImportant"})(A + C) or (D + E) ...Tags
  • 40. Picture is unrelated! Mmm, ice cream.• Fast network•More RAMPerformance
  • 41. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
  • 42. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  • 43. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  • 44. More RAM = expensivePerformancex2 4GB RAM 12 month Prices
  • 45. RAMSSDsSpinning diskCost Speed
  • 46. Softlayer disk pricingPerformance
  • 47. EC2 disk/RAM pricingPerformance$2232/m$2520/m$43/m$295/m
  • 48. SSD vs SpinningPerformance
  • 49. SSD vs SpinningPerformance
  • 50. SSD vs SpinningPerformance
  • 51. Tips: rand()•Field names
  • 52. Tips: rand()•Field names•Covered indexes
  • 53. Tips: rand()•Field names•Covered indexes•Collections / databases
  • 54. BackupsWhat is the use case?
  • 55. BackupsWhat is the use case?Fixing user errors?Point in time restore?Disaster recovery?
  • 56. Backups•Disaster recoveryOffsite
  • 57. Backups•Disaster recoveryAgeOffsite
  • 58. Backups•Disaster recoveryAgeOffsiteRestore time
  • 59. BackupsFrequencyConsistencyVerification
  • 60. www.flickr.com/photos/daddo83/3406962115/Monitoring•SystemDisk i/oDisk use
  • 61. www.flickr.com/photos/daddo83/3406962115/MonitoringDisk i/oDisk use•SystemSwap
  • 62. www.flickr.com/photos/daddo83/3406962115/MonitoringOptimeState•Replication
  • 63. mongostat
  • 64. Monitoring toolsRun yourselfGanglia
  • 65. Monitoring toolswww.serverdensity.com/nosql
  • 66. www.serverdensity.com/nosql
  • 67. David Myttondavid@serverdensity.com@davidmyttonWoop Japan!www.serverdensity.com/nosql