Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
So you’re ready to deploy !
some points to consider
Agenda• A word on performance• Sizing Your Hardware   • memory / cpu / disk io• Software   • os / filesystem• Installing M...
A Word on Performance• Ensure your queries are being executed correctly   • Enable profiling   • db.setProfilingLevel(n)  ...
Sizing Hardware: Memory• Working set should be as much in memory as possible, but   • your whole data set doesn’t have to•...
Sizing Hardware: CPU• MongoDB uses multiple cores   • For working-set queries, CPU usage is minimal   • Generally, faster ...
Sizing Hardware: I/O• Disk I/O determines performance of non-working set queries• More Disks = Better    • Improved throug...
MongoStat• Tool that comes with MongoDB• Shows   • counters for I/O, time spent in write lock, ...
IOStatiostat
‐x
2iostat
‐w
1            disk0                 disk1                 disk2         cpu         load average...
OS• For production: Use a 64bit OS   • 32bit has 2G limit   • Clients can be 32 bit• MongoDB supports (little endian only)...
Filesystem• All data, namespace files stored in data directory   • Possible to create links   • Better to aggregrate IO ac...
Filesystem• Logfiles:    • --logpath <file>    • Rotate:        • db.runCommand(“logRotate”)        • kill -SIGUSR1 <mongo...
MongoDB Version Policy• Production: run even numbers   • 1.4.x, 1.6.x, 1.8.x•Development   •1.5.x, 1.7.x• Critical bugs ar...
Installing MongoDB• Installing from Source    • Requires Scons, C++ compiler, Boost libraries, SpiderMonkey,    PCRE• Inst...
EC2 Notes• Default storage instance is EXT3   • For best performance, reformat to EXT4 / XFS   • Use recent version of EXT...
More EC2 Notes• EBS snapshots can be used for backups   • EBS can disappear• S3 can be used for longer term backups• Use A...
Security• Mongo supports basic security• We encourage to run mongoDB in a safe environment• Authenticates a User on a per ...
Backup• Typically backups are driven from a slave• Eliminates impact to client / application traffic to master
Backup•Two main Strategies   • mongodump / mongorestore   • Filesystem backup / snapshot• fsync+lock
mongodump• binary, compact object dump• each consistent object is written• not necessarily consistent from start to finish...
Filesystem Backup• MUST   • fsync - flushes buffers to disk   • lock - blocks writes      db.runCommand({fsync:1,lock:1})•...
Database Maintenance• When doing a lot of updates or deletes   • occasional database compaction might be needed       • in...
Durability What failures do you need to recover from?• Loss of a single database node?• Loss of a group of nodes?
Durability - Master only• Write acknowledgedwhen in memory onmaster only
Durability - Master + Slaves• W=2• Write acknowledgedwhen in memory onmaster + slave• Will survive failure of asingle node
Durability - Master + Slaves +                 fsync• W=n• Write acknowledgedwhen in memory onmaster + slaves• Pick a “maj...
Slave delay• Protection against appfaults• Protection againstadministration mistakes• Slave runs X amount oftime behind
Scale outread       shard1   shard2   shard3                                    mongos
/
       rep_a1   rep_a2   rep_a3  ...
Monitoring   • We like Munin ..   • ... but other frameworks       work as well   • Primary function:   • Measure stats ov...
So go and deploy!• Next up: Scaling MongoDB• Also, 10gen is hiring. Talk to us!
Deployment Strategies
Upcoming SlideShare
Loading in …5
×

Deployment Strategies

6,177 views

Published on

Richard Kreuter's presentation at MongoSV on December 3, 2010

Published in: Technology
  • Be the first to comment

Deployment Strategies

  1. 1. So you’re ready to deploy !
  2. 2. some points to consider
  3. 3. Agenda• A word on performance• Sizing Your Hardware • memory / cpu / disk io• Software • os / filesystem• Installing MongoDB / Upgrades• EC2 Notes• Security• Backup• Durability• Upgrading• Monitoring• Scaling out
  4. 4. A Word on Performance• Ensure your queries are being executed correctly • Enable profiling • db.setProfilingLevel(n) • n=1: slow operations, n=2: all operations • Viewing profile information • db.system.profile.find({info: /test.foo/}) •http://www.mongodb.org/display/DOCS/Database+Profiler• Query execution plan: •db.xx.find({..}).explain() •http://www.mongodb.org/display/DOCS/Optimization• Make sure your Queries are properly indexed.
  5. 5. Sizing Hardware: Memory• Working set should be as much in memory as possible, but • your whole data set doesn’t have to•Memory Mapped files • Maps Files on Filesystem to Virtual Memory • Not Physical RAM • Page Faults - not in memory - from disk - expensive• Indices • Part of the regular DB files• Consider Warm Starting your Database
  6. 6. Sizing Hardware: CPU• MongoDB uses multiple cores • For working-set queries, CPU usage is minimal • Generally, faster CPU are better• Aggregation, Full Tablescans •Makes heavy use of CPU / Disk •Instead of counting / computing: • cache / precompute• Map Reduce • Currently Single threaded •Can be run in parallel across shards. • This restriction may be eliminated, investigating options
  7. 7. Sizing Hardware: I/O• Disk I/O determines performance of non-working set queries• More Disks = Better • Improved throughput, Reduced Seek times • Raid 0 - Striping: improved write performance • Raid 1 - Mirroring: survive single disk failure • Raid 10 - both• Consider Flash ? • Expensive, getting cheaper • Significantly reduced seek time, increased IO throughput• Network • It’s easy to saturate your network • (Average doc size * number of document writes, reads) / sec
  8. 8. MongoStat• Tool that comes with MongoDB• Shows • counters for I/O, time spent in write lock, ...
  9. 9. IOStatiostat
‐x
2iostat
‐w
1 disk0 disk1 disk2 cpu load average KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m12.83 3 0.04 2.01 0 0.00 12.26 2 0.02 11 5 83 0.35 0.26 0.2511.12 75 0.81 0.00 0 0.00 0.00 0 0.00 60 24 16 0.68 0.34 0.28 4.00 3 0.01 0.00 0 0.00 0.00 0 0.00 60 23 17 0.68 0.34 0.28avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 7.96 29.85 0.50 61.69Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %utilsda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sda2 0.50 4761.19 6.47 837.31 75.62 43681.59 51.86 38.38 42.33 0.46 38.41 Monitor
disk
transfers
:
 >
200
‐
300
Mb/s
on
XL
EC2,

but
your
mileage
may
vary CPU
usage >
30
%
during
normal
operations

  10. 10. OS• For production: Use a 64bit OS • 32bit has 2G limit • Clients can be 32 bit• MongoDB supports (little endian only): • Linux, FreeBSD, OS X • Windows • Solaris (joyent)
  11. 11. Filesystem• All data, namespace files stored in data directory • Possible to create links • Better to aggregrate IO across disks•File Allocation
  12. 12. Filesystem• Logfiles: • --logpath <file> • Rotate: • db.runCommand(“logRotate”) • kill -SIGUSR1 <mongod pid> •Does not work for ./mongod > <file>• MongoDB is filesystem-neutral: • ext3, ext4 and XFS are most used • ext4 / XFS preferred (posix_allocate()) • improved performance for file allocation • Support for NTFS for windows
  13. 13. MongoDB Version Policy• Production: run even numbers • 1.4.x, 1.6.x, 1.8.x•Development •1.5.x, 1.7.x• Critical bugs are back ported to even versions
  14. 14. Installing MongoDB• Installing from Source • Requires Scons, C++ compiler, Boost libraries, SpiderMonkey, PCRE• Installing from Binaries (easiest) • curl -O http://downloads.mongodb.org/_os_/_version_• Upgrading database • Install new version of MongoDB • Stop previous version • Start new version•In case of database file changes, •mongodump / mongorestore
  15. 15. EC2 Notes• Default storage instance is EXT3 • For best performance, reformat to EXT4 / XFS • Use recent version of EXT4• Use Striping (using MDADM or LVM) aggregates I/O •This is a good thing• EC2 can experience spikes in latency • 400-600mS •This is a bad thing
  16. 16. More EC2 Notes• EBS snapshots can be used for backups • EBS can disappear• S3 can be used for longer term backups• Use Amazon availability zones • High Availability • Disaster Recovery
  17. 17. Security• Mongo supports basic security• We encourage to run mongoDB in a safe environment• Authenticates a User on a per Database basis• Start database with --auth• Admin user stored in the admin database use admin db.addUser("administrator", "password") db.auth(“administrator”, “password”)• Regular users stored in other databases use personnel db.addUser("joe", "password") db.addUser(“fred”, “password”, true)
  18. 18. Backup• Typically backups are driven from a slave• Eliminates impact to client / application traffic to master
  19. 19. Backup•Two main Strategies • mongodump / mongorestore • Filesystem backup / snapshot• fsync+lock
  20. 20. mongodump• binary, compact object dump• each consistent object is written• not necessarily consistent from start to finish • unless you lock database: • db.runCommand({fsync:1,lock:1})• mongorestore to restore database • database does not have to be up to restore
  21. 21. Filesystem Backup• MUST • fsync - flushes buffers to disk • lock - blocks writes db.runCommand({fsync:1,lock:1})• Use file-system / LVM / storage snapshot• unlock db.$cmd.sys.unlock.findOne();
  22. 22. Database Maintenance• When doing a lot of updates or deletes • occasional database compaction might be needed • indices and datafiles • db.repair()• With replica sets • Rolling: start up node with --repair param
  23. 23. Durability What failures do you need to recover from?• Loss of a single database node?• Loss of a group of nodes?
  24. 24. Durability - Master only• Write acknowledgedwhen in memory onmaster only
  25. 25. Durability - Master + Slaves• W=2• Write acknowledgedwhen in memory onmaster + slave• Will survive failure of asingle node
  26. 26. Durability - Master + Slaves + fsync• W=n• Write acknowledgedwhen in memory onmaster + slaves• Pick a “majority” ofnodes• fsync in batches (sinceit blocking)
  27. 27. Slave delay• Protection against appfaults• Protection againstadministration mistakes• Slave runs X amount oftime behind
  28. 28. Scale outread shard1 shard2 shard3 mongos
/
 rep_a1 rep_a2 rep_a3 config
server mongos
/
 rep_b1 rep_b2 rep_b3 config
server mongos
/
 rep_c2 rep_c2 rep_c3 config
server write
  29. 29. Monitoring • We like Munin .. • ... but other frameworks work as well • Primary function: • Measure stats over time • Tells you what is going on with your system
  30. 30. So go and deploy!• Next up: Scaling MongoDB• Also, 10gen is hiring. Talk to us!

×