Deployment Strategies (Mongo Austin)

2,336 views

Published on

Bernie Hackett's presentation at Mongo Austin

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,336
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
36
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Deployment Strategies (Mongo Austin)

    1. 1. Deployment Bernie Hackettbernie@10gen.com
    2. 2. Agenda• Sizing Hardware (Memory / CPU / IO)• Operating Systems / Filesystem• Installing / Upgrading• Durability• Security• Backup• Performance / Logging / Monitoring• EC2
    3. 3. So you’re ready to deploy !
    4. 4. Some points to consider...
    5. 5. Sizing Hardware
    6. 6. Sizing Hardware: Memory• Working set should be as much in memory as possible • Your whole data set doesn’t have to be• Memory Mapped files • Maps files on filesystem to Virtual Memory • Not Physical RAM • Page Faults - not in memory - from disk - expensive• Indexes part of the regular DB files• Consider Warm Starting your Database
    7. 7. Sizing Hardware: CPU• MongoDB uses multiple cores • Generally, faster CPUs are better • For working-set queries, CPU usage is minimal• Aggregation, full “table” scans (collection scans) • Make heavy use of CPU / Disk • Instead of counting / computing: • precompute / preaggregate / store results• Map Reduce • Currently Single threaded • Can be run in parallel across shards. • This restriction may be eliminated, investigating options
    8. 8. Sizing Hardware: I/O• Disk I/O determines performance of non-working set queries• More Faster Disks = Better • Raid 10 - Stripe + Mirror • improved write performance • survive single disk failure • double storage needs • Raid 5 or 6 • 1 or 2 additional disks required for parity • survive 1 or 2 disk failures • not all implementation created equally• Flash • Expensive, getting cheaper • Significantly reduced seek time, increased I/O throughput • Potentially slower random writes & sequential reads
    9. 9. OS / Filesystem
    10. 10. Operating Systems• For production: Use a 64bit OS • 32bit has 2G limit • Clients can be 32 bit• MongoDB supports (little endian only): • Linux, FreeBSD, OS X • Windows • Solaris (joyent)
    11. 11. Filesystem• All data/indexes, namespace files stored in data directory • Possible to use symbolic links • Better to distribute IO across disks• File Allocation:
    12. 12. Filesystem Continued• MongoDB is filesystem-neutral: • ext3, ext4, XFS are most used in Linux • ext4 / XFS preferred (posix_fallocate()) • Improved performance for file allocation • Support NTFS, HFS(+), etc.
    13. 13. Installing / Upgrading
    14. 14. MongoDB Version Policy• Production: run even numbers • 1.4.x, 1.6.x, 1.8.x• Development • 1.5.x, 1.7.x• Critical bugs are back ported to even versions
    15. 15. Installing MongoDB• Installing from Source • Requires Scons, C++ compiler, Boost libraries, SpiderMonkey, PCRE(++)• Installing from Binaries (easiest) • http://downloads.mongodb.org/_os_/_version_• Upgrading database • Install new version of MongoDB • Stop previous version • Start new version
    16. 16. Security• We encourage running mongoDB in a safe environment• Authenticate users on a per database basis • Start mongod with --auth• Admin user stored in the admin database use admin db.addUser("administrator", "password") db.auth(“administrator”, “password”)• Regular users stored in other databases use personnel db.addUser("joe", "password") db.addUser(“fred”, “password”, true) # read only access
    17. 17. DurabilityWhat failures do you need to recover from? • Loss of a single database node? • Loss of a group of nodes?
    18. 18. Solution: Replica Sets• One primary node• One to six secondary nodes• Automatic failover on primary failure • Secondaries elect a new primary• Write confirmation is configurable (W=n)
    19. 19. Durability - Primary only• Write acknowledged when in memory on replica set primary only
    20. 20. Durability - Primary + Secondary• W=2• Write acknowledged when in memory on primary + secondary• Will survive failure of a single node
    21. 21. Durability - Primary + Secondaries + fsync• W=n• Write acknowledged when in memory on primary + secondaries• Pick an “n” that is a “majority” of nodes• fsync in batches • blocking operation
    22. 22. Slave delay• Protection against app faults• Protection against administration mistakes• Secondary runs X amount of time behind
    23. 23. Backup
    24. 24. Backup• Typically backups are driven from a replica set secondary• Eliminates impact to client / application traffic to primary
    25. 25. Backup• Two main Strategies • mongodump / mongorestore • Filesystem backup / snapshot • fsync + lock
    26. 26. Backup - mongodump• Compact, binary object dump• Each consistent object is written• Not necessarily consistent from start to finish • Unless you lock database: • db.runCommand({fsync:1,lock:1}• --oplog dumps oplog from start to finish• mongorestore to restore database • Database does not have to be up to restore • --oplogReplay
    27. 27. Filesystem Backup• MUST • fsync - flushes buffers to disk • lock - blocks writes db.runCommand({fsync:1,lock:1})• Use file-system / LVM / storage snapshot• unlock db.$cmd.sys.unlock.findOne();
    28. 28. Database Maintenance• When doing a lot of updates or deletes • occasional database compaction might be needed • indices and data files • db.repair()• With replica sets • Rolling: start up node with --repair param
    29. 29. Monitoring
    30. 30. A Word on Performance• Ensure your queries are being executed optimally• Enable database profiling • db.setProfilingLevel(n) • n=1: slow operations, n=2: all operations • Viewing profile information • db.system.profile.find({info: /test.foo/}) •http://www.mongodb.org/display/DOCS/Database+Profiler• Query execution plan: • db.xx.find({..}).explain() • http://www.mongodb.org/display/DOCS/Optimization• Make sure your Queries are properly indexed.
    31. 31. Logging• Logfiles: • --logpath /path/to/filename.extension • Rotate: • db.runCommand(“logRotate”) • kill -SIGUSR1 <mongod pid> • killall -SIGUSR1 mongod • Does not work for ./mongod > <file>
    32. 32. MongoStat• Tool that comes with MongoDB• Shows counters for I/O, time spent in write lock, ... 31.5
    33. 33. IOStatiostat
‐w
1
(OSX) disk0 disk1 disk2 cpu load average KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m12.83 3 0.04 2.01 0 0.00 12.26 2 0.02 11 5 83 0.35 0.26 0.2511.12 75 0.81 0.00 0 0.00 0.00 0 0.00 60 24 16 0.68 0.34 0.28 4.00 3 0.01 0.00 0 0.00 0.00 0 0.00 60 23 17 0.68 0.34 0.28iostat -x 2 (Linux)avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 7.96 29.85 0.50 61.69Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %utilsda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00sda2 0.50 4761.19 6.47 837.31 75.62 43681.59 51.86 38.38 42.33 0.46 38.41 Monitor
disk
transfers
:
 >
200
‐
300
Mb/s
on
XL
EC2,

but
your
mileage
may
vary CPU
usage >
30%
during
normal
operations

    34. 34. Monitoring Continued • Built in UI, --rest and port # +1000 • Plugins for: Munin, Cacti, Nagios, Zabbix... • Hosted options Server Density, Cloudkick... • Primary function: Measure stats over time Tells you what is going on with your system
    35. 35. Management • Mongo shell • 3rd party tools Fang of Mongo Futon4Mongo Mongo3 MongoHub MongoVUE Mongui Myngo Opricot PHPMoAdmin RockMongo
    36. 36. EC2 Notes• Default instance storage is EXT3 • For best performance, reformat to EXT4 / XFS • Use recent version of EXT4• Use Striping (using MDADM or LVM) to distribute I/O •This is a good thing
    37. 37. EC2 with EBS• EBS can experience spikes in latency • 400-600mS • This is a bad thing• EBS snapshots can be used for backups • Careful - EBS can disappear• S3 can be used for longer term backups
    38. 38. download at mongodb.org We’re Hiring ! bernie@10gen.com conferences,
appearances,
and
meetups http://www.10gen.com/events Facebook









Twitter









LinkedInhttp://bit.ly/mongoO
 @mongodb http://linkd.in/joinmongo

    ×