MongoDB on EC2 and EBS


Published on

Best practices and recommendations for deploying MongoDB on Amazon EC2 and EBS.

Published in: Technology
  • At MongoDirector ( we automate the entire process of deploying and managing Mongo replica sets and shards. Provisioned IOPS are used for optimal performance. We also use LVM snapshots for backup so that your backups take the same amount of time irrespective of the size of data.
    Are you sure you want to  Yes  No
    Your message goes here
  • The suggestions given here were really helpful and thanks a lot for that. We are currently moving our EBS volumes to RAID 10 structure, and I have a question related to that.

    If we use mdadm *only*, it looks like there is no way to increase the size of storage after the initial set-up.

    If I use LVM on top of mdadm, then we can extend the storage online by adding one or more RAID 10 devices to LVM.
    However adding LVM means one more piece and more complexity. It would be great if we could just do it using mdadm, but I can't seem to find a solution. Any tips here would be great. Thanks.
    Are you sure you want to  Yes  No
    Your message goes here
  • Well, Good luck with it...
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MongoDB on EC2 and EBS

  1. 1. MongoDB on Amazon Web Services<br />Webinar 3-14-2011<br />
  2. 2. Agenda<br />MongoDB deployment components<br />Amazon EC2 infrastructure<br />Deployment scenarios <br />
  3. 3. Mongodb components<br />
  4. 4. MongoDB Components<br /><ul><li>Actual data
  5. 5. Needs RAM + Disk IO
  6. 6. Can run as Arbiter
  7. 7. No data
  8. 8. Just votes to elect primary
  9. 9. Stateless router
  10. 10. Typically run on App Servers
  11. 11. Stores sharding configuration
  12. 12. Stores small amounts of data
  13. 13. Infrequently queried/updated by MongoS</li></li></ul><li>Replica Sets<br />Primary<br />Primary<br />Primary<br />Secondary<br />Secondary<br />Secondary<br />Secondary<br />Arbiter<br />Secondary<br />Secondary<br />Secondary<br />
  14. 14. Sharding<br />MongoD<br />MongoD<br />MongoD<br />MongoD<br />MongoS<br />MongoS<br />MongoS<br />ConfigD<br />MongoD<br />MongoD<br />MongoD<br />MongoD<br />ConfigD<br />ConfigD<br />MongoD<br />MongoD<br />MongoD<br />MongoD<br />
  15. 15. Amazon EC2<br />
  16. 16. Amazon EC2 Instance Types<br />32-bit = Don’t Use<br />Typical MongoD<br />ConfigD/ Arbiter<br />Big MongoD<br />32-bit = Don’t Use<br />High CPU not necessary<br />
  17. 17. OS<br />Turn off atime<br />Raise file descriptor limits<br />cat >> /etc/security/limits.conf << EOF<br />* hard nofile 65536<br />* soft nofile 65536<br />EOF<br />DO NOT use large VM pages<br />Use ext4, xfs<br />Use RAID<br />RAID10 on MongoD<br />RAID1 on ConfigbDB<br />Warning! Known problems with Ubuntu 10.04 & EBS<br /><br /><br />
  18. 18. Mongodb components on ec2<br />
  19. 19. MongoD on EC2<br />MongoD<br />64-Bit EC2 Instance<br />LVM or<br />MDADM<br />RAID 10<br />EBS Volumes<br />
  20. 20. MongoS on EC2<br />Language Specific Driver<br />Your App<br />32- or 64-Bit EC2 Instance<br />(Typically your app server)<br />MongoS<br />
  21. 21. Arbiter on EC2<br />64-Bit EC2 Instance<br />(Micro Instance is Fine)<br />Arbiter<br />Local Instance Storage<br />
  22. 22. ConfigDB on EC2<br />64-Bit EC2 Instance<br />(Micro Instance is Fine)<br />ConfigDB<br />LVM or MDADM<br />RAID 1<br />EBS Volumes<br />
  23. 23. Deployment Scenarios<br />
  24. 24. 3-Node Replica Set <br />Your App<br />Large<br />Large<br />Large<br />MongoD<br />(Primary)<br />MongoD<br />(Secondary)<br />MongoD<br />(Secondary)<br />RAID 10<br />RAID 10<br />RAID 10<br />
  25. 25. 2-Nodes + Arbiter<br />Your App<br />Large<br />Large<br />Micro<br />MongoD<br />(Primary)<br />MongoD<br />(Secondary)<br />Arbiter<br />RAID 10<br />RAID 10<br />
  26. 26. 3-Node Replica Set <br />Your App<br />US West<br />US East<br />Large<br />Large<br />Large<br />MongoD<br />(Primary)<br />MongoD<br />(Secondary)<br />MongoD<br />(Secondary)<br />Priority = 0<br />RAID 10<br />RAID 10<br />RAID 10<br />
  27. 27. Config DB<br />Sharding<br />Amazon ELB<br />App Tier<br />Shard 2<br />Shard 1<br />Shard 3<br />
  28. 28. Running in ec2<br />
  29. 29. Operational Considerations <br />Automating Deployment <br />Puppet<br />Chef <br />Amazon CloudFormation<br />
  30. 30. Backup / Restore<br />EBS Snapshots<br />Also for rapid replica provisioning<br />LVM Snapshots<br />Use secondaries for zero downtime backups<br /> <br />
  31. 31. Security Groups<br /><ul><li>All nodes listen on all interfaces unless –bind_ip is specified
  32. 32. Specify alternate port with –port
  33. 33. Don’t mix names & IP’s. Names preferred
  34. 34. Use security groups as access control lists
  35. 35. (e.g. allow 27017 from app servers, but not everybody)</li></li></ul><li>Final thoughts<br />EC2 is a great platform for Mongo<br />Lots of production use <br />Try it out! Test a big cluster for cheap! <br />We’re here to help! <br />
  36. 36. Questions? <br />I am<br />Jared Rosoff ( @forjared<br />Try it out! <br /><br /><br />