Getting Started with MongoDB on Amazon Web Services

10,702 views
10,479 views

Published on

Published in: Technology
1 Comment
19 Likes
Statistics
Notes
  • At MongoDirector (www.mongodirector.com) we automate the entire process of deploying and managing Mongo replica sets and shards using a simple two step wizard. You can pick the number of replicas and shards and the regions in which you want to place them. Provisioned IOPS and RAID can be used for optimal performance. We also use LVM snapshots for backup so that your backups take the same amount of time irrespective of the size of data.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
10,702
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
137
Comments
1
Likes
19
Embeds 0
No embeds

No notes for slide

Getting Started with MongoDB on Amazon Web Services

  1. 1. Getting Started with MongoDB and Amazon Web Services Sandeep Parikh Technical Product Marketing sandeep.parikh@10gen.com
  2. 2. What We’ll Cover• MongoDB components » AWS instances• Deployment configurations• Storage configuration• Production tips• Setting up a three-node replica set
  3. 3. MongoDB Componentsmongod• 64-bit instances• Where your data lives• Needs RAM and Disk I/Omongos• Stateless request router• Lives on your app server• Minimal data needsconfig• Metadata server for sharded configurations• Minimal data needs• 64-bit instances
  4. 4. Amazon EC2 Components • mongod – Standard: Large or Extra Large (typical) – Hi-Memory: XL, XXL or 4XL (large data sets) – Cluster instances provide increased capacity and bandwith • mongos – Deploy onto your app server (32 or 64-bit) • config – Standard instances – Micro instance is sufficient
  5. 5. DEPLOYMENT CONFIGURATIONS
  6. 6. Single Node• mongod – 64-bit EC2 instance mongod• Storage – Multiple EBS volumes – RAID 10 – Configured using mdadm RAID 10
  7. 7. Replica Setmongod mongod mongodprimary secondary secondaryRAID 10 RAID 10 RAID 10
  8. 8. Replica Set – Using Zonesmongod mongod mongodprimary secondary secondaryRAID 10 RAID 10 RAID 10Zone 1 Zone 2 Zone 3
  9. 9. Replica Set – Multiple Regions appmongod mongod mongodprimary secondary secondaryRAID 10 RAID 10 RAID 10 Region 1 Region 2
  10. 10. Replica Set – Security Groups app “application”mongod mongod mongodprimary secondary secondaryRAID 10 RAID 10 RAID 10 “database”
  11. 11. Sharded Deployment configapp server app server app server config mongos mongos mongos config Shard 1 Shard 2 Shard 3
  12. 12. CONFIGURING STORAGE
  13. 13. Recommendations• EBS-backed storage vs. instance-based – Persistent vs. ephemeral• RAID 10: “striped mirrors”• 4-8 EBS volumes for best performance http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1_.2B_0
  14. 14. DEPLOYMENT NOTES
  15. 15. Some Tips• Know your deployment – Security – Instances – Storage• Configure filesystem as Ext4 or XFS• Reduce I/O overhead – noatime, nodiratime• Raise file descriptor limits
  16. 16. More Tips• EBS snapshots are an easy way to back up data• Deploy in a trusted environment – Consider authentication
  17. 17. REPLICA SET DEPLOYMENT STEPS
  18. 18. Security Groups• Create groups for the following – app servers – MongoDB components• Ports to remember – mongod: 27017, 28017 (web-based status) – mongod shard server: 27018 – mongos: 27017 – config: 27019• Security groups act as ACLs• Instances are accessible via SSH keys
  19. 19. Single Node • We’ll use this as a starting pointmongod • Single EC2 instance • 4 EBS volumes • RAID10 – Example:RAID 10 • 100 GiB total • 50 GiB usable
  20. 20. Creating the ComponentsCreate an instance $ ec2-run-instances ami-41814f28 -n 1 -g database -k cluster-keypair -t m1.large -z us-east-1a AMI group size count keypair zoneCreate storage volumes (4x)$ ec2-create-volume –s 25 -z us-east-1a size zone
  21. 21. Storage ConfigurationAttach storage (4x)$ ec2-attach-volume vol-e796108a -i i-11eee072 -d /dev/sdh1 volume device instanceStart the RAID$ sudo mdadm --create -l10 -n4 /dev/md0 /dev/sdh* type devices new device
  22. 22. Storage ConfigurationPartition device and make filesystem$ sudo fdisk /dev/md0$ sudo mkfs.ext4 /dev/md0p1Create mount point and set ownership$ sudo mkdir /data$ sudo chown `id -u` /dataUpdate the filesystem table and mount$ sudo echo „/dev/md0p1 /data auto noatime,noexec,nodiratime 0 0‟ >> /etc/fstab$ sudo mount –a /dev/md0p1 /data
  23. 23. Install MongoDBUpdate local repo settings$ echo "[10gen]name=10gen Repositorybaseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64gpgcheck=0" | sudo tee -a /etc/yum.repos.d/10gen.repoInstall MongoDB and tools$ sudo yum -y install mongo-10gen-server$ sudo yum -y install sysstatChange auto-start settings$ sudo chkconfig --level 3 mongod off$ sudo chkconfig --level 5 mongod off
  24. 24. MongoDB ConfigurationSet the dbpath$ sudo nano /etc/mongod.conf...dbpath=/data...Set to autostart at boot and start the server now$ sudo chkconfig mongod on$ sudo /etc/init.d/mongod startConnect to MongoDB$ mongoMongoDB shell version: 2.0.2connecting to: test>
  25. 25. Adding Additional Nodes• For each node, first repeat steps for – Creating instances and volumes – Configuring storage• Include replica set parameter in MongoDB configuration• Start mongod
  26. 26. MongoDB ConfigurationSet the dbpath$ sudo nano /etc/mongod.conf...dbpath=/datareplSet=replicaSetName replica set name...Set to autostart at boot and start the server now$ sudo chkconfig mongod on$ sudo /etc/init.d/mongod start
  27. 27. Initialize Replica SetConnect to MongoDB$ mongoMongoDB shell version: 2.0.2connecting to: test>Initialize the replica set> rs.initiate(){ "info2" : "no configuration explicitly specified -- making one", "me" : "ip-10-127-127-91:27017", "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1}Add each replica set member> rs.add(“ec2-abc.def.amazonaws.com”) external DNS name{ “ok” : 1 }
  28. 28. Initialize Replica SetThe mongo prompt should go from this>To thisPRIMARY>Or thisSECONDARY>
  29. 29. BACKUP/RESTORE
  30. 30. Backing Up Your DataLock the database$ mongoSECONDARY> use adminSECONDARY> db.runCommand({fsync:1,lock:1});{ "info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock", "ok" : 1}Run mdadm and note the UUID$ mdadm --detail /dev/md0p1Create a snapshot for each attached volume$ ec2-create-snapshot vol-1234abcd --description "MongoDB RAID backup1" volume description
  31. 31. Backing Up Your DataUnlock the database$ mongoSECONDARY> db.$cmd.sys.unlock.findOne();{ "ok" : 1, "info" : "unlock requested" }
  32. 32. Restoring Your DataCreate and attach a volume from each snapshot$ ec2-create-volume --availability-zone us-east-1a --snapshot vol-1234abcd$ ec2-attach-volume vol-1234abcd -i i-aa3bc4c8 -d /dev/sdh1Reassemble the RAID$ sudo mdadm --assemble --auto-update-homehost -u 07552c4d…a9c2f2fc --no-degraded /dev/md0mdadm: /dev/md0 has been started with 4 drives. UUID from mdadmCreate mount point and set ownership$ sudo mkdir /data2$ sudo chown `id -u` /data2
  33. 33. Restoring Your DataUpdate the filesystem table and mount$ sudo echo „/dev/md0p1 /data2 auto noatime,noexec,nodiratime 0 0‟ >> /etc/fstab$ sudo mount –a /dev/md0p1 /data2Verify the restore (optional, requires time and disk space)$ mongod --dbpath /data2$ mongoMongoDB shell version: 2.0.2connecting to: test> use yourDb> db.yourCollection.validate({full:true})Set /data2 to /data$ sudo cp /data2 /data
  34. 34. Recommended Backup Methods• Create backups on secondary servers when using replication – Doesn’t block on the primary – Secondary updates once unlocked• Use mongodump to backup live data – Collections, databases, everything – Restore with mongorestore
  35. 35. QUESTIONS?
  36. 36. Learn More• MongoDB on Amazon Web Services white paper: http://www.10gen.com/white-papers• Amazon EC2 info at MongoDB wiki: http://www.mongodb.org/display/DOCS/Amazon+EC2• Scripts for working with EBS (creating, attaching, etc.): https://gist.github.com/1482182
  37. 37. Contact Info• MongoDB – http://www.mongodb.org – Downloads, Docs, Forums, etc.• 10gen – http://www.10gen.com/contact – info@10gen.com – Consulting, Support, etc.• Sandeep Parikh – sandeep.parikh@10gen.com

×