Your SlideShare is downloading. ×
0
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
MongoDB and AWS: Integrations
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

MongoDB and AWS: Integrations

5,618

Published on

MongoDB is one of the fastest growing NoSQL workloads on AWS due to its simplicity and scalability, and recent product additions by the AWS team have only improved those traits. In this session, we’ll …

MongoDB is one of the fastest growing NoSQL workloads on AWS due to its simplicity and scalability, and recent product additions by the AWS team have only improved those traits. In this session, we’ll talk about various AWS offerings and how they fit together with MongoDB -- including CloudFormation, Elastic MapReduce, Route53, Elastic Beanstalk, Elastic Load Balancing, and more -- and how they can be leveraged to enhance your MongoDB experience.

Published in: Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,618
On Slideshare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
95
Comments
0
Likes
12
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. MongoDB and AWS Integrating with AWS Services Partner Technical Solutions, MongoDB Inc. Sandeep Parikh #mongodb
  • 2. Recap: Deployment and Availability • MongoDB basics • Deployment configurations • Instance types • Best practices • Slides and recording: – http://www.mongodb.com/presentations/mongodb-and- amazon-web-services-deploying-high-availability
  • 3. Recap: Storage Configurations • Storage options • Simple recommendations • Backup and restore • Advanced configurations • Slides and recording: – http://www.mongodb.com/presentations/mongodb-and- amazon-web-services-storage-options-mongodb- deployments
  • 4. Agenda • Available Services • Integrations • Infrastucture • Future Directions • Questions
  • 5. Available Services
  • 6. AWS Services Compute Storage Persistent IPs DNS Hadoop Data Warehouse Stream processing App deployment Orchestration Provisioning App services Caching
  • 7. AWS Services Compute Storage Persistent IPs DNS Hadoop Data Warehouse Stream processing App deployment Orchestration Provisioning Security Caching
  • 8. Integrations
  • 9. CloudFormation • Simplify provisioning and deployment • JSON-based templates • Manage like source code • Specify all manner ofAWS components • Boostrap for other tools like Chef or Puppet
  • 10. "Parameters" : { "KeyPairName" : { "Description" : "EC2 KeyPair to enable SSH access", "Type" : "String" }, "SecurityGroupName" : { "Description" : "EC2 Security Group", "Type" : "String” }, "InstanceType" : { "Type" : "String", "Default" : ”m3.large", "AllowedValues" : [”m3.large”,”m3.xlarge”,”m3.2xlarge”], "Description" : "EC2 instance type" } }, CloudFormation Sample
  • 11. "Properties" : { "InstanceType" : { "Ref" : "InstanceType" }, "ImageId" : { … }, "SecurityGroups" : [{ "Ref”: “SecurityGroupName" }], "KeyName" : { "Ref" : "KeyPairName" }, "EbsOptimized" : "true", "BlockDeviceMappings" : [{ "DeviceName" : "/dev/xvdf", "Ebs" : { "VolumeSize" : "200”, "Iops" : "1000", "VolumeType" : "io1”, "DeleteOnTermination" : “false” }}] CloudFormation Sample
  • 12. CloudFormation Templates • https://github.com/crcsmnky/aws-cfn-mongodb • Templates to launch single-node MongoDB deployment • Each one implements our best practices – EBS-optimized, PIOPS, ulimit, readahead • Used to generateAWS Marketplace instances
  • 13. CloudFormation Templates Clone the repo Upload the CF template Instance provisioning starts Instance clones repo Instance runs setup script Instance provisioned and deployed
  • 14. CloudFormation Tools • https://github.com/cloudtools/troposphere • Python package to generate CF templates • Next versions of our templates will leverage this • Coming soon: Replica Sets • Coming later: Sharded Cluster
  • 15. Elastic Map Reduce • Quickly deploy and run Hadoop in AWS • Tuned distributions to run on top of EC2 • Provision deployments with any number of nodes • Supports Spot and Reserved pricing for savings
  • 16. EMR and MongoDB • https://github.com/mongodb/mongo-hadoop • MongoDB-Hadoop connector – Bi-directional access to/from MongoDB • Supports MapReduce, Hive, Pig, Streaming • Read/write from – MongoDB deployments or – BSON backup files
  • 17. EMR with MongoDB MongoDB BSON S3 EMR EMR EMR EMR EMR EMR EMR EMR EMR EMR
  • 18. EMR Workflow Bootstrap script • MongoDB-Hadoop • MongoDB Java driver Copy resources • Bootstrap script • MapReduce job Launch EMR • Instance type • Instance count • Arguments MapReduce Output • MongoDB • BSON in S3 EMR Logs • Written to S3
  • 19. $ elastic-mapreduce --create --jobflow ENRON000 --instance-type m1.xlarge --num-instances 5 --bootstrap-action s3://$S3_BUCKET/bootstrap.sh --log-uri s3://$S3_BUCKET/enron_logs --jar s3://$S3_BUCKET/enron-example.jar --arg -D --arg mongo.job.input.format = com.mongodb.hadoop.BSONFileInputFormat --arg -D --arg mapred.input.dir = s3n://mongo-test-data/messages.bson --arg -D --arg mapred.output.dir = s3n://$S3_BUCKET/BSON_OUT --arg -D --arg mongo.job.output.format = com.mongodb.hadoop.BSONFileOutputFormat EMR Launch
  • 20. Elastic Beanstalk • Deploy and manage applications • Handles provisioning, scaling, load balancing • Built on EC2, S3, SNS,Auto Scaling
  • 21. Elastic Beanstalk Architecture App Serve r App Serve r App Serve r Security Group Elastic Load Balancer Auto Scaling Group
  • 22. Elastic Beanstalk with MongoDB App Serve r App Serve r App Serve r Security Group Elastic Load Balancer Auto Scaling Group mongo s mongo s mongo s MongoDB
  • 23. Elastic Beanstalk with MongoDB • Customize and configure software that your app needs (e.g. mongos) • Install packages • Create files • Execute commands (before or after app is setup) • Control system services • http://docs.aws.amazon.com/elasticbeanstalk/latest/ dg/customize-containers-ec2.html
  • 24. Infrastructure
  • 25. Elastic IPs • EC2 instances use dynamic IP addresses • EIPs are static addresses that can be assigned to individual EC2 instances • Unfortunately you have a limited number
  • 26. Route53 • Highly available and scalable DNS service in AWS • Hostnames can be assigned to EC2 instances, ELB instances, or S3 buckets • DNS load balancing with weighted-round-robin • Supports hostnames for non-AWS infrastructure
  • 27. Route53 and MongoDB • Short answer: use hostnames for all components • With replica sets, hostnames can ease machine replacement • With sharded clusters, hostnames can simplify config server maintenance
  • 28. VPC • Virtual Private Cloud lets you provision a logically isolated network insideAWS • You manage all aspects of networking including – IP address ranges – Subnets – Routing tables and gateways • Can be used as an extension to an offsite data center with Hardware VPN
  • 29. VPC Public and Private http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_In troduction.html • Private subnets hidden to outside world • Internet Gateway and EIPs can be used to access • Web tier in public subnet • Data tier in private subnet
  • 30. ElastiCache • Distributed in-memory cache • Backed by Memecached or Redis • Can be a drop-in replacement for existing cache deployments • Supports auto-discovery and read-replicas
  • 31. Future Directions
  • 32. RedShift • Fully-managed petabyte-scale data warehouse service • MongoDB not natively supported as a data source • … So how do you get your data in?
  • 33. Data Pipeline • Process and move data between differentAWS compute and storage services • Date Pipeline handles resources, failures, and dependencies http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html
  • 34. Data Pipeline with MongoDB AWS Data Pipeline MongoDB S3 EMRor Redshift
  • 35. OpsWorks • Complete DevOps stack • Model and manage apps, load balancers, databases • Uses Chef recipes • Load or time-based scaling • Deploying MongoDB with OpsWorks: – http://blogs.aws.amazon.com/application- management/post/Tx1RB65XDMNVLUA/Deploying- MongoDB-with-OpsWorks
  • 36. CloudWatch • Monitoring for AWS resources • Supports custom metrics http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatch.ht ml
  • 37. aws cloudwatch put-metric-data --metric-name ResidentMemory --namespace MongoDB --timestamp 2014-02-14T20:30:00Z --value 32 --unit Gigabytes CloudWatch Custom Metrics
  • 38. Questions?
  • 39. MongoDB World New York City, June 23-25 #MongoDBWorld See what’s next in MongoDB including • MongoDB 2.6 • Sharding • Replication • Aggregation http://world.mongodb.com Save 25% with discount code 25SandeepParikh

×