Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
#MongoDB 
Advanced MongoDB 
for Development, Deployment 
and Operation 
Daniel Coupal 
Technical Services Engineer, Palo A...
2 
MongoDB Overview 
400+ employees 1,000+ customers 
13 offices around the world Over $231 million in funding
3 
This presentation is not … 
• an introduction to MongoDB 
First steps with MongoDB 
by Nuri Halperin 
5:00 PM Saturday ...
4 
This presentation is about … 
• Making you successful in developing, 
deploying and operating an application with 
Mong...
5 
Agenda 
1. Some Concepts 
2. The Story of your Application 
I. Prototype and Development 
II. Deployment 
III. Operatio...
1. Some Concepts
7 
Some Concepts 
• Oplog 
• Working set 
• MMS 
• Collection scans 
• Deployments/elections
8 
What is a Replica Set Oplog? 
• A capped collection that stores an ordered 
history of logical writes to a MongoDB 
dat...
9 
Sizing the Oplog collection 
• The capped collection dictates the amount 
of hours a secondary/backup agent can stop 
t...
Working set 
10 
• Working Set: The total body of data+indexes 
that the application uses in the course of 
normal operati...
The MMS Components 
A. Monitoring 
1. Cloud: Sept 2011 
2. On-Prem: July 2013 
B. Backups 
1. Cloud: April 2013 
2. On-Pre...
The MMS User Interface 
12
Overview of MMS Monitoring 
13
Overview of MMS Backup 
14
MMS under the hood: monitoring and backup agents 
15
Collection scan 
16 
• Very bad if you have a large collection 
• One of the main performance issue see in our 
customers’...
Deployments/elections 
17 
• 3 data nodes 
• If even number of data nodes, add an arbiter 
– Don’t use more than one arbit...
2. The Story of your Application
I. Prototype and Development 
19 
1. Schema, schema, schema! 
2. What happens when a failure is returned 
by the database?...
Think about data sizing and growth 
20 
• How much data will you have initially? 
• How will your data set grow over time?...
Performance Tuning 
1. Assess the problem and establish acceptable behavior 
2. Measure the current performance 
3. Find t...
II. Deploy 
22 
1. Deployment topology 
2. Have a test/staging environment 
– Track slow queries and collection scans 
3. ...
Storage considerations 
23 
• RAID 
=> 0+1 
• NAS, SAN or Direct Attached? 
=> Direct Attached 
• HDD or SSD 
=> SSD, if b...
III. Operation 
24 
1. Monitor 
2. Upgrade 
3. Backup 
4. Troubleshoot
Disaster will strike 
25 
“Shit will happen!” 
• Are you prepared? 
• Have backups? 
• Have a good picture of your “normal...
Monitor 
26 
• iostat, top, vmstat, sar 
• mongostat, mongotop 
• MMS Monitoring 
– Use Munin extensions
Upgrade 
27 
• Major versions have same binary format, 
same protocol, etc 
• MMS Automation handles automatic 
upgrades
Comparing MongoDB backup approaches 
28 
Mongodump File system MMS Backup 
Cloud 
MMS Backup 
On-Prem 
Initial complexity ...
3. Wrapping up
Common Mistakes 
30 
1. Missing indexes 
2. Not testing before deploying application changes 
3. ulimits 
a. number of ope...
Resources 
31 
• MongoDB Professional Customer Support 
– 24x7 support 
– the sun never set on MongoDB Customer Support Te...
Static Resources 
32 
• MongoDB on-line presentations mongodb.com/presentations 
• Free MongoDB classes university.mongodb...
Summary 
33 
• Use available resources 
• Testing 
– Plan for it, plan resources for it, do it before deploying
Take away 
34 
I hope you walk out of this presentation and 
you make at least one single change in your 
application, dep...
We hire 
35 
Positions open in Palo Alto, Austin and NYC 
• http://www.mongodb.com/careers 
Technical service engineer in ...
4. Q&A
Thank you for using 
Daniel Coupal, Technical Services Engineer
Upcoming SlideShare
Loading in …5
×

Silicon Valley Code Camp 2014 - Advanced MongoDB

649 views

Published on

MongoDB presentation from Silicon Valley Code Camp 2014.
Walkthrough developing, deploying and operating a MongoDB application, avoiding the most common pitfalls.

Published in: Software
  • Be the first to comment

Silicon Valley Code Camp 2014 - Advanced MongoDB

  1. 1. #MongoDB Advanced MongoDB for Development, Deployment and Operation Daniel Coupal Technical Services Engineer, Palo Alto, CA Silicon Valley Code Camp 2014
  2. 2. 2 MongoDB Overview 400+ employees 1,000+ customers 13 offices around the world Over $231 million in funding
  3. 3. 3 This presentation is not … • an introduction to MongoDB First steps with MongoDB by Nuri Halperin 5:00 PM Saturday • about code examples Beer Locker: Building a RESTful API with Node.js by Scott Smith 2:45 PM Sunday Get MEAN! MongoDb + express + angular + node by Ward Bell 1:45 PM Saturday Getting RESTless with MeteorJS and MongoDB in the browser by Ryan Jarvinen 2:45 PM Sunday
  4. 4. 4 This presentation is about … • Making you successful in developing, deploying and operating an application with MongoDB • I do expect you to know the basics of MongoDB. • …even better if you already have an application about to be deployed
  5. 5. 5 Agenda 1. Some Concepts 2. The Story of your Application I. Prototype and Development II. Deployment III. Operation 3. Wrapping up 4. Q&A
  6. 6. 1. Some Concepts
  7. 7. 7 Some Concepts • Oplog • Working set • MMS • Collection scans • Deployments/elections
  8. 8. 8 What is a Replica Set Oplog? • A capped collection that stores an ordered history of logical writes to a MongoDB database – Does not store operations like increment, add to set, etc. Those are translated to the final document. – Safe to replay old oplogs. Needs to play all of them in the right order. • Enables replication • Enables backups
  9. 9. 9 Sizing the Oplog collection • The capped collection dictates the amount of hours a secondary/backup agent can stop talking to the primary • MMS Monitoring has a Replication Oplog Window graph • Higher rate of writes to the DBs requires a larger Oplog collection
  10. 10. Working set 10 • Working Set: The total body of data+indexes that the application uses in the course of normal operation. – http://docs.mongodb.org/manual/faq/storage/#what-is-the- working-set – MongoDB v2.4 added a working set estimator to the serverStatus command – http://docs.mongodb.org/manual/reference/command/ serverStatus/#serverStatus.workingSet
  11. 11. The MMS Components A. Monitoring 1. Cloud: Sept 2011 2. On-Prem: July 2013 B. Backups 1. Cloud: April 2013 2. On-Prem: April 2014 C. Automation 11 1. Cloud: October 2014
  12. 12. The MMS User Interface 12
  13. 13. Overview of MMS Monitoring 13
  14. 14. Overview of MMS Backup 14
  15. 15. MMS under the hood: monitoring and backup agents 15
  16. 16. Collection scan 16 • Very bad if you have a large collection • One of the main performance issue see in our customers’ application • Can be identified in the logs with the ‘nscanned’ attribute on slow queries
  17. 17. Deployments/elections 17 • 3 data nodes • If even number of data nodes, add an arbiter – Don’t use more than one arbiter • Many Data Centers or availability zones • What is important for you? => can be chosen per operation – Durability of writes – Performance
  18. 18. 2. The Story of your Application
  19. 19. I. Prototype and Development 19 1. Schema, schema, schema! 2. What happens when a failure is returned by the database? 3. Index correctly 4. Incorporate testability in your application 5. Think about data sizing and growth 6. Performance Tuning
  20. 20. Think about data sizing and growth 20 • How much data will you have initially? • How will your data set grow over time? • How big is your working set? • Will you be loading huge bulk inserts, or have a constant stream of writes? • How many reads and writes will you need to service per second? • What is the peak load you need to provision for?
  21. 21. Performance Tuning 1. Assess the problem and establish acceptable behavior 2. Measure the current performance 3. Find the bottleneck* 4. Remove the bottleneck 5. Re-test to confirm 6. Repeat * - (This is often the hard part) (Adapted from http://en.wikipedia.org/wiki/Performance_tuning ) 21
  22. 22. II. Deploy 22 1. Deployment topology 2. Have a test/staging environment – Track slow queries and collection scans 3. MongoDB production notes – http://docs.mongodb.org/manual/administration/production-notes 4. Storage considerations
  23. 23. Storage considerations 23 • RAID => 0+1 • NAS, SAN or Direct Attached? => Direct Attached • HDD or SSD => SSD, if budget permit
  24. 24. III. Operation 24 1. Monitor 2. Upgrade 3. Backup 4. Troubleshoot
  25. 25. Disaster will strike 25 “Shit will happen!” • Are you prepared? • Have backups? • Have a good picture of your “normal state”
  26. 26. Monitor 26 • iostat, top, vmstat, sar • mongostat, mongotop • MMS Monitoring – Use Munin extensions
  27. 27. Upgrade 27 • Major versions have same binary format, same protocol, etc • MMS Automation handles automatic upgrades
  28. 28. Comparing MongoDB backup approaches 28 Mongodump File system MMS Backup Cloud MMS Backup On-Prem Initial complexity Medium High Low High System overhead High Low Low Medium Point in time Yes * No Yes Yes recovery of replica set Consistent snapshot of sharded system Yes * Yes * Yes Yes Scalable No Yes Yes Yes Restore time Slow Fast Medium Medium * Possible, but need to write the tools and go though a lot of pain
  29. 29. 3. Wrapping up
  30. 30. Common Mistakes 30 1. Missing indexes 2. Not testing before deploying application changes 3. ulimits a. number of open files => 64000 b. number of processes/threads => 64000 4. Appropriate schema 5. Hardware a. right disks for the job b. enough RAM 6. Not seeking help early enough
  31. 31. Resources 31 • MongoDB Professional Customer Support – 24x7 support – the sun never set on MongoDB Customer Support Team • MongoDB Consulting Days • MongoDB World (@NYC in June) • MongoDB Days (@SF on Dec 3, 2014) • MongoDB Office Hours • Google Groups
  32. 32. Static Resources 32 • MongoDB on-line presentations mongodb.com/presentations • Free MongoDB classes university.mongodb.com • MMS Cloud mms.mongodb.com • MMS Cloud Documentation mms.mongodb.com/help-classic • mtools to analyze MongoDB logs github.com/rueckstiess/mtools
  33. 33. Summary 33 • Use available resources • Testing – Plan for it, plan resources for it, do it before deploying
  34. 34. Take away 34 I hope you walk out of this presentation and you make at least one single change in your application, deployment, configuration, etc that will prevent one issue from happening.
  35. 35. We hire 35 Positions open in Palo Alto, Austin and NYC • http://www.mongodb.com/careers Technical service engineer in Palo Alto • http://www.mongodb.com/careers/ positions/technical-services-engineer
  36. 36. 4. Q&A
  37. 37. Thank you for using Daniel Coupal, Technical Services Engineer

×