What to look for when deploying MongoDB in Production, and a list of the most common issues reported by customers to the support team

1. MongoDB in Production
Daniel Coupal
Senior Technical Services Engineer
MongoDB

2. Goal of the Presentation
1. Walk you over common deployment problems seen by the support team
2. Have an interactive session

3. Who am I?
• Daniel Coupal
• Previously a developer, manager and consultant
• 3 years with MongoDB
• Roles
• Technical Support Engineer
• Consulting engagements with customers
• Trainer on our Advanced Internals class for new hires

4. MongoDB Support Team
Main support offices in …
New York, Austin, Palo Alto, Sydney, Delhi, Tel Aviv and Dublin

5. Who are you?
• How long have you been using MongoDB?
• In Production or Development stage?
• What presentations have you seen about MongoDB?
• Been to MongoDB Conferences?
• MongoDB Days?
• Meetups or other MongoDB events?
• What have you learned?
• On-line training classes from University.mongodb.com?
• Books on MongoDB?

6. What do you want to learn today?
• The Story of NoSQL
• MongoDB, the Company
• Deploying guidelines
• Common issues seen in Production
•
•
•

7. The Story of NoSQL

8. Once upon a time
The Sun was shinning on the land of the Oracle …

9. Then came the Web

10. We’re gonna need a bigger database

11. NoSQL databases
• Make use of newer technology
• Scale horizontally (add more servers, instead of a bigger server)
• Store richer data shapes vs “square data” in RDBMS
• Faster for a given application
• High Availability and Replication built in
• Distributed Systems

12. Data is like Oil
• Precious resource
• Even more when refined
• What was easy to extract/store has been done
• Most of the data not in databases is likely not square and/or large datasets
• New technology help you address what has not been processed
• MongoDB is in a good position to be the database of choice for the remaining
data to be discovered

13. MongoDB – The Company

14. MongoDB - Company Overview
600 employees 1,000+ customers
Over $300 million in funding30+ offices around the world

15. MongoDB - Timeline
• Originaly, 10gen
• Founded in 2007
• Released MongoDB 1.0 in 2009
• MongoDB Inc
• Since 2013
• Acquired WiredTiger in 2014
• MongoDB
• Open source software
• Contributions on tools, drivers, …
• Most popular NoSQL database

16. Forrester Analysis – Q3 2016
• MongoDB Leader in
• Strategy
• Current Offering

17. MongoDB - We hire!
Positions open in Palo Alto, Austin, NYC and for remote workers
• http://www.mongodb.com/careers/positions
Technical service engineers in Palo Alto
• MongoDB
• MongoDB Tools
• Proactive support

18. Deploying Guidelines

19. Security
• SSL/X.509
• Kerberos/LDAP
• Data encryption at rest
• Users
• Firewall/security groups

20. Where are you deploying?
• Own servers
• AWS
• Azure
• Google

21. Deployment platform
• MongoDB Ops Manager
• MongoDB Cloud Manager
• Chef
• Puppet
• Docker

22. Topology
• How many nodes?
• Primary / Secondary / Secondary (PSS)
• Primary / Secondary / Arbiter (PSA)
• More data nodes?
• How many data centers?

23. Backups
• Ops/Cloud Manager
• File System Snapshots
• mongodump/mongorestore

24. Most Common Problems in the Field
(not in any order)

25. Issue #1 – The NoSQL World is different

26. #1 – The NoSQL World is different

27. #1 – The NoSQL World is different
Relational World NoSQL World
Step 1 - Model the data Step 1 - Define the access patterns to the data
Step 2 - Write the queries to access the data Step 2 - Model the data
Step 3 - Denormalize the data

28. Example: Schema in MongoDB
physicians
hospitals
patients
records
procedures

29. Issue #2 – Have a Test Environment

30. #2 – Have a Test Environment
• Excellent (A)
• identical cluster
• restore production data in test/staging/QA environment
• Acceptable (B)
• less shards with identical hardware and same amount of data per shard
• Less desirable (C)
• smaller hardware
• much less data
• non representative data
• Not acceptable (F)
• No test environment

31. Issue #3 – Look for Clues of Bad Queries

32. #3 – Look for Clues of Bad Queries
(slow queries are listed in mongod log)
3.2 format
2016-05-05T12:52:05.071-0700 I COMMAND [conn16] command test.twitter
command: find { find: "twitter", filter: { user.profile_background_image_url: /
biatwitter/ } } planSummary: COLLSCAN keysExamined:0 docsExamined:
102856 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:803
nreturned:2 reslen:3077 locks:{ Global: { acquireCount: { r: 1608 } }, Database:
{ acquireCount: { r: 804 } }, Collection: { acquireCount: { r: 804 } } }
protocol:op_command 151ms
Amount of work done
How many results?
Query should be fast!

33. #3 – Look for Clues of Bad Queries
(slow queries in mongod log)
2.6 format
2016-05-05T12:54:04.432-0700 [conn30] query test.twitter query: { user.profile_background_image_url: /biatwitter/ }
planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:102856 nscannedObjects:102856 keyUpdates:0 numYields:0
locks(micros) r:170768 nreturned:2 reslen:2988 170ms
3.0 format
2016-05-05T12:53:42.291-0700 I COMMAND [conn9039] query test.twitter query: { user.profile_background_image_url: /
biatwitter/ } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:102856 keyUpdates:0
writeConflicts:0 numYields:803 nreturned:2 reslen:2988 locks:{ Global: { acquireCount: { r: 1608 } }, MMAPV1Journal:
{ acquireCount: { r: 804 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { R: 804 } } } 159ms
3.2 format
2016-05-05T12:52:05.071-0700 I COMMAND [conn16] command test.twitter command: find { find: "twitter", filter:
{ user.profile_background_image_url: /biatwitter/ } } planSummary: COLLSCAN keysExamined:0 docsExamined:102856
cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:3077 locks:{ Global: { acquireCount: { r:
1608 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { r: 804 } } } protocol:op_command 151ms

34. #3 – Look for Clues of Bad Queries
• mtools for plots and
stats on queries

35. #3 – Look for Clues of Bad Queries
• Cloud Manager (Premium groups) or Ops Manager
• Suggested indexes !!!
• Cloud Manager and Ops Manager
• charts
• Logs
• Profiler
• explain()

36. Issue #4 – Incorrect Production Settings

37. #4 – Incorrect Production Settings
• Hardware
• Don’t use:
• NAS
• other FS than xfs on Linux for WiredTiger
• other FS than ext4 or xfs on Linux for MMAPv1
• VM Settings
• Don’t use ballooning/overcommit resources
• OS Settings
• ulimits high enough
• Transparent Huge Pages off
• Turn NUMA off, or use numactl to start the processes
• low readahead for disks (32 instead of default of 256)

38. #4 – Incorrect Production Settings
• Production notes
https://docs.mongodb.org/manual/administration/production-notes/
• Warnings
• At startup and in mongod.log
• In Cloud Manager/Ops Manager
• Support ticket
mdiag (in GitHub) will gather info to check

39. Issue #5 - Cloud/Ops Manager Agents always initiate
communications

40. #5 - Cloud/Ops Manager Agents always initiate
communications

41. #5 - Cloud/Ops Manager Agents always initiate
communications
Monitoring
Agent
Secondary SecondaryPrimary

42. Issue #6 - Knowledge is King

43. #6 - Knowledge is King
• Use the right resources
• MongoDB.com/presentations
• Conferences: MongoDB Days/MongoDB World
• Consulting packages
• Production readiness, Rapid Start, Performance Tuning, Database Modernization, …
• Commercial Support
• Free Support (Google groups and Stack Overflow)
• Training:
• On-site classes
• Public trainings classes @ MongoDB offices
• Free education classes (M101 per language, M102, M202)

44. Conclusion

45. Take Away
1. Remember 3 problems out of the 6 from this presentation
2. Make 1 change in your application or deployment to prevent one issue
3. If you do, this will be a good return on your hour spent at this presentation

46. Thank you!

47. Extras