Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Silicon Valley Code Camp 2016 - MongoDB in production

258 views

Published on

What to look for when deploying MongoDB in Production, and a list of the most common issues reported by customers to the support team

Published in: Software
  • Be the first to comment

Silicon Valley Code Camp 2016 - MongoDB in production

  1. 1. MongoDB in Production Daniel Coupal Senior Technical Services Engineer MongoDB
  2. 2. Goal of the Presentation 1.  Walk you over common deployment problems seen by the support team 2.  Have an interactive session
  3. 3. Who am I? •  Daniel Coupal •  Previously a developer, manager and consultant •  3 years with MongoDB •  Roles •  Technical Support Engineer •  Consulting engagements with customers •  Trainer on our Advanced Internals class for new hires
  4. 4. MongoDB Support Team Main support offices in … New York, Austin, Palo Alto, Sydney, Delhi, Tel Aviv and Dublin
  5. 5. Who are you? •  How long have you been using MongoDB? •  In Production or Development stage? •  What presentations have you seen about MongoDB? •  Been to MongoDB Conferences? •  MongoDB Days? •  Meetups or other MongoDB events? •  What have you learned? •  On-line training classes from University.mongodb.com? •  Books on MongoDB?
  6. 6. What do you want to learn today? •  The Story of NoSQL •  MongoDB, the Company •  Deploying guidelines •  Common issues seen in Production •  •  • 
  7. 7. The Story of NoSQL
  8. 8. Once upon a time The Sun was shinning on the land of the Oracle …
  9. 9. Then came the Web
  10. 10. We’re gonna need a bigger database
  11. 11. NoSQL databases •  Make use of newer technology •  Scale horizontally (add more servers, instead of a bigger server) •  Store richer data shapes vs “square data” in RDBMS •  Faster for a given application •  High Availability and Replication built in •  Distributed Systems
  12. 12. Data is like Oil •  Precious resource •  Even more when refined •  What was easy to extract/store has been done •  Most of the data not in databases is likely not square and/or large datasets •  New technology help you address what has not been processed •  MongoDB is in a good position to be the database of choice for the remaining data to be discovered
  13. 13. MongoDB – The Company
  14. 14. MongoDB - Company Overview 600 employees 1,000+ customers Over $300 million in funding30+ offices around the world
  15. 15. MongoDB - Timeline •  Originaly, 10gen •  Founded in 2007 •  Released MongoDB 1.0 in 2009 •  MongoDB Inc •  Since 2013 •  Acquired WiredTiger in 2014 •  MongoDB •  Open source software •  Contributions on tools, drivers, … •  Most popular NoSQL database
  16. 16. Forrester Analysis – Q3 2016 •  MongoDB Leader in •  Strategy •  Current Offering
  17. 17. MongoDB - We hire! Positions open in Palo Alto, Austin, NYC and for remote workers •  http://www.mongodb.com/careers/positions Technical service engineers in Palo Alto •  MongoDB •  MongoDB Tools •  Proactive support
  18. 18. Deploying Guidelines
  19. 19. Security •  SSL/X.509 •  Kerberos/LDAP •  Data encryption at rest •  Users •  Firewall/security groups
  20. 20. Where are you deploying? •  Own servers •  AWS •  Azure •  Google
  21. 21. Deployment platform •  MongoDB Ops Manager •  MongoDB Cloud Manager •  Chef •  Puppet •  Docker
  22. 22. Topology •  How many nodes? •  Primary / Secondary / Secondary (PSS) •  Primary / Secondary / Arbiter (PSA) •  More data nodes? •  How many data centers?
  23. 23. Backups •  Ops/Cloud Manager •  File System Snapshots •  mongodump/mongorestore
  24. 24. Most Common Problems in the Field (not in any order)
  25. 25. Issue #1 – The NoSQL World is different
  26. 26. #1 – The NoSQL World is different
  27. 27. #1 – The NoSQL World is different Relational World NoSQL World Step 1 - Model the data Step 1 - Define the access patterns to the data Step 2 - Write the queries to access the data Step 2 - Model the data Step 3 - Denormalize the data
  28. 28. Example: Schema in MongoDB physicians hospitals patients records procedures
  29. 29. Issue #2 – Have a Test Environment
  30. 30. #2 – Have a Test Environment •  Excellent (A) •  identical cluster •  restore production data in test/staging/QA environment •  Acceptable (B) •  less shards with identical hardware and same amount of data per shard •  Less desirable (C) •  smaller hardware •  much less data •  non representative data •  Not acceptable (F) •  No test environment
  31. 31. Issue #3 – Look for Clues of Bad Queries
  32. 32. #3 – Look for Clues of Bad Queries (slow queries are listed in mongod log) 3.2 format 2016-05-05T12:52:05.071-0700 I COMMAND [conn16] command test.twitter command: find { find: "twitter", filter: { user.profile_background_image_url: / biatwitter/ } } planSummary: COLLSCAN keysExamined:0 docsExamined: 102856 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:3077 locks:{ Global: { acquireCount: { r: 1608 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { r: 804 } } } protocol:op_command 151ms Amount of work done How many results? Query should be fast!
  33. 33. #3 – Look for Clues of Bad Queries (slow queries in mongod log) 2.6 format 2016-05-05T12:54:04.432-0700 [conn30] query test.twitter query: { user.profile_background_image_url: /biatwitter/ } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:102856 nscannedObjects:102856 keyUpdates:0 numYields:0 locks(micros) r:170768 nreturned:2 reslen:2988 170ms 3.0 format 2016-05-05T12:53:42.291-0700 I COMMAND [conn9039] query test.twitter query: { user.profile_background_image_url: / biatwitter/ } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:102856 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:2988 locks:{ Global: { acquireCount: { r: 1608 } }, MMAPV1Journal: { acquireCount: { r: 804 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { R: 804 } } } 159ms 3.2 format 2016-05-05T12:52:05.071-0700 I COMMAND [conn16] command test.twitter command: find { find: "twitter", filter: { user.profile_background_image_url: /biatwitter/ } } planSummary: COLLSCAN keysExamined:0 docsExamined:102856 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:803 nreturned:2 reslen:3077 locks:{ Global: { acquireCount: { r: 1608 } }, Database: { acquireCount: { r: 804 } }, Collection: { acquireCount: { r: 804 } } } protocol:op_command 151ms
  34. 34. #3 – Look for Clues of Bad Queries •  mtools for plots and stats on queries
  35. 35. #3 – Look for Clues of Bad Queries •  Cloud Manager (Premium groups) or Ops Manager •  Suggested indexes !!! •  Cloud Manager and Ops Manager •  charts •  Logs •  Profiler •  explain()
  36. 36. Issue #4 – Incorrect Production Settings
  37. 37. #4 – Incorrect Production Settings •  Hardware •  Don’t use: •  NAS •  other FS than xfs on Linux for WiredTiger •  other FS than ext4 or xfs on Linux for MMAPv1 •  VM Settings •  Don’t use ballooning/overcommit resources •  OS Settings •  ulimits high enough •  Transparent Huge Pages off •  Turn NUMA off, or use numactl to start the processes •  low readahead for disks (32 instead of default of 256)
  38. 38. #4 – Incorrect Production Settings •  Production notes https://docs.mongodb.org/manual/administration/production-notes/ •  Warnings •  At startup and in mongod.log •  In Cloud Manager/Ops Manager •  Support ticket mdiag (in GitHub) will gather info to check
  39. 39. Issue #5 - Cloud/Ops Manager Agents always initiate communications
  40. 40. #5 - Cloud/Ops Manager Agents always initiate communications
  41. 41. #5 - Cloud/Ops Manager Agents always initiate communications Monitoring Agent Secondary SecondaryPrimary
  42. 42. Issue #6 - Knowledge is King
  43. 43. #6 - Knowledge is King •  Use the right resources •  MongoDB.com/presentations •  Conferences: MongoDB Days/MongoDB World •  Consulting packages •  Production readiness, Rapid Start, Performance Tuning, Database Modernization, … •  Commercial Support •  Free Support (Google groups and Stack Overflow) •  Training: •  On-site classes •  Public trainings classes @ MongoDB offices •  Free education classes (M101 per language, M102, M202)
  44. 44. Conclusion
  45. 45. Take Away 1.  Remember 3 problems out of the 6 from this presentation 2.  Make 1 change in your application or deployment to prevent one issue 3.  If you do, this will be a good return on your hour spent at this presentation
  46. 46. Thank you!
  47. 47. Extras

×