Who, what, how & in the cloud :)                                       Mark	  Hillick                                     ...
AgendaTuesday 3 July 2012
Agenda                      • Who                      • What                      • Example Deployments                  ...
Who, what & where!Tuesday 3 July 2012
Who?       • 10gen -> company behind MongoDB       • Created by Dwight & Eliot       • MongoDB is open-source & community ...
What? (1)       • Powerful, flexible, scalable, fast data store       • Document-oriented           • Embedded docs & array...
What? (2)       • Indexing       • Stored Javascript       • Javascript Shell       • Fixed-size collections       • File ...
On What?       • Linux         • 10gen repos for Debian & Red Hat like distros       • Windows       • MAC OS (primarily d...
Today’s ChallengesTuesday 3 July 2012
Where?Tuesday 3 July 2012
The Past & NowTuesday 3 July 2012
The Past & NowTuesday 3 July 2012
A bit deeperTuesday 3 July 2012
Getting Started       • Document -> basic unit of data ~ a row in RBMS       • Collection -> schema equivalent of a table ...
Collection                      > show collections                          file_tweets                         mini_tweets...
Document         { "_id" : 122, "user_id" : 123456, "total" : 100, "items" :            [ { "item_name" : "my_item_one", "...
The ShellTuesday 3 July 2012
Drivers       • Python       • Ruby       • JavaScript [node.js]       • PHP       • Perl       • Scala       • And more.....
Indexes       • Data structure collecting info       • Enhance query performance       • Optimiser sorts through & orders ...
Wire Protocol       • Simple socket-based, request-response style       protocol       • MongoDB stores “BSON” documents, ...
DeploymentsTuesday 3 July 2012
Deployment Types       • Master-Slave (not recommended)       • Replica Sets       • Shards       • Some notes on EC2 depl...
Replica Set Configurations                       Primary     Primary      Primary                      Secondary   Secondar...
Replica Set Info       • Asynchronous replication (single primary)       • Automatic failover       • App-level definition ...
Sharding                                                                                 config DB                      mon...
Sharding Notes       • Each “shard” usually a Replica Set (same options)       • Copy of meta data stored in-memory by mon...
Shard Server in EC2 (1)                      Category/Impact   Low   Medium   High                        Disk Speed      ...
Shard Server in EC2 (2)       • MongoDB designed for OS defaults on 64 bit       instance       • Use standard virtual mem...
Server in EC2 (1)       • kernel >= 2.6.23/2.6.25 respectively       • Readahead: how much more to read than what you     ...
Server in EC2 (2)       • Turn off atime on filesystem (pre-2.6.30 especially)       • RAID 10 is recommended everywhere   ...
Config Server       • Meta Data for shard stored in ConfigDB       • Copy of meta data stored in-memory by mongos       • Co...
Config Server in EC2 (1)                       Category/Impact   Low   Medium   High                         Disk Speed    ...
Config Server in EC2 (2)       • Use Raid10       • Use 64 bit instance         • Can run on shard serversTuesday 3 July 2012
Mongos in EC2 (1)                      Category/Impact   Low   Medium   High                        Disk Speed            ...
Mongos in EC2 (2)       • Often run on application servers       • 32 bit mongos ok with 64-bit mongodTuesday 3 July 2012
Arbiter in EC2 (1)                      Category/Impact   Low   Medium   High                        Disk Speed       x   ...
Arbiter in EC2 (2)       • Can use micro instance         • Elections may be slower       • Can use instance store        ...
HA in EC2       • Replica Sets       • EC2 Availability ZonesTuesday 3 July 2012
DR in EC2       • Replica Sets       • EC2 RegionsTuesday 3 July 2012
Security       • Turn on authentication       • Create a key between shards       • EC2 Security Groups         • Can refe...
Monitoring           • Links in with Cacti, Nagios, Munin-Node etc.           • MMS - > it’s freeTuesday 3 July 2012
Mongo Ports       • mongos, unsharded mongod : 27017       • sharded mongod : 27018       • config mongod : 27019       • h...
Some More EC2 NotesTuesday 3 July 2012
Instances Guidelines (1)     • Use 64-bit only, 32-bit is not recommended     • Primary/Secondary should be equal*     • H...
Instances Guidelines (2)       • Note the trade-offs - memory/network         • m1.large to m2.xlarge = 2x Mem, 0.5x Netwo...
Backups         • EBS Snapshots - RAID complicates things         • Single EBS volume, with journaling means:           • ...
EC2/MongoDB Best Practices       • https://wiki.10gen.com/display/DOCS/Amazon+EC2       • https://wiki.10gen.com/display/D...
node.jsTuesday 3 July 2012
node.js       • server-side written in javascript       • orginally written for push web apps       • created by Ryan Dahl...
Sample node.js code                       var express = require(express),                         Db = require(mongodb).Db...
node.js with mongo       • https://github.com/christkv/node-mongodb-native       • http://www.nodebeginner.orgTuesday 3 Ju...
Need Help?Tuesday 3 July 2012
• MongoDB Google User Group       • New MongoDB Docs & Old MongoDB Docs       • Presentations       • If you’re curious :)...
Credits       • Credit must go to @comerford, @jonnyeight &       @mikefiedler as I borrowed some of the knowledgeable     ...
Upcoming SlideShare
Loading in...5
×

MongoDB - Who, What & Where!

1,670

Published on

Introductory presentation to MongoDB - who, what, where & usage in the cloud :) Yes, the #cloud!!!

Brown Bag session with UCD.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,670
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

MongoDB - Who, What & Where!

  1. 1. Who, what, how & in the cloud :) Mark  Hillick mark@10gen.com http://www.10gen.com  &  http://www.mongodb.orgTuesday 3 July 2012
  2. 2. AgendaTuesday 3 July 2012
  3. 3. Agenda • Who • What • Example Deployments • EC2 Notes • EC2 Best Practices • Further TuningTuesday 3 July 2012
  4. 4. Who, what & where!Tuesday 3 July 2012
  5. 5. Who? • 10gen -> company behind MongoDB • Created by Dwight & Eliot • MongoDB is open-source & community is key • Offices in California, NY, Dublin, London & Sydney • $73.4 million in VC fundingTuesday 3 July 2012
  6. 6. What? (1) • Powerful, flexible, scalable, fast data store • Document-oriented • Embedded docs & arrays • Scale out • Easy to start & develop withTuesday 3 July 2012
  7. 7. What? (2) • Indexing • Stored Javascript • Javascript Shell • Fixed-size collections • File StorageTuesday 3 July 2012
  8. 8. On What? • Linux • 10gen repos for Debian & Red Hat like distros • Windows • MAC OS (primarily dev) • Raspberry PI :)Tuesday 3 July 2012
  9. 9. Today’s ChallengesTuesday 3 July 2012
  10. 10. Where?Tuesday 3 July 2012
  11. 11. The Past & NowTuesday 3 July 2012
  12. 12. The Past & NowTuesday 3 July 2012
  13. 13. A bit deeperTuesday 3 July 2012
  14. 14. Getting Started • Document -> basic unit of data ~ a row in RBMS • Collection -> schema equivalent of a table • Single instance can have multiple dbs • JS Shell -> administration • Event document has special, unique key -> _idTuesday 3 July 2012
  15. 15. Collection > show collections file_tweets mini_tweets system.indexesTuesday 3 July 2012
  16. 16. Document { "_id" : 122, "user_id" : 123456, "total" : 100, "items" : [ { "item_name" : "my_item_one", "price" : 20 }, { "item_name" : "my_item_one", "price" : 20 }, { "item_name" : "my_item_three", "price" : 30 } ] }Tuesday 3 July 2012
  17. 17. The ShellTuesday 3 July 2012
  18. 18. Drivers • Python • Ruby • JavaScript [node.js] • PHP • Perl • Scala • And more...... • http://docs.mongodb.org/manual/applications/drivers (Wiki’s & Docs)Tuesday 3 July 2012
  19. 19. Indexes • Data structure collecting info • Enhance query performance • Optimiser sorts through & orders docs • db.file_tweets.ensureIndex({“user_id” : 1}) • http://www.mongodb.org/display/DOCS/IndexesTuesday 3 July 2012
  20. 20. Wire Protocol • Simple socket-based, request-response style protocol • MongoDB stores “BSON” documents, which are binary encoded serialisation of JSON documents • TCP /IP socket on port 27017 - fire & forget •http://www.mongodb.org/pages/viewpage.action?pageId=131351Tuesday 3 July 2012
  21. 21. DeploymentsTuesday 3 July 2012
  22. 22. Deployment Types • Master-Slave (not recommended) • Replica Sets • Shards • Some notes on EC2 deploymentsTuesday 3 July 2012
  23. 23. Replica Set Configurations Primary Primary Primary Secondary Secondary Secondary Arbiter Secondary Secondary (Minimum) (Typical) Secondary SecondaryTuesday 3 July 2012
  24. 24. Replica Set Info • Asynchronous replication (single primary) • Automatic failover • App-level definition of “write replication” • Secondary nodes can replicate with a slaveDelay • Secondary nodes can be hidden • Max of 12 nodes, with 7 votingTuesday 3 July 2012
  25. 25. Sharding config DB mongos mongos mongos mongos config DB config DB Primary Primary Primary Primary Secondary Secondary Secondary Secondary Secondary Secondary Secondary SecondaryTuesday 3 July 2012
  26. 26. Sharding Notes • Each “shard” usually a Replica Set (same options) • Copy of meta data stored in-memory by mongos • Data split into chunks, using range based shard key • Chunks may be migrated between shards • New chunks created by “splitting” old chunksTuesday 3 July 2012
  27. 27. Shard Server in EC2 (1) Category/Impact Low Medium High Disk Speed x Disk Capacity x RAM x CPU xTuesday 3 July 2012
  28. 28. Shard Server in EC2 (2) • MongoDB designed for OS defaults on 64 bit instance • Use standard virtual memory page size • Raise “nofiles” ulimit (20,000) • Use RAID10 & modern f/s -> ext4, xfs etc • Use “noatime” mount optionTuesday 3 July 2012
  29. 29. Server in EC2 (1) • kernel >= 2.6.23/2.6.25 respectively • Readahead: how much more to read than what you asked for • If too high => possible performance impact • Set to 0 on EBS devices • Set to desired value on RAID deviceTuesday 3 July 2012
  30. 30. Server in EC2 (2) • Turn off atime on filesystem (pre-2.6.30 especially) • RAID 10 is recommended everywhere • mitigates slow EBS volumes (fail the bad volume) • Do not use large VM pages • Do configure swap to prevent OOM KillerTuesday 3 July 2012
  31. 31. Config Server • Meta Data for shard stored in ConfigDB • Copy of meta data stored in-memory by mongos • Config DB cluster is *not* a replica set -> run 3!! • If config server goes down then: • no splits and migrates • new mongos cannot be started • running mongos can still use cache to route r/wTuesday 3 July 2012
  32. 32. Config Server in EC2 (1) Category/Impact Low Medium High Disk Speed x Disk Capacity x RAM x CPU xTuesday 3 July 2012
  33. 33. Config Server in EC2 (2) • Use Raid10 • Use 64 bit instance • Can run on shard serversTuesday 3 July 2012
  34. 34. Mongos in EC2 (1) Category/Impact Low Medium High Disk Speed Disk Capacity RAM x CPU xTuesday 3 July 2012
  35. 35. Mongos in EC2 (2) • Often run on application servers • 32 bit mongos ok with 64-bit mongodTuesday 3 July 2012
  36. 36. Arbiter in EC2 (1) Category/Impact Low Medium High Disk Speed x Disk Capacity x RAM x CPU xTuesday 3 July 2012
  37. 37. Arbiter in EC2 (2) • Can use micro instance • Elections may be slower • Can use instance store • Still want backups :)Tuesday 3 July 2012
  38. 38. HA in EC2 • Replica Sets • EC2 Availability ZonesTuesday 3 July 2012
  39. 39. DR in EC2 • Replica Sets • EC2 RegionsTuesday 3 July 2012
  40. 40. Security • Turn on authentication • Create a key between shards • EC2 Security Groups • Can reference other Security Groups • EC2 Regions • Follow SDLC in coding your appTuesday 3 July 2012
  41. 41. Monitoring • Links in with Cacti, Nagios, Munin-Node etc. • MMS - > it’s freeTuesday 3 July 2012
  42. 42. Mongo Ports • mongos, unsharded mongod : 27017 • sharded mongod : 27018 • config mongod : 27019 • http port : 28017Tuesday 3 July 2012
  43. 43. Some More EC2 NotesTuesday 3 July 2012
  44. 44. Instances Guidelines (1) • Use 64-bit only, 32-bit is not recommended • Primary/Secondary should be equal* • High CPU is not necessary • High Memory for large mongod instances • Network capacity is also IO capacity (EBS)Tuesday 3 July 2012
  45. 45. Instances Guidelines (2) • Note the trade-offs - memory/network • m1.large to m2.xlarge = 2x Mem, 0.5x Network • Do not use micro except for testing & config • m1.medium is usually sufficient for config DB • m1.small can be used for ArbitersTuesday 3 July 2012
  46. 46. Backups • EBS Snapshots - RAID complicates things • Single EBS volume, with journaling means: • No fsync & lock required • Similar applies to LVM snapshots • EC2 • GeneralTuesday 3 July 2012
  47. 47. EC2/MongoDB Best Practices • https://wiki.10gen.com/display/DOCS/Amazon+EC2 • https://wiki.10gen.com/display/DOCS/Production+NotesTuesday 3 July 2012
  48. 48. node.jsTuesday 3 July 2012
  49. 49. node.js • server-side written in javascript • orginally written for push web apps • created by Ryan DahlTuesday 3 July 2012
  50. 50. Sample node.js code var express = require(express), Db = require(mongodb).Db, Server = require(mongodb).Server, Connection = require(mongodb).Connection; var host = localhost; var port = Connection.DEFAULT_PORT; var db = new Db(node-mongo-examples, new Server(host, port, {}), {native_parser:false}); var app = express.createServer(); app.get(/, function(req, res){ res.send(Hello World); }); db.open(function(err, db) { if(err) throw err app.listen(8124); });Tuesday 3 July 2012
  51. 51. node.js with mongo • https://github.com/christkv/node-mongodb-native • http://www.nodebeginner.orgTuesday 3 July 2012
  52. 52. Need Help?Tuesday 3 July 2012
  53. 53. • MongoDB Google User Group • New MongoDB Docs & Old MongoDB Docs • Presentations • If you’re curious :) Image Source: http://www.cannotstartoutlook.com/wp-content/uploads/2012/06/outlook-problems-help.jpgTuesday 3 July 2012
  54. 54. Credits • Credit must go to @comerford, @jonnyeight & @mikefiedler as I borrowed some of the knowledgeable slides :)Tuesday 3 July 2012

×