DMDW Extra Lesson - NoSql and MongoDB


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

DMDW Extra Lesson - NoSql and MongoDB

  1. 1. STUDIEREN<br />UND DURCHSTARTEN.<br />Author: Dip.-Inf. (FH) Johannes Hoppe<br />Date: 06.05.2011 <br />
  2. 2. NoSQL and MongoDB<br />Author: Dip.-Inf. (FH) Johannes Hoppe<br />Date: 06.05.2011 <br />
  3. 3. 01<br />Not only SQL<br />3<br />
  4. 4. Trends<br />4<br />
  5. 5. Trends<br />Data<br />Facebook had 60k servers in 2010<br />Google had 450k servers in 2006 (speculated)<br />Microsoft: between 100k and 500k servers (since Azure)<br />Amazon: likely has a similar numbers, too (S3)<br />Facebook Server Footprint<br />5<br />
  6. 6. Trends<br />Trend 1: increasing data sizes<br />Trend 2: more connectedness (“web 2.0”)<br />Trend 3:moreindividualization (feverstructure)<br />6<br />
  7. 7. NoSQL<br />7<br />
  8. 8. NoSQL<br />Database paradigms<br />Relational (RDBMS)<br />NoSQL<br />Key-Value stores<br />Document databases<br />Wide column stores (BigTable and clones)<br />Graph databases<br />Other<br />8<br />
  9. 9. NoSQL<br />Some NoSQL use cases<br />1. Massive data volumes<br />Massively distributed architecture required to store the data<br />Google, Amazon, Yahoo, Facebook…<br />2. Extreme query workload<br />Impossible to efficiently do joins at that scale with an RDBMS<br />3. Schema evolution<br />Schema flexibility (migration) is not trivial at large scale<br />Schema changes can be gradually introduced with NoSQ<br />9<br />
  10. 10. NoSQL - CAP theorem<br />Requirements for distributed systems:<br />Consistency<br />Availability<br />Partition tolerance<br />10<br />
  11. 11. NoSQL - CAP theorem<br />Consistency<br />The system is in a consistent state after an operation<br />All clients see the same data<br />Strong consistency (ACID)vs. eventual consistency (BASE)<br />ACID: Atomicity, Consistency, Isolation and Durability<br />BASE: Basically Available, Soft state, Eventually consistent<br />11<br />
  12. 12. NoSQL - CAP theorem<br />Availability<br />The system is “always on”, no downtime<br />Node failure tolerance– all clients can find some available replica<br />Software/hardware upgrade tolerance<br />12<br />
  13. 13. NoSQL - CAP theorem<br />Partition tolerance<br />The system continues to function even when <br />Split into disconnected subsets (by a network disruption)<br />Not only for reads, but writes as well!<br />13<br />
  14. 14. NoSQL<br />CAP Theorem<br />E. Brewer, N. Lynch<br />You can satisfyat most 2 out of the 3 requirements<br />14<br />
  15. 15. NoSQL<br />CAP Theorem  CA<br />Single site clusters(easier to ensure all nodes are always in contact)<br />When a partition occurs, the system blocks<br />e.g. usable for two-phase commits (2PC) which already require/use blocks <br />15<br />
  16. 16. NoSQL<br />CAP Theorem  CA<br />Single site clusters(easier to ensure all nodes are always in contact)<br />When a partition occurs, the system blocks<br />e.g. usable for two-phase commits (2PC) which already require/use blocks <br />Obviously, any horizontal scaling strategy is based on data partitioning; therefore, designers are forced to decide between consistency and availability.<br />16<br />
  17. 17. NoSQL<br />CAP Theorem  CP<br />Some data may be inaccessible (availability sacrificed), but the rest is still consistent/accurate<br />e.g. sharded database<br />17<br />
  18. 18. NoSQL<br />CAP Theorem  AP<br />System is still available under partitioning,but some of the data returned my be inaccurate<br />Need some conflict resolution strategy<br />e.g. Master/Slave replication<br />18<br />
  19. 19. NoSQL<br />RDBMS<br />Guaratnee ACID by CA(two-phasecommits)<br />SQL<br />Mature:<br />19<br />
  20. 20. NoSQL<br />NoSQL DBMS<br />No relational tables<br />No fixed table schemas<br />No joins<br />No risk, no fun!<br />CP and AP<br /> (and sometimes even AP and on top of CP  MongoDB*)<br />* This is damn cool!<br />20<br />
  21. 21. NoSQL<br />Key-value<br />One key  one value, very fast<br />Key: Hash (no duplicates)<br />Value: binary object („BLOB“)<br /> (DB does not understand your content)<br />Players: Amazon Dynamo, Memcached…<br />21<br />
  22. 22. NoSQL<br />key<br />value<br />?=PQ)“§VN? =§(Q$U%V§W=(BN W§(=BU&W§$()= W§$(=%<br />GIVE ME A MEANING!<br />customer_22<br />22<br />
  23. 23. NoSQL<br />Document databases<br />Key-value store, too<br />Value is „understood“ by the DB<br />Querying the data is possible(not just retrieving the key‘s content)<br />Players: Amazon SimpleDB, CouchDB, MongoDB …<br />23<br />
  24. 24. NoSQL<br />key<br />value<br />{<br /> Type: “Customer”,<br /> Name: "Norbert“,<br />Invoiced: 2222 <br />}<br />customer_22<br />24<br />
  25. 25. NoSQL<br />key<br />value / documents<br />{ <br /> Type: "Customer",<br /> Name: "Norbert",<br /> Invoiced: 2222<br /> Messages: [<br /> { Title: "Hello",<br /> Text: "World" },<br /> { Title: "Second",<br /> Text: "message" }<br /> ] <br />}<br />customer_22<br />25<br />
  26. 26. NoSQL<br />(Wide) column stores<br />Often referred as “BigTable clones”<br />Each key is associated with many attributes (columns)<br />NoSQL column stores are actually hybrid row/column stores<br />Different from “pure” relational column stores!<br />Players: Google BigTable, Cassandra (Facebook), HBase…<br />26<br />
  27. 27. NoSQL<br />Won‘t be stored as: It will be stored as:<br />22;Norbert;22222 22;23;24<br />23;Hans;50000 Norbert;Hans;Franz<br />24;Franz;44000 22222;50000;44000<br />27<br />
  28. 28. NoSQL<br />Graph databases<br />Multi-relational graphs<br />SPARQL query language (W3C Recommendation!)<br />Players: Neo4j, InfoGrid …<br />(note: graph DBs are special and somehow the “black sheep” in the NoSQL world –the following PROs/CONs don’t apply very well)<br />28<br />
  29. 29. NoSQL<br />PROs (& Promisses)<br />Scheme-free / semi-structured data<br />Massive data stores<br />Scaling is easy<br />Very, very high availability<br />Often simpler to implement<br /> (and OR Mappers aren’t required)<br />„Web 2.0 ready“<br />29<br />
  30. 30. NoSQL<br />CONSs<br />NoSQL implementations often „alpha“, no standards<br />Data consistency, no transactions,<br />Insufficient access control<br />SQL: strong for dynamic, cross-table queries (JOIN)<br />Relationships aren‘t enforced<br /> (conventions over constrains – except for graph DBs (of course))<br />Premature optimization: Scalability<br /> (Don’t build for scalability if you never need it!)<br />30<br />
  31. 31. 02<br />MongoDB<br />31<br />
  32. 32. NoSQL<br /> Lets rock! <br />MongoDB Quick Reference Cards<br /><br />32<br />
  33. 33. Basic Deployment<br />Create the default data directory in c:datadb<br />Start mongod.exe<br />Optionally: mongod.exe --dbpath c:datadb --port 27017 --logpath c:datamongodb.log<br />Start the shell: mongo.exe<br />33<br />
  34. 34. Data Import<br />cd c:dba-training-datadata<br />mongoimport -d twitter -c tweets twitter.json<br />cd c:dba-training-datadatadumptraining<br />mongorestore -d training -c scores scores.bson<br />cd c:dba-training-datadatadump<br />mongorestore -d diggdigg<br />34<br />
  35. 35. 35<br />
  36. 36. MongoDB Documents<br />(in the shell)<br />use digg<br />db.stories.findOne();<br />36<br />
  37. 37. JSON  BSON<br />All JSON documents are stored in a binary format called BSON. BSON supports a richer set of types than JSON.<br /><br />37<br />
  38. 38. CRUD – Create<br />(in the shell)<br />{name: 'Smith', age: 30});<br />See how the save command works:<br /><br />38<br />
  39. 39. CRUD – Create<br />How training.scores was created:<br />for(i=0; i<1000; i++) {<br /> ['quiz', 'essay', 'exam'].forEach(function(name) {<br />var score = Math.floor(Math.random() * 50) + 50;<br />{student: i, name: name, score: score});<br /> });<br /> }<br />db.scores.count();<br />39<br />
  40. 40. CRUD – Read<br />Queries are specified using a document-style syntax!<br />use training<br />db.scores.find({score: 50});<br />db.scores.find({score: {"$gte": 70}});<br />db.scores.find({score: {"$gte": 70}});<br />Cursor!<br />40<br />
  41. 41. Exercises<br />Find all scores less than 65. <br />Find the lowest quiz score. Find the highest quiz score. <br />Write a query to find all digg stories where the view count is greater than 1000. <br />Query for all digg stories whose media type is either 'news' or 'images' and where the topic name is 'Comedy’.(For extra practice, construct two queries using different sets of operators to do this. )<br />Find all digg stories where the topic name is 'Television' or the media type is 'videos'. Skip the first 5 results, and limit the result set to 10.<br />41<br />
  42. 42. CRUD – Update<br />use digg; <br />db.people.update({name: 'Smith'}, {'$set': {interests: []}});<br />db.people.update({name: 'Smith'}, {'$push': {interests: ['chess']}});<br />42<br />
  43. 43. Exercises<br />Set the proper 'grade' attribute for all scores. For example, users with scores greater than 90 get an 'A.' Set the grade to ‘B’ for scores falling between 80 and 90.<br />You're being nice, so you decide to add 10 points to every score on every “final” exam whose score is lower than 60. How do you do this update?<br />43<br />
  44. 44. CRUD – Delete<br />db.dropDatabase();<br />;<br />;<br />44<br />
  45. 45. “Map Reduce is the Uzi of aggregation tools. Everything described with count, distinct and group can be done with MapReduce, and more.”<br />Kristina Chadorow, Michael Dirolf in MongoDB – The Definitive Guide<br />45<br />
  46. 46. MapReduce<br />To use map-reduce, you first write a map function.<br />map = function() {<br /> emit(, {diggs: this.diggs, posts: 0});<br />}<br />46<br />
  47. 47. MapReduce<br />The reduce functions then aggregation those docs by key.<br />reduce = function(key, values) {<br />vardiggs = 0;<br />var posts = 0;<br />values.forEach(function(doc) {<br />diggs += doc.diggs;<br /> posts += 1;<br /> });<br /> return {diggs: diggs, posts: posts};<br />}<br />47<br />
  48. 48. MapReduce<br />Now both are used to perform custom aggregation.<br />db.stories.mapReduce(map, reduce, {out: 'digg_users'});<br />48<br />
  49. 49. THANK YOU<br />FOR YOUR ATTENTION<br />49<br />
  50. 50. Farben Primär<br />
  51. 51. Farben Code<br />