Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to leverage what's new in MongoDB 3.6

48 views

Published on

A dense presentation of most of the new features in MongoDB 3.6 : Change streams, retryable writes, etc.

Published in: Software
  • Be the first to comment

  • Be the first to like this

How to leverage what's new in MongoDB 3.6

  1. 1. 1 2 APR IL , 2018 # M D B l o c a l How to leverage what’s new in MongoDB 3.6
  2. 2. # M D B l o c a l Maxime Beugnet Developer Advocate EMEA @MongoDB Paris Twitter : @MBeugnet Github : MaBeuLux88 Email : maxime@mongodb.com Meetups Hackathons Workshops Conferences
  3. 3. # M D B l o c a l MONGODB 3.6 THEMES
  4. 4. # M D B l o c a l To make your life easier…
  5. 5. # M D B l o c a l# M D B l o c a l Realtime Expressibili ty Analytics Application Availability Operations
  6. 6. # M D B l o c a l REALTIME CHANGE STREAMS
  7. 7. # M D B l o c a l# M D B l o c a l updatereplaceinsertdeleteMongoDB Application Action
  8. 8. # M D B l o c a l# M D B l o c a l db.coll.watch() db.coll.watch([{$match: {operationType: “insert”}}]) db.coll.watch([], {fullDocument:“updateLookup”}) db.coll.watch([], {resumeAfter:<cachedResumeToken>}) Change streams • $match • $project • $addFields • $replaceRoot • $redact
  9. 9. # M D B l o c a l# M D B l o c a l cursor = db.users.watch([ ], {"fullDocument":"updateLookup"}); while ( !cursor.isExhausted() ) { if ( cursor.hasNext() ) { print(tojson(cursor.next())); } } Change streams => cursors
  10. 10. # M D B l o c a l# M D B l o c a l •Resumable •Targeted Changes •Total ordering •Durability •Security •Ease of use •Idempotence Characteristics of Change Streams
  11. 11. # M D B l o c a l EXPRESSIBILITY More expressive query language Array updates
  12. 12. # M D B l o c a l# M D B l o c a l db.products.find( { $expr: { $gt: [ "$currRating" , "$prevRating" ] } } ) Comparing Fields within a document
  13. 13. # M D B l o c a l# M D B l o c a l db.products.find( { $expr: { $gt: [ {$subtract: ["$currRating" ,"$prevRating"] }, 2] } } ) Comparing Fields within a document
  14. 14. # M D B l o c a l# M D B l o c a l Comparing Fields within a document db.supplies.find( { $expr: { $lt:[ { $cond: { if: { $gte: ["$qty", 100] }, then: { $divide: ["$price", 2] }, else: { $divide: ["$price", 4] } } }, 5 ] } })
  15. 15. # M D B l o c a l# M D B l o c a l Expressive Array updates Update all matching items in an array Match Nested Arrays
  16. 16. # M D B l o c a l# M D B l o c a l Expressive Array updates MongoDB V3.4 • $ • $addToSet • $pop • $pull • $push • $pullAll { _id: 1, grades: [ { exam: 80, quizz: 75}, { exam: 85, quizz: 90} ] } db.students.update( { _id: 1, "grades.exam": 85 }, { $set: { "grades.$.quizz" : 95 } } )
  17. 17. # M D B l o c a l# M D B l o c a l Expressive Array updates MongoDB V3.6 • $[<id>] • $[] { _id: 1, grades: [ { exam: 80, quizz: 75}, { exam: 85, quizz: 90} ] } db.students.update( { }, { $set: { "grades.$[elem].quizz" : 100 } }, { multi: true, arrayFilters: [ { "elem.exam": { $gte: 80 } } ] } ) db.students.update( { }, { $set: { "grades.$[].quizz" : 100 } }, { multi: true } )
  18. 18. # M D B l o c a l# M D B l o c a l Expressive Array updates { _id: 1, name: "X", Medications: [ { id: 23, name: "DrugName99", Sched:"I", Rx: [ { id:13, Qty: 60, started: "2009-01-01" }, { id:77, Qty: 30, started: "2011-02-01", current:true } ]}, { id: 41, name: "OtherDrugName", Sched: "II, Rx: […] }, { id: 59, name: "ThirdDrug", Sched:"I", Rx:[ { id:994, Qty: 60, started: "2012-01-01", current:true }, { id:1034, Qty: 90, started: "2007-02-01" } ]}, "lastVisit": ISODate("2017-01-22T13:01:13.000Z") }
  19. 19. # M D B l o c a l# M D B l o c a l Expressive Array updates { _id: 1, name: "X", Medications: [ { id: 23, Sched:"I", Rx: [ { id:13, Qty: 60}, { id:77, Qty: 30, current:true} ]}, { id: 41, Sched: "II" }, { id: 59, Sched:"I", Rx:[ { id:994, Qty: 60, current:true}, { id:1034, Qty: 90} ]} ] } db.patientRx.update( {}, {"$set": {"Medications.$[med].Rx.$[rx].Qty": 20}}, { "multi": true, "arrayFilters": [ {"med.Sched": "I"}, {"rx.current":true, "rx.Qty":{$gt:30}} ]} )
  20. 20. # M D B l o c a l# M D B l o c a l Expressive Array updates { _id: 1, name: "X", Medications: [ { id: 23, Sched:"I", Rx: [ { id:13, Qty: 60}, { id:77, Qty: 30, current:true} ]}, { id: 41, Sched: "II" }, { id: 59, Sched:"I", Rx:[ { id:994, Qty: 60, current:true}, { id:1034, Qty: 90} ]} } db.patientRx.update( {"Medications": {"$elemMatch": { "Sched": "II", "Rx": {"$elemMatch": { "current": true, "Qty": {"$gt": 30} }} }}}, {"$set": {"Medications.$[med].Rx.$[rx].Qty": 20}}, { "multi": true, "arrayFilters": [ {"med.Sched": "I"}, {"rx.current":true, "rx.Qty":{$gt:30}} ]} )
  21. 21. # M D B l o c a l ANALYTICS New operators Timezone support Expressive $lookup R Driver BI Connector
  22. 22. # M D B l o c a l# M D B l o c a l New operators
  23. 23. # M D B l o c a l# M D B l o c a l New operators : $arrayToObject { "_id" : 1, dimensions: [ { "k": "l", "v": 25} , { "k": "w", "v": 10 }, { "k": "uom", "v": "cm" } ] } { "_id" : 2, dimensions: [ [ "l", 50 ], [ "w", 25 ], [ "uom", "cm" ] ] } { "_id" : 3, dimensions: [ [ "l", 50 ], [ "l", 25 ], [ "l", "cm" ] ] } db.inventory.aggregate( [ { $project: { dimensions: { $arrayToObject: "$dimensions" } } } ] ) { "_id" : 1, "dimensions" : { "l" : 25, "w" : 10, "uom" : "cm" } } { "_id" : 2, "dimensions" : { "l" : 50, "w" : 25, "uom" : "cm" } } { "_id" : 3, "dimensions" : { "l" : 50 } }
  24. 24. # M D B l o c a l# M D B l o c a l New operators : $objectToArray { $objectToArray: { item: "foo", qty: 25, size: { len: 25, w: 10, uom: "cm" } } } [ { "k" : "item", "v" : "foo" }, { "k" : "qty", "v" : 25 }, { "k" : "size", "v" : { "len" : 25, "w" : 10, "uom" : "cm" } } ]
  25. 25. # M D B l o c a l# M D B l o c a l New operators : $mergeObjects { $mergeObjects: [ { a: 1 }, { a: 2, b: 2 }, { a: 3, b: null, c: 3 } ] } { a: 3, b: null, c: 3 }
  26. 26. # M D B l o c a l# M D B l o c a l New operators : $dateFromString { _id: 1, date: "2017-02-08T12:10:40.787, timezone: "America/New_York" }, { _id: 2, date: "2017-02-08" , timezone: "-05:00" }, { _id: 3 } db.logmessages.aggregate( [ { $project: { date: { $dateFromString: { dateString: '$date', timezone: '$timezone’ } } } } ] ) { "_id" : 1, "date" : ISODate("2017-02-08T17:10:40.787Z") } { "_id" : 2, "date" : ISODate("2017-02-08T05:00:00Z") } { "_id" : 3, "date" : null }
  27. 27. # M D B l o c a l# M D B l o c a l New operators : $dateFromParts db.sales.aggregate([ { $project: { date: { $dateFromParts: { 'year' : 2017, 'month' : 2, 'day': 8, 'hour' : 12 } }, date_iso: { $dateFromParts: { 'isoWeekYear' : 2017, 'isoWeek' : 6, 'isoDayOfWeek' : 3, 'hour' : 12 } }, date_timezone: { $dateFromParts: { 'year' : 2016, 'month' : 12, 'day' : 31, 'hour' : 23, 'minute' : 46, 'second' : 12, 'timezone' : 'America/New_York’ } } } }]) { "_id" : 1, "date" : ISODate("2017-02-08T12:00:00Z"), "date_iso" : ISODate("2017-02-08T12:00:00Z"), "date_timezone" : ISODate("2017-01-01T04:46:12Z") }
  28. 28. # M D B l o c a l# M D B l o c a l New operators : $dateToParts db.sales.aggregate([ { $project: { date: { $dateToParts: { date: "$date" } }, date_iso: { $dateToParts: { date: "$date", iso8601: true } }, date_timezone: { $dateToParts: { date: "$date", timezone: "America/New_York" } } } }]) { "_id" : 2, "date" : { "year" : 2017, "month" : 1, "day" : 1, "hour" : 1, "minute" : 29, "second" : 9, "millisecond" : 123 }, "date_iso" : { "isoWeekYear" : 2016, "isoWeek" : 52, "isoDayOfWeek" : 7, "hour" : 1, "minute" : 29, "second" : 9, "millisecond" : 123 }, "date_timezone" : { "year" : 2016, "month" : 12, "day" : 31, "hour" : 20, "minute" : 29, "second" : 9, "millisecond" : 123 } }
  29. 29. # M D B l o c a l# M D B l o c a l Timezone support $dayOfYear, $dayOfMonth, $dayOfWeek, $year, $month, $week, $hour, $minute, $second, $millisecond, $isoDayOfWeek, $isoWeek, $isoWeekYear { $operator: { date: <isoDateExpression>, timezone: <tzExpression> } }
  30. 30. # M D B l o c a l# M D B l o c a l >db.collection.aggregate({$lookup:{ from:"coll2", localField:"x", foreignField:"y", as:"coll2details" }}]) More expressive $lookup >db.collection.aggregate({$lookup:{ from:"coll2", let: {x: "$x"}, pipeline: [ {$match: {$expr: {$eq: [ "$y", "$$x"] } } } ] as:"coll2details" }}]) Prior to 3.6 New in 3.6
  31. 31. # M D B l o c a l# M D B l o c a l $lookup example orders: { line_items : [ { id: 123, title : “USB Battery”, price: 15.0 }, { id: 512, title : “Hip T-shirt”, price : 45.0 } ] } db.orders.aggregate([ {$unwind: "$line_items"}, {$lookup:{ from: "reviews", let: {p_id: "$line_items.id"}, pipeline: [ {$match: {$expr: {$eq: ["$p_id","$$p_id"]}}}, {$group: { _id: 1, rating: {$avg:"$rating"}}} ], as: "avgRating" } } ])
  32. 32. # M D B l o c a l# M D B l o c a l Recommended MongoDB R driver for data scientists, developers & statisticians • MongoDB read & write concerns to control data consistency & durability • Idiomatic, native language access to the database • Data security with enterprise authentication mechanisms • BSON data type support, e.g., Decimal 128 for high precision scientific & financial analysis R Driver for MongoDB
  33. 33. # M D B l o c a l# M D B l o c a l MongoDB Connector for BI + many more
  34. 34. # M D B l o c a l# M D B l o c a l MongoDB Connector for BI • Faster • Takes advantage of 3.6 expressive Lookup to push more computation into the database • Supports Show Status function to enable deeper performance optimization • Simpler • Lifecycle management with Ops Manager • Schema sampling and mapping now managed by the mongosqld process, rather than separate utility (mongodrdl) • Authenticate via client-side plugins, rather than managing TLS certificates. Kerberos support added
  35. 35. # M D B l o c a l APPLICATIONS AVAILABILITY Retryable writes DNS seed list Read Concern: Available & Tunable consistency
  36. 36. # M D B l o c a l# M D B l o c a l Retryable writes PS S Application P write successful :D write unsuccessful :O
  37. 37. # M D B l o c a l# M D B l o c a l Retryable writes Application MongoDB { _id: 1, team : "Manchester United", gameID: "game123", coach : "José Mourinho", score : 2, league: "Premier League" } { _id: 2, team : "Chelsea", gameID: "game456", coach : "Antonio Conte", score : 2, league: "Premier League" } db.games.update( {gameID:"game123"}, {$inc: {score:1}} ) db.games.update( {team: "Chelsea"}, {$set: {coach:"Zidane"}} )
  38. 38. # M D B l o c a l# M D B l o c a l • Automatic Drivers logic • Network errors • Elections • NOT for logic errors • Safe • For both non-idempotent and idempotent writes • NOT for multi: true Characteristics of Retryable Writes
  39. 39. # M D B l o c a l# M D B l o c a l Retryable writes uri = "mongodb://example.com:27017/?retryWrites=true" client = MongoClient(uri) database = client.database collection = database.collection
  40. 40. # M D B l o c a l# M D B l o c a l DNS seed list "mongodb://example1.com:27017,example2.com:27017,example3.com:27017 "mongodb+srv://my-dns-server.mongodb.com "mongodb://example1.com:27017
  41. 41. # M D B l o c a l# M D B l o c a l Tunable consistency Availability Consistencymagic
  42. 42. # M D B l o c a l# M D B l o c a l • readConcern (Read Isolation) • Local • Majority • Linearizable • writeConcern (write acknowledgement) • <number> (i.e. 1) • Majority • Tag What is readConcern and writeConcern?
  43. 43. # M D B l o c a l# M D B l o c a l Tunable consistency mongos 3 1 2 Shard 1 Shard 2 Shard 3 readConcern: available
  44. 44. # M D B l o c a l# M D B l o c a l Tunable consistency readConcern: available is equivalent to readConcern: local on replica sets you can pass readConcern: available to a primary in a sharded cluster readConcern: available is default for secondaries in a sharded cluster Secondaries in sharded clusters will now respect readConcern : local for safe reads
  45. 45. # M D B l o c a l# M D B l o c a l Tunable consistency mongos 3 1 2 Shard 1 Shard 2 Shard 3 Global Logical Clock wait until cluster time has moved past the last time you saw Causal consistency: guarantees monotonic, logically consistent reads from any replica node in the same user session
  46. 46. # M D B l o c a l# M D B l o c a l Tunable consistency //start client session, which is causally consistent by default try (ClientSession session = client.startSession(ClientSessionOptions.builder().build())) { //Run causally related operations within the session collection.insertOne(session, ... ); collection.updateOne(session, ...); try (MongoCursor<Document> cursor = collection.find(session).filter(...).iterator()) { while (cursor.hasNext()) { Document cur = cursor.next(); } }
  47. 47. # M D B l o c a l OPERATIONS JSON Schema Network security Session management End-to-end Compression
  48. 48. # M D B l o c a l# M D B l o c a l Network Security Bind to localhost by Default IP Whitelisting • Associate IP addresses or ranges with roles in auth • If a the IP restrictions are not met, fail to authenticate • Able to restrict __system user to authenticate from only cluster nodes
  49. 49. # M D B l o c a l# M D B l o c a l Network Security 192.168.1.25 Application Application System Administrator 192.168.1.48 172.16.4.13 172.16.4.88 172.33.20.62 172.33.20.11 Restrict each user’s authentication based on: • Client IP Address Range and/or • Server IP Listen Address
  50. 50. # M D B l o c a l# M D B l o c a l JSON Schema Enforces strict schema structure over a complete collection for data governance & quality • Builds on document validation introduced by restricting new content that can be added to a document • Enforces presence, type, and values for document content, including nested array • Simplifies application logic Tunable: enforce document structure, log warnings, or allow complete schema flexibility Queryable: identify all existing documents that do not comply
  51. 51. # M D B l o c a l# M D B l o c a l JSON Schema db.createCollection( "orders", {validator: {$jsonSchema: {properties: {line_items: {type: "array", items: {properties: {title: {type: "string"}, price: {type: "number", minimum: 0.0} }, required: ["_id", "title", "price"], additionalProperties: false}}}, required: ["line_items"]}}} ) http://json-schema.org/
  52. 52. # M D B l o c a l# M D B l o c a l Session Management Server sessions • Every Operation is wrapped in a server session by default in 3.6 • killSessions by user Client sessions • Every operation within a defined client session have causal consistency • Not by default, must be explicitly defined
  53. 53. # M D B l o c a l# M D B l o c a l MongoDB 3.6 adds compression of wire protocol traffic between client and database • Up to 80% bandwidth savings MongoDB End to End Compression • Wire protocol • Intra-cluster • Indexes in memory • Storage Application MongoDB Primary Replica Wire Protocol Compression MongoDB Secondary Replica Single ViewMongoDB Secondary Replica Single ViewMongoDB Secondary Replica Single ViewMongoDB Secondary Replica Single ViewMongoDB Secondary Replica MongoDB Secondary Replica Intra-Cluster Compression Compression of Data on Disk Compression of Indexes in Memory End to End Compression
  54. 54. # M D B l o c a l
  55. 55. # M D B l o c a l THANK YOU!

×