More Related Content Similar to MongoDB World 2018: Using Change Streams to Keep Up with Your Data (20) MongoDB World 2018: Using Change Streams to Keep Up with Your Data8. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:00
Stillw
aiting
7:15
Stillw
aiting
9. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:15
Stillw
aiting
7:00
Stillw
aiting
12. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
8:00
A
nnouncem
ent
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting
7:45
Stillw
aiting
17. const changeStream =
db.collection('train').watch();
c h a n g e S t r e a m . o n ( ' c h a n g e ' , ( c h a n g e ) = > { c o n s o l e . l o g ( c h a n g e ) } ) ;
Added in 3.6
Improved in 4.0
{
"_id" : (resumeToken),
"operationType" : "insert",
"ns" : { "db" : "test", "coll" : "train" },
"documentKey" : {
"_id" : 123
},
"fullDocument" : {
"_id" : 123,
"text": "hello"
},
}
19. 1. Use Collection Access Controls
2. Present a Defined API
3. Scale Across Nodes
Change Streams
24. Change Streams are Resumable
collection.watch()
PP
{
_id: <resumeToken>,
operationType: 'update'
...
}
SP
S
25. Change Streams are Resumable
changeStream.on ('change', (change) => {
console.log(change);
cachedResumeToken = change["_id"];
});
changeStream.on ('error', () => {
if (cachedResumeToken) {
establishChangeStream (cachedResumeToken);
}
});
26. Change Streams Utilize the Power of the
Aggregation Framework
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([{ $match: {operationType: {$in: ['delete', 'replace']}}}]);
27. 1. Collection Access Controls
2. Defined API
3. Enable Scaling
4. Total Ordering
5. Durable
6. Resumable
7. Power of Aggregation
30. Update
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“text”: “updated”} })
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "update",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"updateDescription" : {
"updatedFields" : {
"text" : "updated"
},
"removedFields" : [ ]
}
}
31. Update with fullDocument: updateLookup
> db.collection.watch([],
{ fullDocument: “updateLookup”,
maxAwaitTimeMS: 30000 }).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“number”: “5”} })
Shell 1 Shell 2
{ "_id" : (resumeToken),
"operationType" : "update",
"fullDocument" : {
"_id" : 1,
"text" : "updated",
"number" : 5
},
"ns": {"db": "test", "coll": "collection"},
"documentKey" : { "_id" : 1 },
"updateDescription" : {
"updatedFields" : { "number" : 5 },
"removedFields" : [ ]
}
}
41. _id: 11 (40.0, -74.0)
db.train.find().readConcern("majority")
App Cache
Populate the Cache
42. _id: 11 (40.0, -74.0)
db.train.watch([])
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: { lat: 40.41169 }
},
documentKey: {_id: 11}
Handled
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Handled?
App Cache _id: 11 (40.4, -74.0)
Watch for changes
46. db.train.watch([{
$match: {
$or: [
{
operationType: 'update',
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
},
{ operationType: { $in: ['delete', 'insert', 'replace'] } }
]
}
}])
App Cache _id: 11 (40.4, -74.0)
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Handled
_id: 16 (40.4, -74.0)
_id: (resumeToken),
operationType: 'delete',
documentKey: {_id: 11}Handled
47. Race Between Populating and Listening!
db.train.find().readConcern("majority")
.
.
.
db.train.watch()
update occurs!
49. startAtOperationTime
const session = client.startSession();
const cursor = db.collection('train').find({}, { session: session });
populateApplicationCache(cursor);
const changeStream = db.collection('train').watch([], {
session: session,
startAtOperationTime: session.operationTime
});
changeStream.on('change', updateApplicationCache);