The document discusses how change streams in MongoDB can be used to keep application data in real-time by watching for changes to documents in a collection. Change streams provide a defined API and enable scaling across nodes while ensuring total ordering of changes. They allow applications to populate an initial cache of documents and then receive notifications of inserts, updates, replaces, deletes, and invalidates in order to maintain an up-to-date cache. The document provides examples of filtering change streams and handling different operation types.

1. Using Change Streams to
Keep Up With Your Data
Kevin Albertson, Software Engineer II

2. Kevin Albertson
Software Engineer II, MongoDB
@kevinAlbs

3. Why Real-Time Data?

4. March 8th, 2018

5. March 8th, 2018
Point A
Point B

8. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:00
Stillw
aiting
7:15
Stillw
aiting

9. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:15
Stillw
aiting
7:00
Stillw
aiting

10. 6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting

11. 6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting
7:45
Stillw
aiting

12. March 8th, 2018
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
8:00
A
nnouncem
ent
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Stillw
aiting
7:45
Stillw
aiting

13. March 8th, 2018

14. Unnecessary
6:33 6:45 6:54
Tw
eet:everything
ok
Iarrive
atstation
Train
scheduled
8:00
A
nnouncem
ent
7:30
Stillw
aiting
7:15
Stillw
aiting
7:00
Train
passes
7:45
Stillw
aiting
Arrive
in
N
YC
9:30
Take
train
in
otherdirection
9:30
Take
train
to
N
YC 11:00

15. Real-time is an Expectation

16. Change Streams Enable Existing Apps to
Provide Real-time Data

17. const changeStream =
db.collection('train').watch();
c h a n g e S t r e a m . o n ( ' c h a n g e ' , ( c h a n g e ) = > { c o n s o l e . l o g ( c h a n g e ) } ) ;
Added in 3.6
Improved in 4.0
{
"_id" : (resumeToken),
"operationType" : "insert",
"ns" : { "db" : "test", "coll" : "train" },
"documentKey" : {
"_id" : 123
},
"fullDocument" : {
"_id" : 123,
"text": "hello"
},
}

18. Change Stream
Characteristics

19. 1. Use Collection Access Controls
2. Present a Defined API
3. Scale Across Nodes
Change Streams

20. Return 5 Operation Types
1. Insert
2. Update
3. Replace
4. Delete
5. Invalidate

21. mongos
3 2
1
Shard 1 Shard 2 Shard 3
Total Ordering of Changes Across Shards

22. Documents Uniquely Identified
Shard key combined with _id to uniquely identify documents
mongos
Shard 2 Shard 3Shard 1

23. Changes are Durable
S
S
collection.watch()
P

24. Change Streams are Resumable
collection.watch()
PP
{
_id: <resumeToken>,
operationType: 'update'
...
}
SP
S

25. Change Streams are Resumable
changeStream.on ('change', (change) => {
console.log(change);
cachedResumeToken = change["_id"];
});
changeStream.on ('error', () => {
if (cachedResumeToken) {
establishChangeStream (cachedResumeToken);
}
});

26. Change Streams Utilize the Power of the
Aggregation Framework
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([{ $match: {operationType: {$in: ['delete', 'replace']}}}]);

27. 1. Collection Access Controls
2. Defined API
3. Enable Scaling
4. Total Ordering
5. Durable
6. Resumable
7. Power of Aggregation

28. Types of Changes

29. Insert
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.insert({
_id: 1,
text: "hello"
})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "insert",
"ns" : {
"db" : "test",
"coll" : "test"
},
"documentKey" : {
"_id" : 1
},
"fullDocument" : {
"_id" : 1,
"text": "hello"
},
}

30. Update
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“text”: “updated”} })
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "update",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"updateDescription" : {
"updatedFields" : {
"text" : "updated"
},
"removedFields" : [ ]
}
}

31. Update with fullDocument: updateLookup
> db.collection.watch([],
{ fullDocument: “updateLookup”,
maxAwaitTimeMS: 30000 }).pretty()
> db.collection.updateOne({_id: 1},
{ $set: {“number”: “5”} })
Shell 1 Shell 2
{ "_id" : (resumeToken),
"operationType" : "update",
"fullDocument" : {
"_id" : 1,
"text" : "updated",
"number" : 5
},
"ns": {"db": "test", "coll": "collection"},
"documentKey" : { "_id" : 1 },
"updateDescription" : {
"updatedFields" : { "number" : 5 },
"removedFields" : [ ]
}
}

32. Replace
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.replaceOne({_id: 1},
{text: "replaced"})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "replace",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
},
"fullDocument" : {
"_id" : 1,
"text" : "replaced"
}
}

33. Delete
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.remove({_id: 1})
Shell 1 Shell 2
{
"_id" : (resumeToken),
"operationType" : "delete",
"ns" : {
"db" : "test",
"coll" : "collection"
},
"documentKey" : {
"_id" : 1
}
}

34. Invalidate
> db.collection.watch([],
{maxAwaitTimeMS: 30000}).pretty()
> db.collection.drop()
Shell 1 Shell 2
{
"_id" : (token),
"operationType" : "invalidate"
}

35. New in 4.0

36. 1. startAtOperationTime
2. Change stream on db or client
3. Resumable before first change

37. Use Case

38. Real-time Train Arrival Estimates

39. Collection
> db.train.find().pretty()
{
"_id" : 11,
"position" : {
"lat" : 40.04714,
"lon" : -74.0351
}
}

40. App Caches State of All Trains
App Cache

41. _id: 11 (40.0, -74.0)
db.train.find().readConcern("majority")
App Cache
Populate the Cache

42. _id: 11 (40.0, -74.0)
db.train.watch([])
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: { lat: 40.41169 }
},
documentKey: {_id: 11}
Handled
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Handled?
App Cache _id: 11 (40.4, -74.0)
Watch for changes

43. $match to Filter Irrelevant Changes

44. db.train.watch([{
$match: {
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
}
}])
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Ignored?
_id: (resumeToken),
operationType: 'update',
updateDescription: {
updatedFields: {
passengerCount: 5
}
}
documentKey: {_id: 11}
Ignored
App Cache _id: 11 (40.4, -74.0)

45. Handle All Document Changes
If Caching State of Documents:

46. db.train.watch([{
$match: {
$or: [
{
operationType: 'update',
$or: [
{'updateDescription.updatedFields.lat': {$exists: 1}},
{'updateDescription.updatedFields.lon': {$exists: 1}}
]
},
{ operationType: { $in: ['delete', 'insert', 'replace'] } }
]
}
}])
App Cache _id: 11 (40.4, -74.0)
_id: (resumeToken),
operationType: 'insert',
fullDocument: {
_id: 16,
lat: 40.5889, lon: -76.1938
},
documentKey: {_id: 16}
Handled
_id: 16 (40.4, -74.0)
_id: (resumeToken),
operationType: 'delete',
documentKey: {_id: 11}Handled

47. Race Between Populating and Listening!
db.train.find().readConcern("majority")
.
.
.
db.train.watch()
update occurs!

48. find
Client Server
[ {_id: 11}…], $clusterTime: X
watch, startAtOperationTime: X
start session
startAtOperationTime

49. startAtOperationTime
const session = client.startSession();
const cursor = db.collection('train').find({}, { session: session });
populateApplicationCache(cursor);
const changeStream = db.collection('train').watch([], {
session: session,
startAtOperationTime: session.operationTime
});
changeStream.on('change', updateApplicationCache);

50. App Complete!
1. Doesn't miss updates
2. Handles network blips
3. Reduces bandwidth
4. Easy to prototype