Using Change Streams to Keep Up with Your Data

#MDBlocal
Using Change Streams to
Keep Up With Your Data
Asya Kamsky, Lead Product Manager
#MongoDB @MDBLocal @asya999 #askAsya

#MDBlocal
“TIME IS MONEY.”
B E N J A M I N F R A N K L I NW A R R E N B U F F E T

#MDBlocal
REAL-TIME IS AN EXPECTATION

#MDBlocal
MongoD
B
Action
Handler

#MDBlocal
const changeStream=
db.collection('temperature').watch();
c h a n g e S t r e a m . o n ( ' c h a n g e ' , f u n c t i o n ( c h a n g e ) {
c o n s o l e . l o g ( c h a n g e ) ;
} ) ;

#MDBlocal
CHANGE STREAM
CHARACTERISTICS

#MDBlocal
5 OPERATION TYPES
INSERT
DELETE
REPLACE
UPDATE
INVALIDATE

#MDBlocal
CHANGE STREAMS PRESENT A DEFINED
API

#MDBlocal
API, UTILIZE COLLECTION ACCESS
CONTROLS

#MDBlocal
CONTROLS, SHOW ONLY DURABLE
CHANGES

#MDBlocal
CONTROLS, SHOW ONLY DURABLE
CHANGES, AND ENABLE SCALING
ACROSS ALL YOUR NODES.

#MDBlocal
TOTAL ORDERING OF CHANGES
ACROSS SHARDS
mongos
3 1
2Shard
1
Shard
2
Shard
3

#MDBlocal
PS
CHANGES ARE DURABLE
P
S

#MDBlocal
SP
CHANGE STREAMS ARE RESUMABLE
P
S
{_id: <resumeToken>,
operationType: ‘update’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘room12345’},
updateDescription: {updatedFields: {temperature: 70},
removedFields: [‘humidity’]}
}
var newChangeStream = coll.watch({ resumeAfter: <cachedResumeToken> });

#MDBlocal
CHANGE STREAMS UTILIZE THE POWER
OF THE AGGREGATION FRAMEWORK
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([{ $match: {$or: [{operationType: 'delete' }, {operationType:'replace'}]}}]);

#MDBlocal
1. DEFINED API
2. COLLECTION ACCESS CONTROLS
3. ENABLE SCALING
4. TOTAL ORDERING
5. DURABLE
6. RESUMABLE
7. POWER OF AGGREGATION

#MDBlocal
DELETEREPLACEUPDATEINVALIDATE5 OPERATION TYPES
{ _id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")
},
fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"),
userName: ‘alice123’,
name: ‘Alice’
}
}
INSERT
{_id: (resumeToken),
operationType: ‘delete’,
}
}
operationType: ‘replace’,
},
fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"),
name: ‘Alice’
}
}
},
updateDescription: {updatedFields: {email: ‘alice@10gen.com’},
removedFields: [‘phoneNumber’]}
}
operationType: ‘invalidate’,
ns: {db: ‘test’, coll: ‘foo’}}

#MDBlocal
_id: ObjectId("58a4eb4a30c75625e00d2820")},
removedFields: [‘phoneNumber’]}
}
UPDATE
var changeStream = coll.watch({ fullDocument: 'updateLookup' });
UPDATE fullDocument: updateLookup
_id: ObjectId("58a4eb4a30c75625e00d2820")},
removedFields: [‘phoneNumber’]},
fullDocument: { _id: ObjectId("58a4eb4a30c75625e00d2820"),
name: ‘Alice’,
email: ‘alice@10gen.com’,
team: ‘replication’}
}

#MDBlocal
1 . S H AR D K E Y I S U S E D ( B Y T H E M O N G O S ) T O R O U T E
O P E R AT I O N S
2 . _ I D I S U S E D T O U N I Q U E L Y I D E N T I F Y A D O C U M E N T
mongos
3 1
2Shard
1
Shard
2
Shard
3
DEFINING THE documentKey

#MDBlocal
BACK TO
THE USE CASE

#MDBlocal
operationType: 'insert',
ns: {db: 'data', coll: 'temperature'},
documentKey: {_id: 'orangeRoom123'},
fullDocument: {_id: 'orangeRoom123',
temperature: 76,
username: 'eliot_horowitz' }
}
operationType: 'update',
documentKey: {_id: 'greenRoom123'},
updateDescription: {updatedFields: {temperature: 68},
fullDocument: {_id: 'greenRoom123',
temperature: 68,
username: 'aly_cabral'}
}
temperature: 75,
}
Action Handler
var changeStream = coll.watch(
[{$match: {'fullDocument.temperature': {$gte : 70 }}}],
{fullDocument: 'updateLookup'});
temperature: 75
username: aly_cabral

#MDBlocal
IF YOUR APPLICATION REQUIRES STATE
ALWAYS MATCH ON UNCHANGING
FIELDS

#MDBlocal
operationType: 'replace',
documentKey: {_id: 'blueRoom123'},
fullDocument: {_id: 'blueRoom123',
temperature: 71,
username: 'asya_kamsky'}
}
fullDocument: {_id: 'blueRoom123',
temperature: 85,
}
Action Handler
var changeStream = coll.watch([{ $match: {'fullDocument.username': 'aly_cabral'}}],
temperature: 75
temperature: 85
operationType: 'delete',
documentKey: {_id: 'blueRoom123'}
}

#MDBlocal
IF YOUR APPLICATION NEEDS TO NOTIFY ON DELETED
DATA
ALWAYS HANDLE DELETES AND
REPLACES APPROPRIATELY

#MDBlocal
var changeStream = coll.watch([{ $match: {$or:
[ { 'fullDocument.username': 'aly_cabral' },
{ operationType: 'delete' },
{ operationType: 'replace' }
]}}],
Insert Update Delete
In the update notification fullDocument: null
updateLookup
operationType: 'update',
documentKey: {userName: 'greenRoom123'},
updateDescription: {updatedFields: {temperature: 78}},
fullDocument: null
}

#MDBlocal
IF YOU NEED TO SEE EVERY CHANGE (EVEN OUTDATED
CHANGES) TO A DOCUMENT
HAVE MATCHING FIELDS IN THE
DOCUMENT KEY

#MDBlocal
documentKey: {_id: 'blueRoom123',
username: 'aly_cabral'},
fullDocument: {_id: ‘blueRoom123’,
temperature: 85,
}
documentKey: {_id: 'blueRoom123',
username: 'aly_cabral'}}
documentKey: {_id: 'greenRoom123',
username: 'aly_cabral'},
temperature: 71,
username: “asya_kamsky”}
}
Action Handler
var changeStream = coll.watch([{ $match: {'documentKey.username': 'aly_cabral'}}],
temperature: 75
temperature: 85

#MDBlocal
temperature: 71,
username: “asya_kamsky”}
}
fullDocument: {_id: ‘blueRoom123’,
temperature: 85,
}
documentKey: {_id: 'blueRoom123'}
}
Action Handler
var changeStream = coll.watch([{ $match: {'documentKey._id':
{$in:['greenRoom123','blueRoom123']}
}}],
temperature: 75
temperature: 85

#MDBlocal
TO SEE EVERY CHANGE TO A DOCUMENT
CONSIDER ALL EDGE CASES

#MDBlocal
UNDERLYING
TECHNOLOGY

#MDBlocal
RESUMABLE
304 270 233 208 201 190
C AC H E D R E S U M E T O K E N : 2 0 8
N E W E S T O L D E S T
D E L E T E D

#MDBlocal
304 270 233 208 201 190
C AC H E D R E S U M E T O K E N : 1 9 0
N E W E S T O L D E S T
D E L E T E D
NON-RESUMABLE

Using Change Streams to Keep Up with Your Data

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Using Change Streams to Keep Up with Your Data

Similar to Using Change Streams to Keep Up with Your Data (20)

Recently uploaded

Recently uploaded (20)

Using Change Streams to Keep Up with Your Data

Editor's Notes