More Related Content Similar to Bridge OLTP and Stream Processing with Amazon Kinesis, AWS Lambda, & MongoDB Atlas (ARC330-S) - AWS re:Invent 2018 (20) More from Amazon Web Services (20) Bridge OLTP and Stream Processing with Amazon Kinesis, AWS Lambda, & MongoDB Atlas (ARC330-S) - AWS re:Invent 20182. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bridge OLTP and stream processing
with Amazon Kinesis, AWS Lambda &
MongoDB Atlas
Eliot Horowitz
CTO & Co-Founder
MongoDB
A R C 3 3 0 - S
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
The modern data architecture
MongoDB's place in the
AWS ecosystem
Technical use cases
Example architectures
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake
5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake
6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
MongoDB
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
Services
Analytics
Snowflake
10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
Services
Persistence / Transactional
MongoDB
Analytics
Snowflake
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Persistence / Transactional
Analytics
Snowflake
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB in the AWS ecosystem
JSON
Dynamic schema
Idiomatic drivers
High Availability
Workload Isolation
Flexibility
Aggregation framework
$lookup
Fully ACID transactions
Scale out
Zoned Sharding
Flexible Storage
Power
Documents Distributed Systems
13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB in AWS: Self-service
• Manage nodes on Amazon
Elastic Compute Cloud
(Amazon EC2)
• Responsible for topology
maintenance
• Responsible for version
maintenance
• Requires custom code
running on Amazon EC2Region 1 Region 2
MongoDB Replica Set
MongoDB SDK
AWS Service
App Servers
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB in AWS: Managed service
• Provisioning
• Monitoring
• Data Access
• Maintenance
AWS Services
MongoDB
Atlas
• Triggers
• Functions
• JS wrapper for
AWS SDK for Go
MongoDB
Stitch
AWS SDK
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview
MongoDB's serverless platform
Connects Atlas to the AWS
ecosystem
16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Query Anywhere
• Declarative ACL
• Native SDKs
• Auth integrations
• Full MongoDB query
language
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Functions
• Serverless JavaScript
• Service integrations
• Lightweight
• Query MongoDB Atlas
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Stitch Functions and AWS Lambda
Stitch Functions
• Per ms billing
• Brief executing functions
• Service orchestration
• Database triggers
• JavaScript
Lambda
• Per 100ms billing
• Longer running functions
• Packaging with libraries
• Many languages
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Triggers
• Observe collections for
document updates
• Real-time
• Trigger function calls to
handle
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Mobile Sync
• Embedded MongoDB on
mobile platforms
• Change merges & conflict
resolution
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detailed technical use cases
Amazon Kinesis Data
Streams
• Inserting data
• Consuming data
Event processing
• Lambda
• MongoDB Stitch Functions
Analytics
• Amazon RedShift
• Amazon QuickSight
• MongoDB Charts
Archiving
• Getting documents to Amazon
Simple Storage Service (Amazon S3)
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Inserting
MongoDB
Atlas
db.observations.insert( { sensor_id: 1,
value: 482.48,
time: ISODate("2018-10-23T18:17:55.048Z") }
)
MongoDB
Stitch Trigger
MongoDB
Stitch Function
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Inserting
Setting up Amazon Kinesis and MongoDB Stitch
> aws kinesis create-stream --stream-name sensorStream --shardCount n
exports = function(sensorEvent){
const sensorReading = sensorEvent.fullDocument;
const kinesis = context.services.get('aws').kinesis();
try {
kinesis.PutRecord({
Data:JSON.stringify(sensorReading), StreamName: "sensorStream", PartitionKey:
sensorReading.sensorId
}).then(function(response){
return response;
});
} catch(error) { console.log(JSON.parse(error)); }
};
MongoDB
Stitch Function
25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
KCL/MongoDB
Client App
MongoDB
Atlas
Amazon Kinesis Data Streams: Consuming
db.eventTotals.updateOne( { _id: sensorId },
{ $inc: { observations: batchObservationTotal } } );
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
// set up connection to mongoDB
function recordProcessor() {
let shardId;
return {
initialize: function(initializeInput, completeCallback) { ... },
processRecords: function(processRecordsInput, completeCallback) { ... },
shutdown: function(shutdownInput, completeCallback) { ... }
}
};
kcl(recordProcessor().run());
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
initialize: function(initializeInput, completeCallback) {
shardId = initializeInput.shardId;
database.connect(mongodbConnectString, function(err) {
if (err) {
log.info(util.format('Error connecting to %s',
mongodbConnectString));
}
completeCallback();
})
}
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
processRecords: function(processRecordsInput, completeCallback) {
if (!processRecordsInput || !processRecordsInput.records) {
completeCallback(); return;
}
let records = processRecordsInput.records;
let counts = {};
records.forEach((rec) => {
let count = counts[rec.sensor_id] || 0;
counts[rec.sensor_id] = count + 1;
});
for (sensor in counts) {
mongoDB.update({"sensor_id": sensor},
{$inc: {"car_count": counts[sensor]}})
}
}
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
shutdown: function(shutdownInput, completeCallback) {
if (shutdownInput.reason !== 'TERMINATE') {
completeCallback();
return;
}
shutdownInput.checkpointer.checkpoint(function(err) {
completeCallback();
});
}
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: Lambda
MongoDB
Atlas
db.orders.updateOne( { _id: 32 },
{ $set: { status: "shipped" }});
MongoDB
Stitch Trigger
MongoDB
Stitch Function
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: Lambda
MongoDB Stitch Function
exports = async function() {
const lambda = context.services.get('aws-lambda-pdf-generator').lambda();
try {
const result = await lambda.Invoke({
FunctionName: "Hello_World_Test",
Payload: '{ "invoice_id": context.invoice.id,
"name": context.user.name,
"address": context.user.address,
"RMA_code": context.rma.code }'
});
return result;
} catch(error) {
console.error(EJSON.stringify(error));
}
};
MongoDB
Stitch Function
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: Lambda
AWS Lambda
import json
from fpdf import FPDF
def lambda_handler(event, context):
invoice = event['invoice_id']
name = event['name']
address = event['address']
rma_code = event['RMA_code']
pdf = FPDF()
# Create your amazing PDF Document here
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB
Stitch Function
Event processing: MongoDB Stitch
MongoDB
Atlas
db.products.updateOne( { sku: "MON2018SAUCE" },
{ $inc: { inventory: -40 } } );
Twilio
MongoDB
Stitch Trigger
35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: MongoDB Stitch
exports = function(productEvent){
const product = productEvent.fullDocument;
const aws = context.services.get('AWSService').ses();
const input = {
Destination: { ToAddresses: ['admin@store.com'] },
Message: {
Body: { Html: { Charset:"UTF-8", Data: `ID: ${product.id}<br>Remaining:
${product.inventory}` }},
// ... etc ...
}
};
try {
aws.SendEmail(input).then(function (result) {
console.log(JSON.stringify(result));
});
} catch(error) {
console.log(JSON.parse(error));
};
};
MongoDB
Stitch Function
36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics: Amazon RedShift
MongoDB
Atlas
MongoDB
Stitch
37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics: Amazon QuickSight
BI
Connector
MongoDB
Atlas
(SQL Proxy for MongoDB)
38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics: MongoDB Charts
MongoDB
Atlas
Charts
40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Charts: Chart Builder
41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Charts: Dashboards
42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Archiving
MongoDB
Atlas
MongoDB
Stitch
43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
NUMBER ONE
28
14
40
MILLION TICKETS
PER YEAR
MILLION
FANS
YEARS’
EXPERIENCE
AUSTRALIA AND
NEW ZEALAND’S
EVENT TICKETING AND
DIGITAL MARKETING
COMPANY
—
Independently verified by L.E.K Consulting
45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ticketek Realtime Data Pipeline Architecture
MongoDB
Mo2SKI production_au
Customer
Sales
CustomerSales_Internal
ticketek.backup/
customersales
Ticketek
Customer Sales
Ticketek
Data
Warehouse
Ticketek
Realtime
Salesvista-cluster
vista.backup/seatstatus
Vista Performance
Venue Cache
seatstatus_src_stream
performance_arc_stream
Seat S3
Cache
SQL Server
Lotus
VistaVenue
CacheFrom
Lotus_uat
ConfigPerformance
Kinesis Proxy
SeatStatus
Kinesis Proxy
Ticketek_ODS_
publisher_stream
Ticketek_Realtime_
publisher_stream
vista_publisher_stream
to vista
Vista Seat Status
Origin
46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Atlas: Global Cloud Database
Atlas
• Available in 15 AWS regions
• Full cross-region global distribution
of data
• Fully managed
Stitch
• Execute on changes in your data in
real time
• Integrate with 3rd party services and
entire AWS ecosystem
• Securely expose power of DB queries
out to client applications
47. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
http://mongodb.com/reinvent2018
for $200 in Atlas credit. Live migrate with no
downtime!
Join us for a pub crawl at AquaKnox, 6pm
Visit our booths at the Venetian and the Builders Fair at the Aria
48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.