Seamlessly bridge your application’s real-time operational surfaces—which face customers, devices, and latency-sensitive components—to its data processing inner surfaces—which enable BI, machine learning, and other massive analytical workloads. Learn how to compose Amazon Kinesis, Amazon Redshift, Amazon S3, AWS Lambda, and MongoDB into a flexible, secure, and performant architectural core. Then learn through concrete examples how that core extends in order to handle a wide range of applications, from rich financial accounting platforms to cloud-based, open IoT operating systems. This session is brought to you by AWS partner, MongoDB.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bridge OLTP and stream processing
with Amazon Kinesis, AWS Lambda &
MongoDB Atlas
Eliot Horowitz
CTO & Co-Founder
MongoDB
A R C 3 3 0 - S

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
The modern data architecture
MongoDB's place in the
AWS ecosystem
Technical use cases
Example architectures

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Analytics
Snowflake

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
MongoDB
Persistence / Transactional
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
Services
Analytics
Snowflake

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
Services
Persistence / Transactional
MongoDB
Analytics
Snowflake

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The modern data architecture
Apache
kafka
Streaming
Archival
Clients
Web
IoT
Mobile
App Servers
Event Processing
Twilio
ServicesMongoDB
Persistence / Transactional
Analytics
Snowflake

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB in the AWS ecosystem
JSON
Dynamic schema
Idiomatic drivers
High Availability
Workload Isolation
Flexibility
Aggregation framework
$lookup
Fully ACID transactions
Scale out
Zoned Sharding
Flexible Storage
Power
Documents Distributed Systems

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB in AWS: Self-service
• Manage nodes on Amazon
Elastic Compute Cloud
(Amazon EC2)
• Responsible for topology
maintenance
• Responsible for version
maintenance
• Requires custom code
running on Amazon EC2Region 1 Region 2
MongoDB Replica Set
MongoDB SDK
AWS Service
App Servers

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB in AWS: Managed service
• Provisioning
• Monitoring
• Data Access
• Maintenance
AWS Services
MongoDB
Atlas
• Triggers
• Functions
• JS wrapper for
AWS SDK for Go
MongoDB
Stitch
AWS SDK

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview
MongoDB's serverless platform
Connects Atlas to the AWS
ecosystem

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Query Anywhere
• Declarative ACL
• Native SDKs
• Auth integrations
• Full MongoDB query
language

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Functions
• Serverless JavaScript
• Service integrations
• Lightweight
• Query MongoDB Atlas

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Stitch Functions and AWS Lambda
Stitch Functions
• Per ms billing
• Brief executing functions
• Service orchestration
• Database triggers
• JavaScript
Lambda
• Per 100ms billing
• Longer running functions
• Packaging with libraries
• Many languages

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Triggers
• Observe collections for
document updates
• Real-time
• Trigger function calls to
handle

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stitch overview: Mobile Sync
• Embedded MongoDB on
mobile platforms
• Change merges & conflict
resolution

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detailed technical use cases
Amazon Kinesis Data
Streams
• Inserting data
• Consuming data
Event processing
• Lambda
• MongoDB Stitch Functions
Analytics
• Amazon RedShift
• Amazon QuickSight
• MongoDB Charts
Archiving
• Getting documents to Amazon
Simple Storage Service (Amazon S3)

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Inserting
MongoDB
Atlas
db.observations.insert( { sensor_id: 1,
value: 482.48,
time: ISODate("2018-10-23T18:17:55.048Z") }
)
MongoDB
Stitch Trigger
MongoDB
Stitch Function

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Inserting
Setting up Amazon Kinesis and MongoDB Stitch
> aws kinesis create-stream --stream-name sensorStream --shardCount n
exports = function(sensorEvent){
const sensorReading = sensorEvent.fullDocument;
const kinesis = context.services.get('aws').kinesis();
try {
kinesis.PutRecord({
Data:JSON.stringify(sensorReading), StreamName: "sensorStream", PartitionKey:
sensorReading.sensorId
}).then(function(response){
return response;
});
} catch(error) { console.log(JSON.parse(error)); }
};
MongoDB
Stitch Function

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
KCL/MongoDB
Client App
MongoDB
Atlas
Amazon Kinesis Data Streams: Consuming
db.eventTotals.updateOne( { _id: sensorId },
{ $inc: { observations: batchObservationTotal } } );

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
// set up connection to mongoDB
function recordProcessor() {
let shardId;
return {
initialize: function(initializeInput, completeCallback) { ... },
processRecords: function(processRecordsInput, completeCallback) { ... },
shutdown: function(shutdownInput, completeCallback) { ... }
}
};
kcl(recordProcessor().run());

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
initialize: function(initializeInput, completeCallback) {
shardId = initializeInput.shardId;
database.connect(mongodbConnectString, function(err) {
if (err) {
log.info(util.format('Error connecting to %s',
mongodbConnectString));
}
completeCallback();
})
}

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
processRecords: function(processRecordsInput, completeCallback) {
if (!processRecordsInput || !processRecordsInput.records) {
completeCallback(); return;
}
let records = processRecordsInput.records;
let counts = {};
records.forEach((rec) => {
let count = counts[rec.sensor_id] || 0;
counts[rec.sensor_id] = count + 1;
});
for (sensor in counts) {
mongoDB.update({"sensor_id": sensor},
{$inc: {"car_count": counts[sensor]}})
}
}

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams: Consuming
KCL-based stream consumer
shutdown: function(shutdownInput, completeCallback) {
if (shutdownInput.reason !== 'TERMINATE') {
completeCallback();
return;
}
shutdownInput.checkpointer.checkpoint(function(err) {
completeCallback();
});
}

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: Lambda
MongoDB
Atlas
db.orders.updateOne( { _id: 32 },
{ $set: { status: "shipped" }});
MongoDB
Stitch Trigger
MongoDB
Stitch Function

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: Lambda
MongoDB Stitch Function
exports = async function() {
const lambda = context.services.get('aws-lambda-pdf-generator').lambda();
try {
const result = await lambda.Invoke({
FunctionName: "Hello_World_Test",
Payload: '{ "invoice_id": context.invoice.id,
"name": context.user.name,
"address": context.user.address,
"RMA_code": context.rma.code }'
});
return result;
} catch(error) {
console.error(EJSON.stringify(error));
}
};
MongoDB
Stitch Function

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: Lambda
AWS Lambda
import json
from fpdf import FPDF
def lambda_handler(event, context):
invoice = event['invoice_id']
name = event['name']
address = event['address']
rma_code = event['RMA_code']
pdf = FPDF()
# Create your amazing PDF Document here

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB
Stitch Function
Event processing: MongoDB Stitch
MongoDB
Atlas
db.products.updateOne( { sku: "MON2018SAUCE" },
{ $inc: { inventory: -40 } } );
Twilio
MongoDB
Stitch Trigger

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event processing: MongoDB Stitch
exports = function(productEvent){
const product = productEvent.fullDocument;
const aws = context.services.get('AWSService').ses();
const input = {
Destination: { ToAddresses: ['admin@store.com'] },
Message: {
Body: { Html: { Charset:"UTF-8", Data: `ID: ${product.id}<br>Remaining:
${product.inventory}` }},
// ... etc ...
}
};
try {
aws.SendEmail(input).then(function (result) {
console.log(JSON.stringify(result));
});
} catch(error) {
console.log(JSON.parse(error));
};
};
MongoDB
Stitch Function

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics: Amazon RedShift
MongoDB
Atlas
MongoDB
Stitch

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics: Amazon QuickSight
BI
Connector
MongoDB
Atlas
(SQL Proxy for MongoDB)

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analytics: MongoDB Charts
MongoDB
Atlas
Charts

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Charts: Chart Builder

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Charts: Dashboards

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Archiving
MongoDB
Atlas
MongoDB
Stitch

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
NUMBER ONE
28
14
40
MILLION TICKETS
PER YEAR
MILLION
FANS
YEARS’
EXPERIENCE
AUSTRALIA AND
NEW ZEALAND’S
EVENT TICKETING AND
DIGITAL MARKETING
COMPANY
—
Independently verified by L.E.K Consulting

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ticketek Realtime Data Pipeline Architecture
MongoDB
Mo2SKI production_au
Customer
Sales
CustomerSales_Internal
ticketek.backup/
customersales
Ticketek
Customer Sales
Ticketek
Data
Warehouse
Ticketek
Realtime
Salesvista-cluster
vista.backup/seatstatus
Vista Performance
Venue Cache
seatstatus_src_stream
performance_arc_stream
Seat S3
Cache
SQL Server
Lotus
VistaVenue
CacheFrom
Lotus_uat
ConfigPerformance
Kinesis Proxy
SeatStatus
Kinesis Proxy
Ticketek_ODS_
publisher_stream
Ticketek_Realtime_
publisher_stream
vista_publisher_stream
to vista
Vista Seat Status
Origin

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MongoDB Atlas: Global Cloud Database
Atlas
• Available in 15 AWS regions
• Full cross-region global distribution
of data
• Fully managed
Stitch
• Execute on changes in your data in
real time
• Integrate with 3rd party services and
entire AWS ecosystem
• Securely expose power of DB queries
out to client applications

47. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
http://mongodb.com/reinvent2018
for $200 in Atlas credit. Live migrate with no
downtime!
Join us for a pub crawl at AquaKnox, 6pm
Visit our booths at the Venetian and the Builders Fair at the Aria

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.