Dynamo Streams provides a stream of all the updates done to your DynamoDB table. It is a simple but extremely powerful primitive which will enable developers to easily build solutions like cross-region replication, and to host additional materialized views, for instance an ElasticSearch index, on top of DynamoDB tables. In this session we will dive deep into details of Dynamo Streams, and how customers can leverage Dynamo Streams to build custom solutions and to extend the functionality of DynamoDB. We will give a demo of an example application built on top of Dynamo Streams to demonstrate the power and simplicity of Dynamo Streams.

1. SDD424
Simplifying Scalable Distributed
Applications Using DynamoDB Streams

2. •WhyDynamoDBStreams
•What DynamoDBStreams
•How
Demo
@akshatvig

3. use cases
Online
gaming
Ad tech
Livevoting
Social
media
Mobile
messaging
Backup & restore
@akshatvig

4. replication
@akshatvig

5. backups
@akshatvig

6. views
@akshatvig

7. Reactors –the new triggers
@akshatvig

8. Logs
@akshatvig

9. Op: PUT
John
Tokyo
Op: UPDATE
John
Pluto
Op: UPDATE
John
Mars
Logsand databases
Name
Destination
Tokyo
Mars
@akshatvig

10. istruthconstruct state
@akshatvig

11. Logs arepowerful.
Atomicity, consistency, and durability
Replication
Point-in-time restores
Materialized views
Auditing
And much more...
@akshatvig

12. What isDynamoDBStreams?
It is a stream of updates
Scales with your table
DynamoDBStreams
DynamoDB
@akshatvig

13. Social networkapplication
Comments and notifications
@akshatvig

14. Take 1
Comments Table
@akshatvig

15. post
Take 2
Comments Table
Processor
@akshatvig

16. Take3
Comments Table
@akshatvig

17. Globalusers
@akshatvig

18. Cross region library
@akshatvig

19. groups
@akshatvig

20. Cross region replication
post
@akshatvig

21. Cross region library
Horizontal scaling : Workers
Load balancing
@akshatvig

22. Cross region library
Fault tolerant
Checkpointing
@akshatvig

23. Cross region replication
Shard 1
Partition 1
Shard 2
Shard 3
Partition 2
KCL Worker
KCL Worker
KCL Worker
@akshatvig

24. Cross region replication
Partition 1
Partition 2
Partition 3
Partition 4
Partition 5
Shard 1
Shard 2
Shard 3
Shard 4
KCL
Worker
KCL
Worker
KCL
Worker
KCL
Worker
@akshatvig

25. Cross region replication library
@akshatvig

26. Cross region replication library
@akshatvig

27. DynamoDB Streams and DynamoDB Connectors simplify cross region replication!
@akshatvig

28. Materialized view
Extending DynamoDBcapabilities
@akshatvig

29. Cross region
Streams
Streams
Cross region
@akshatvig

30. Data Tier
Amazon
DynamoDB
Amazon
RDS
Amazon
ElastiCache
Amazon
S3
Amazon
CloudSearch
Amazon
Redshift
archive
rich search
read replicas
hot reads
analytics
complex queries
& transactions
purpose
@akshatvig

31. Features of
@akshatvig

32. View Type
Destination
Old Image –Before update
Name= John, Destination = Mars
New Image –After update
Name = John,Destination = Pluto
Old and New Images
Name = John,Destination = Mars
Name = John, Destination = Pluto
Keys Only
Name = Johntypes
@akshatvig

33. Asynchronous
i=A
i=A
ack
@akshatvig

34. Exactlyonce
i=A
i=B
i=B
@akshatvig

35. Strictly ordered records
i=A
i=B
i=C
i=C
@akshatvig

36. Durability & high availability
High throughput consensus protocol
Replicated across multiple AZs
@akshatvig

37. Managed
@akshatvig

38. Elasticity
Adjusts
@akshatvig

39. Performance
Sub-second latency
@akshatvig

40. Retention period
Records available for 24 hours
@akshatvig

41. DynamoDB Local
Desktopinstallable
Development & testing
Publiclyavailable -http://bit.ly/1yt1r9q
Now supports DynamoDB Streams
@akshatvig

42. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start();
@akshatvig

43. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start();
@akshatvig

44. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start();
@akshatvig

45. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start();
@akshatvig

46. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ...
@akshatvig

47. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ...
@akshatvig

48. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ...
@akshatvig

49. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ...
@akshatvig

50. Cross region replication
@akshatvig

52. DynamoDB Reactors
Trigger Lambda functions
Example –Validate address, send notifications
Console support
@akshatvig

53. Preview available
Temporary endpoints: N. Virginia & Ireland
Available until global launch
Registerfor preview at http://amzn.to/11dh9M0
@akshatvig

54. Preview console
TBD: Console snapshot

55. @akshatvig

56. Landing pagehttp://amzn.to/1xodVzQ
@akshatvig

57. Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals