Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

1,939 views

Published on

Dynamo Streams provides a stream of all the updates done to your DynamoDB table. It is a simple but extremely powerful primitive which will enable developers to easily build solutions like cross-region replication, and to host additional materialized views, for instance an ElasticSearch index, on top of DynamoDB tables. In this session we will dive deep into details of Dynamo Streams, and how customers can leverage Dynamo Streams to build custom solutions and to extend the functionality of DynamoDB. We will give a demo of an example application built on top of Dynamo Streams to demonstrate the power and simplicity of Dynamo Streams.

Published in: Technology
  • Be the first to comment

(SDD424) Simplifying Scalable Distributed Applications Using DynamoDB Streams | AWS re:Invent 2014

  1. 1. SDD424 Simplifying Scalable Distributed Applications Using DynamoDB Streams
  2. 2. •WhyDynamoDBStreams •What DynamoDBStreams •How Demo @akshatvig
  3. 3. use cases Online gaming Ad tech Livevoting Social media Mobile messaging Backup & restore @akshatvig
  4. 4. replication @akshatvig
  5. 5. backups @akshatvig
  6. 6. views @akshatvig
  7. 7. Reactors –the new triggers @akshatvig
  8. 8. Logs @akshatvig
  9. 9. Op: PUT John Tokyo Op: UPDATE John Pluto Op: UPDATE John Mars Logsand databases Name Destination Tokyo Mars @akshatvig
  10. 10. istruthconstruct state @akshatvig
  11. 11. Logs arepowerful. Atomicity, consistency, and durability Replication Point-in-time restores Materialized views Auditing And much more... @akshatvig
  12. 12. What isDynamoDBStreams? It is a stream of updates Scales with your table DynamoDBStreams DynamoDB @akshatvig
  13. 13. Social networkapplication Comments and notifications @akshatvig
  14. 14. Take 1 Comments Table @akshatvig
  15. 15. post Take 2 Comments Table Processor @akshatvig
  16. 16. Take3 Comments Table @akshatvig
  17. 17. Globalusers @akshatvig
  18. 18. Cross region library @akshatvig
  19. 19. groups @akshatvig
  20. 20. Cross region replication post @akshatvig
  21. 21. Cross region library Horizontal scaling : Workers Load balancing @akshatvig
  22. 22. Cross region library Fault tolerant Checkpointing @akshatvig
  23. 23. Cross region replication Shard 1 Partition 1 Shard 2 Shard 3 Partition 2 KCL Worker KCL Worker KCL Worker @akshatvig
  24. 24. Cross region replication Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Shard 1 Shard 2 Shard 3 Shard 4 KCL Worker KCL Worker KCL Worker KCL Worker @akshatvig
  25. 25. Cross region replication library @akshatvig
  26. 26. Cross region replication library @akshatvig
  27. 27. DynamoDB Streams and DynamoDB Connectors simplify cross region replication! @akshatvig
  28. 28. Materialized view Extending DynamoDBcapabilities @akshatvig
  29. 29. Cross region Streams Streams Cross region @akshatvig
  30. 30. Data Tier Amazon DynamoDB Amazon RDS Amazon ElastiCache Amazon S3 Amazon CloudSearch Amazon Redshift archive rich search read replicas hot reads analytics complex queries & transactions purpose @akshatvig
  31. 31. Features of @akshatvig
  32. 32. View Type Destination Old Image –Before update Name= John, Destination = Mars New Image –After update Name = John,Destination = Pluto Old and New Images Name = John,Destination = Mars Name = John, Destination = Pluto Keys Only Name = Johntypes @akshatvig
  33. 33. Asynchronous i=A i=A ack @akshatvig
  34. 34. Exactlyonce i=A i=B i=B @akshatvig
  35. 35. Strictly ordered records i=A i=B i=C i=C @akshatvig
  36. 36. Durability & high availability High throughput consensus protocol Replicated across multiple AZs @akshatvig
  37. 37. Managed @akshatvig
  38. 38. Elasticity Adjusts @akshatvig
  39. 39. Performance Sub-second latency @akshatvig
  40. 40. Retention period Records available for 24 hours @akshatvig
  41. 41. DynamoDB Local Desktopinstallable Development & testing Publiclyavailable -http://bit.ly/1yt1r9q Now supports DynamoDB Streams @akshatvig
  42. 42. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
  43. 43. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
  44. 44. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
  45. 45. Consuming AWSDynamoDBstreamsAdapterClient adapterClient = newAWSDynamoDBstreamsAdapterClient(streamsCredentials,.. ); .. AmazonDynamoDBClient dynamoDBClient = newAmazonDynamoDBClient(dynamoDBCredentials, ..); .. KinesisClientLibConfiguration workerConfig = newKinesisClientLibConfiguration(..,streamId, streamsCredentials,..) .withMaxRecords(100) .withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON); Worker worker = newWorker(recordProcessorFactory,workerConfig,adapterClient, dynamoDBClient,..); Thread t =newThread(worker); t.start(); @akshatvig
  46. 46. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
  47. 47. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
  48. 48. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
  49. 49. Processingpublic class StreamsRecordProcessorimplementsIRecordProcessor { .. @Overridepublic void processRecords(List<Record>records,.. ){ for(Record record :records){ if (record instanceofRecordAdapter){ Record ddbStreamRecord =((RecordAdapter) record).getInternalObject(); switch(ddbStreamRecord.getEventName()){ case"INSERT":case"MODIFY": DemoHelper.putItem(dynamoDBClient,tableName, ddbStreamRecord.getDynamodb().getNewImage()); break; ... @akshatvig
  50. 50. Cross region replication @akshatvig
  51. 51. DynamoDB Reactors Trigger Lambda functions Example –Validate address, send notifications Console support @akshatvig
  52. 52. Preview available Temporary endpoints: N. Virginia & Ireland Available until global launch Registerfor preview at http://amzn.to/11dh9M0 @akshatvig
  53. 53. Preview console TBD: Console snapshot
  54. 54. @akshatvig
  55. 55. Landing pagehttp://amzn.to/1xodVzQ @akshatvig
  56. 56. Please give us your feedback on this session. Complete session evaluations and earn re:Invent swag. http://bit.ly/awsevals

×