Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[NEW LAUNCH!] Building modern applications using Amazon DynamoDB transactions (DAT374) - AWS re:Invent 2018

907 views

Published on

DynamoDB transactions enables developers to maintain correctness of their data at scale by adding atomicity and isolation guarantees for multi-item conditional updates. With transactions, you can perform a batch of conditional operations including, PutItem, UpdateItem, and DeleteItem with guarantees. Come to this session to learn about how DynamoDB transactions works, the primary use cases in enables, and how to build modern applications that require transactions.

  • Get access to 16,000 woodworking plans, Download 50 FREE Plans...  http://ishbv.com/tedsplans/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[NEW LAUNCH!] Building modern applications using Amazon DynamoDB transactions (DAT374) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Building Modern Applications Using Amazon DynamoDB Transactions Yossi Levanoni Senior Manager, Software Development Amazon DynamoDB D A T 3 7 4
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda 1. Introduction to atomicity, consistency, isolation, and durability (ACID) in DynamoDB, and the new transactional API 2. Modern application use cases for the transactional API 3. Important things to know when integrating the transactional API into your application: metering, concurrency control, and interaction with other features
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What you will learn • How the new transactional API works • How you can use the new API in your application through case studies demonstrating transactional design patterns • How the new API interacts with DynamoDB features
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why is this important? The transactional API can be used to implement real-world transactional scenarios The transactional API can be used to simplify the implementation of cloud application flows
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Prerequisites You should be familiar with DynamoDB concepts such as tables, items, partition keys and sort keys, indexes, Time To Live, and streams. We will only briefly introduce these concepts in this session before using them in the use cases.
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Transactional API at a glance TransactWriteItems Transact up to 10 items Evaluate conditions If all conditions are simultaneously true, perform write operations TransactGetItems Transact up to 10 items Return a consistent, isolated snapshot of all items
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Atomicity, consistency, isolation, and durability (ACID) in DynamoDB • Before: ACID support for single-item operations • NEW: ACID support for multi-item operations Nontransactional Transactional New TransactGetItems TransactWriteItems Existing BatchGetItem, BatchWriteItem Query, Scan GetItem, PutItem, UpdateItem, DeleteItem
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Transactions at DynamoDB scale • Apply transactions to items in: • Same partition key, or across partition keys • Same table, or across tables • Same Region only • Same account only • DynamoDB tables only • Transactions scale like the rest of DynamoDB • Limitless concurrent transactions • Low latency, high availability • Scales horizontally • Your responsibility: Design to avoid hot key
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Path of success with NoSQL transactions Design choices • Transact any items in your account • Service APIs vs. client-side library • Request/response-based • Limit of 10 items How they help you • Scale your data and relations within it • Reduces complexity, and improves cost and performance • Keeps business logic in your application and not in the database layer • Consistent low latency, never deadlock • Consistent low latency, high availability • Addresses vast majority of use cases
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. User profile management (use case #1) • Problem statement • Users identified by user name and optionally email • No two users can share the same identifier (user name or email) • Users can change their email • What is demonstrated • Enforcing unique constraints • Strongly consistent global indexing • Heterogeneous tables
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. User profile management • Solution • Users table contains two kinds of records • Profile record, keyed by opaque user name and containing user details • Alias records, keyed by user email and referencing the profile record Id UserNameRef Email Phone # “John123” “john@example.com” “444-555-0110” “john@example.com” “Joe123” “jane456” “444-555-0199”
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Insert new profile data = await dynamoDb.transactWriteItems({ TransactItems: [ { Put: { TableName: 'Users', Item: { Id: { S: 'John123' }, Email: {S: 'john123@example.com'}, ...}, ConditionExpression: 'attribute_not_exists(Id)‘ }}, { Put: { TableName: 'Users', Item: { Id: { S: 'john123@example.com' }, UserNameRef: {S: 'John123'}}, ConditionExpression: 'attribute_not_exists(Id)',}} ] }).promise();
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Update email address data = await dynamoDb.transactWriteItems({ TransactItems: [ { Put: { TableName: 'Users', Item: { Id: { S: 'John123' }, Email: {S: 'john123@example.org'}, ...}, ConditionExpression: 'attribute_exists(Id) and Email = :old_email', ExpressionAttributeValues: {':old_email' :{'S': 'john123@example.com'}}}}, { Put: { TableName: 'Users', Item: { Id: { S: 'john123@example.org' }, UserNameRef: {S: 'John123'}}, ConditionExpression: 'attribute_not_exists(Id)', }}, { Delete: { TableName: 'Users', Item: { Id: { S: 'john123@example.com' }}}).promise();
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management (use case #2) • Problem statement • Guests create room reservations • Guests check in to rooms, fulfilling a reservation • Guests check out of rooms, closing a reservation • What is demonstrated • Ensuring idempotency – operations with side effects happen at most once • Ensuring multi-item changes are only triggered when valid (consistency) • Ensuring multi-item changes happen all together (atomicity)
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management data model Guests Id (primary key) Name, Email, etc. Reservations (set) OccupiesRooms (set) Reservations Id (primary key) GuestId FulfilledByRoom State Rooms Id (primary key) RentedToReservation State
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management data model Record Type Id (PK) Attributes Guest “John” Reservations : { “500”, “501” } OccupiesRooms : { “20014” } Reservation “500” GuestId: “John” State: “PENDING” Reservation “501” GuestId: “John” State: “FULFILLED” FulfilledByRoom: “20014” Room “20014” State: “OCCUPIED” RentedToReservation : “501”
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management: Create a reservation • Idempotency – how to ensure only a single reservation is created upon retries? • Solution • Client generates a unique reservation ID • Client generates TransactWriteItems request with the following operations • Conditional Put for the new reservation • Update the customer reservations set: Add the new reservation • Client transaction is now idempotent and can be retried without double booking
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management: Check-in Record Type Id (PK) Attributes Customer “John123” Reservations : { “567” } OccupiesRooms : {} Reservation “567” CustomerId: “John123” State: “PENDING” FulfilledByRoom: null Room “20014” State: “FREE” RentedToReservation : null Before
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management: Check-in Record Type Id (PK) Attributes Customer “John123” Reservations : { “567” } OccupiesRooms : { “20014” } Reservation “567” CustomerId: “John123” State: “FULFILLED” FulfilledByRoom: “20014” Room “20014” State: “OCCUPIED” RentedToReservation : “567” After
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management: Checkout Record Type Id (PK) Attributes Customer “John123” Reservations : { “567” } OccupiesRooms : { “20014” } Reservation “567” CustomerId: “John123” State: “FULFILLED” FulfilledByRoom: “20014” Room “20014” State: “OCCUPIED” RentedToReservation : “567” Before
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hotel reservation management: Checkout Record Type Id (PK) Attributes Customer “John123” Reservations : { “567” } OccupiesRooms : { “20014” } Reservation “567” CustomerId: “John123” State: “CLOSED” FulfilledByRoom: “20014” Room “20014” State: “FREE” RentedToReservation : “567” After
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Concurrency control • Optimistic • No deadlocks possible • Low latency • Optimized for scale-out • Your responsibility • Design for scale-out • Avoid unnecessary conflicts
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Metering transactions Transactions perform 2x the work of nontransactional counterparts Transactions are metered 2x their constituent operations
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why didn’t my transaction succeed? • Per-item failure reasons • Precondition failure • Insufficient capacity • Transactional conflicts • Other reasons • Transaction is still in progress • Service error • Malformed request • Permissions
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Retrying transactions: ClientRequestToken • Ensures SDK can retry without “replaying” the same intent multiple times • ClientRequestToken – idempotency token at the SDK or API level • Autogenerated, you can override • Applies for 10 minutes • Provides information about transaction state and outcome • Sometimes doesn’t remove the need for application-level idempotency or intent design
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Application-level retries • When the SDK gives up • Build a new view of the world • Retry with new preconditions and desirable end state • How to build a snapshot of the world? Three options: • Set ReturnValuesOnConditionCheckFailure = ALL_OLD • Perform TransactGetItems • Work from an eventually consistent view (results of queries, etc.)
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Integrating with other DynamoDB features • General rule about DynamoDB data • Committed state only • But might not be isolated • Examples • Streams – Committed writes, sharded • Backups and point-in-time recovery (PITR) – Subset of a committed transaction is possible • Global secondary index (GSI) – Committed writes are indexed individually • Global tables – Transactional API disabled by default; writes are replicated individually • Amazon DynamoDB Accelerator (DAX) support is on the roadmap
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Access control • Individual operations are authorized separately • (NEW) dynamodb:ConditionCheck • dynamodb:UpdateItem • dynamodb:PutItem • dynamodb:DeleteItem • No separate permissions for top-level new APIs
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Attachment management (use case #3) • Problem statement • A social media site allows attaching media to posts • Posts need to share media efficiently • Posts can have multiple attachments • Unreferenced media needs to be deleted • What is demonstrated • External resource metadata and lifetime management • Using Time To Live (TTL) and streams • Reference counting using transactions
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Attachment management data model Attachment item S3Ref (PK) – Reference to an object in Amazon S3 RefCount – Number of posts referencing this attachment TTL – Time when this attachment is eligible to be deleted Post item Id (PK) – Post ID Attachments (set) – Set of attachment IDs
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Attachment management invariants 1. RefCount = number of posts referencing the attachment 2. If RefCount > 0, then S3Ref refers to a valid Amazon S3 object 3. TTL is set if, and only if, RefCount = 0
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Attachment management: Create attachment and link it to a post • Generate unique Amazon S3 object name • Create attachment record in DynamoDB • Set RefCount = 0, • Set TTL = 24 hours from now • Upload object to Amazon S3 • In a TransactWriteItems Request • Check that the attachment is not already referenced by the post • Increase RefCount by 1, remove TTL • Add attachment reference to post
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Attachment management: Clone attachment reference • Copy the reference of an attachment from one post to another • In a TransactWriteItems Request: • Check that the attachment is not already referenced by the new post • Increase RefCount by 1, remove TTL • Add attachment reference to new post
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Attachment management: Delete unused external resources • Attachment records will be deleted a day after their reference count drops to zero • In AWS Lambda, process stream records representing items deleted by TTL • Delete the corresponding objects from Amazon S3
  39. 39. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Yossi Levanoni Contact us through the AWS forums or @dynamodb on Twitter Get the Database Freedom T-shirt in our swag booth!
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×