Application Development and Data Modeling on Amazon DynamoDB

1,560 views
1,475 views

Published on

2014/06/26
AWSプロダクトシリーズ | よくわかるAmazon DynamoDB
David Yanacekによる講演資料

Application Development and Data Modeling on Amazon DynamoDB

  1. 1. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Application Development and Data Modeling on Amazon DynamoDB David Yanacek, Sr. Software Development Engineer, Amazon DynamoDB
  2. 2. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Compute Networking & CDN Databases Analytics Deployment & Management Application Services Storage
  3. 3. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture
  4. 4. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture High performance databases throughput elasticity availability latency query flexibility storage capacity operational complexity
  5. 5. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Scale Performance Predictable, consistent Degraded performance with scale
  6. 6. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture = more problems Data sharding Data caching Provisioning Cluster management Fault management
  7. 7. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Undifferentiated heavy lifting
  8. 8. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Amazon DynamoDB
  9. 9. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Highlights Flexible Durable storage Predictable performance Zero admin Low latency Large scale Seamless scaling Excellent availability
  10. 10. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Plan • What is DynamoDB? • In-depth application modeling – Website Session Data (basic data model) – Tic Tac Toe (conditional writes) – Image Tagging (indexes and querying) – Gaming Leaderboard (materialized views) • High-level use cases – Application architecture, data models
  11. 11. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data basic data model and APIs Web Sessions
  12. 12. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data Web Sessions
  13. 13. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data EC2 Instance (web server) Users (first hit to website) set-cookie: { “SessionId”: “af872e”, “History”: [“index.html”] } Web Sessions
  14. 14. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data: ElastiCache Elastic Load Balancing EC2 Instances (web server) Users ElastiCache • ElastiCache distributed cache • Fast, scales • Node failures cause loss of session data • Adding and Removing nodes is tricky Web Sessions
  15. 15. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data: RDBMS Elastic Load Balancing Users RDS • Durable • Adding and Removing nodes is tricky • Database blips impact site availability • Traffic can interfere with other DB workloads Web Sessions EC2 Instances (web server)
  16. 16. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Website Session Data: DynamoDB Elastic Load Balancing Users • Durable • Transparently scalable • Low latency • High availability • Read consistency • Simple to develop • Simple to operate DynamoDB Web Sessions EC2 Instances (web server)
  17. 17. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture DynamoDB Session Handling Sessions Table SessionId Last Accessed Shopping Cart dc32f 2014-06-10T12:... {“Products”: [“Harry Potter DVD”]} 2d231 2014-06-10T12:... {“Products”: [“Amazon Fire Phone”]} e0da3 2014-06-10T12:... {“Products”: [“Kindle Fire”]} 13f47 2014-06-10T12:... Web Sessions
  18. 18. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture GetItem: Getting the current session Web Sessions
  19. 19. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture PutItem: Updating the session state Web Sessions
  20. 20. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Using higher-level SDKs Web Sessions
  21. 21. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture DynamoDB Session Handlers (even simpler) • Open-source session plugins in: – Java (Tomcat) – Ruby (Rails) – .NET – PHP – Python (Django) – Node.js (Connect) – … Web Sessions
  22. 22. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Offline development • DynamoDB Local: Offline DynamoDB for testing, development Web Sessions
  23. 23. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Website Session Data • DynamoDB’s scalability, performance, availability, and durability, and consistency make it a great fit for website session data • Very easy to use for your website – already implemented open source • Other key/value lookup use cases are natural fits too Web Sessions
  24. 24. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Game State conditional writes Social Gaming
  25. 25. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Social Gaming
  26. 26. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe { Id : abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right : O } Game Item Social Gaming
  27. 27. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Amazon DynamoDB Alice Bob Social Gaming
  28. 28. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Amazon DynamoDB Alice Bob Update: Top-Right : O Turn : Bob Social Gaming
  29. 29. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe Amazon DynamoDB Alice Bob Update: Top-Left : X Turn : Alice Social Gaming
  30. 30. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3) Social Gaming
  31. 31. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3) Social Gaming
  32. 32. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3) Social Gaming
  33. 33. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O Social Gaming
  34. 34. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Low-Right : X Social Gaming
  35. 35. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Gaming the System Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X, Mid: X, Low-Right: X Update: Turn : Alice Low-Right : X Social Gaming
  36. 36. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Conditional Writes • Accept a write only if values are as expected • Otherwise reject the write • Performance like normal writes Social Gaming
  37. 37. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe (Fixed) Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Social Gaming
  38. 38. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe (Fixed) Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Social Gaming
  39. 39. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Tic Tac Toe (Fixed) Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Social Gaming
  40. 40. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Game State • UpdateItem can update specific attributes • PutItem replaces the entire item • Conditional writes synchronize state transitions Social Gaming
  41. 41. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging querying tables, secondary indexes Tagging
  42. 42. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging: Requirements • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  43. 43. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table Tagging
  44. 44. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Tagging
  45. 45. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Hash and Range primary key schema (composite key) Range: Index is sorted by range key, can do range queries Hash: Equality queries only Tagging
  46. 46. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Bob Query for Bob’s Images Query FROM Images WHERE User=Bob Tagging
  47. 47. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  48. 48. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Local Secondary Indexes • Alternate Range Key for your table • More flexible Query patterns • Local to the Hash Key – (more query options within “Bob’s” images) Tagging
  49. 49. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Local Secondary Index on Date Tagging
  50. 50. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Query for Bob’s two most recent imagesQuery FROM Image-Index WHERE User=Bob, DESC, LIMIT 2 Tagging
  51. 51. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  52. 52. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging Images Table ImageTags Table Tagging
  53. 53. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob Tagging
  54. 54. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob Hash and Range Primary Key Schema Tagging
  55. 55. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image User aed4c Alice aed4c Bob f93bae Bob Image Tagging ImageTags Table Bob Query FROM ImageTags WHERE Image=aed4c Tagging
  56. 56. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Bob Tag Alice in Image f93bae Tagging
  57. 57. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging • Query a user’s images • Query a user’s images by date • Tag other users in images • Query images a user is tagged in Tagging
  58. 58. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Global Secondary Indexes • Alternate Hash and/or Range Key for your table • Even more flexible Query patterns Tagging
  59. 59. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Tagging
  60. 60. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Global Secondary Index on User, Image User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Alternate Hash and Range Keys Tagging
  61. 61. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Image Tagging ImageTags Table Alice Query for images tagged Alice User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Query FROM ImageTags-Index WHERE User=Alice Tagging
  62. 62. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Image Tagging • Schema design on DynamoDB is all about the indexes • Local Secondary Indexes support flexible queries • Global Secondary Indexes unlock even more flexible queries • Many other use cases for indexing metadata on DynamoDB Tagging
  63. 63. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Social Leaderboard materialized views Leaderboards
  64. 64. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard • Find friends playing the same game • See where you stack up against your friends Leaderboards
  65. 65. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScores Table User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin Leaderboards
  66. 66. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScores Table User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin Alice Leaderboards
  67. 67. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Friends Leaderboard User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScores Table User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin 1. Query 2. BatchGetItemAlice Leaderboards
  68. 68. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Inefficiencies in friends leaderboard • Always retrieve all friends – Some friends don’t play – Might care about top 10 scores only • Single-item operations for each friend – Retrieving 10 items individually is more expensive than 10 items in a row Leaderboards
  69. 69. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Improved Friends Leaderboard: Reads User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 FriendScores Table HighScores Table Leaderboards
  70. 70. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Improved Friends Leaderboard: Reads User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User H.S. Friend Bob 20 Chuck Bob 1000 Alice Alice 20 Chuck Alice 470 Bob Alice 580 Erin Table Local Secondary Index FriendScores Table HighScores Table Leaderboards
  71. 71. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Improved Friends Leaderboard: Reads User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User H.S. Friend Bob 20 Chuck Bob 1000 Alice Alice 20 Chuck Alice 470 Bob Alice 580 Erin Table Local Secondary Index FriendScores Table HighScores Table Query FROM FriendScores-Index WHERE User=Alice, DESC, LIMIT 10Alice Leaderboards
  72. 72. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture User HighScore Chuck 30 Alice 1000 Bob 470 Dave 850 Erin 580 Improved Friends Leaderboard: Writes User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 30 Alice Erin 580 FriendScores Table HighScores Table Chuck 1. UpdateItem 2. UpdateItem(s) Leaderboards
  73. 73. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap: Friends Leaderboard • BatchGetItem helps retrieve many items • Query is cheaper than retrieving items individually • Trade off: reads can be made faster, cheaper, by doing more writes Leaderboards
  74. 74. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture High-level use cases application architecture Architecture
  75. 75. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture High-level use cases • Online game coordination • Real-time ad impression bids • Indexed event logging • Social messaging • Real-time voting • Real-time analytics Architecture
  76. 76. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 CloudFron t Amazon S3 (content) (client app) (game servers) (lobby, stats service) Architecture
  77. 77. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 1) Login 2) Request matchmaking 3) Get game server IP CloudFron t Amazon S3 Architecture
  78. 78. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 1) Login 2) Request matchmaking 3) Get game server IP 4) Connect to server 5) Pull down assets 6) Other players join CloudFron t Amazon S3 Architecture
  79. 79. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Online Game Coordination Availability Zone #1 DynamoDB region EC2 Availability Zone #1 EC2 EC2 1) Login 2) Request matchmaking 3) Get game server IP 4) Connect to server 5) Pull down assets 6) Other players join 7) Game ends 8) Update stats CloudFron t Amazon S3 Architecture
  80. 80. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Indexed event logging User DateTime Product David 2014-06-22T21:01 Fire TV David 2014-06-22T21:02 Fire Phone David 2014-06-23T08:32 Fire Game Controller Alice 2014-06-23… … … … … Index webpage clicks by user Architecture
  81. 81. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time ad impression bids Architecture
  82. 82. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Social networks, messaging Thread MessageId User Message Fri Night 21:01.1 David Hi! Fri Night 21:01.2 Steve What’s up? Fri Night 21:02.1 David Happy Hour? Participants Table, Secondary Index Messages Table Thread User Fri Night David Fri Night Steve DynamoDB? David User Thread David DynamoDB? David Fri Night Steve Fri Night Architecture
  83. 83. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time voting Voters AggregateVotes Table RawVotes Table Voting App Amazon Redshift Amazon EMR Architecture
  84. 84. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time analytics Kinesis-enabled app Amazon Kinesis DynamoDB Ingestion In-memory first-pass aggregation Durable aggregation Architecture
  85. 85. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time analytics Page Time Views index.html 2014-06-25T12:00 120 index.html 2014-06-25T12:01 122 index.html 2014-06-25T12:02 125 search.html 2014-06-25T12:00 200 PageViews Table Kinesis-enabled app UPDATE PageViews ADD 5 to Views WHERE Page=index.html AND Time=2014-06-25T12:00 Architecture
  86. 86. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Real-time analytics Page Time Views index.html 2014-06-25T12:00 120 index.html 2014-06-25T12:01 122 index.html 2014-06-25T12:02 125 search.html 2014-06-25T12:00 200 PageViews Table Kinesis-enabled app Query PageViews WHERE Page=index.html AND Time >= 2014-06-25T12:00 LIMIT 10 Architecture
  87. 87. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Recap • What is DynamoDB? • In-depth application modeling – Website Session Data (basic data model) – Tic Tac Toe (conditional writes) – Image Tagging (indexes and querying) – Gaming Leaderboard (materialized views) • High-level use cases – Application architecture, data models
  88. 88. Basics Web Sessions Social Gaming Tagging Leaderboards Architecture Thanks! aws.amazon.com/dynamodb/developer-resources

×